Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
Authors: Boqian Wu
Qiao Xiao
Shunxin Wang
Nicola Strisciuglio
Mykola Pechenizkiy
Maurice van Keulen
Decebal Constantin Mocanu
Elena Mocanu
Venue: ICLR 2025 Poster (2025)
Keywords: Dynamic Sparse Training
Image Corruption Robustness
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, \textit{we claim that}, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10\% and up to 50\%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.

Published as a conference paper at ICLR 2025
DYNAMIC SPARSE TRAINING VERSUS DENSE TRAIN- ING: THE UNEXPECTED WINNER IN IMAGE CORRUP- TION ROBUSTNESS
Boqian Wu Qiao X Shunxin Wang', Nicola Strisciuglio', Mykola Pechenizkiy", Maurice van Keulen', Decebal Constantin Mocanu 2.3 Elena Mocanu' University of Twente, University of Luxembourg, Eindhoven University of Technology {b. wu, s wang 2, r. stri s. www.ssiiiiiiiiio.o.ooo.. m. vankeulen, e (orcaannu@uutnennnne nl (q.xiao, m. www.ppeeiiiiyytteeeeee nl, {decebal catteeaal.mccauueenne @unii lu
ABSTRACT
is generally perceived that Dynamic Sparse Training opens the door to new cra of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Train- ing is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corrup- claim that,
tion. In this paper, we question this general practice. Consequently, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, partic- ularly if the efficiency aspect is not considered as main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal new yet-unknown benefit of Dynamic Sparse Training and open new possibilities improving deep learning robustness beyond the current state of the art.
INTRODUCTION
If one would like maximize as much possible the robustness of a deep learning model on noisy more realistic data (e.g., image corruption, out of distribution), what would be the typical approach? Would one train directly dense neural network (referred further as Dense Training) sparse ral network (referred further as Sparse Training)? Of course, there are many techniques to enhance the model robustness, but still they would be applied top of basic optimization process which usually requires training an artificial neural network. In order to give an answer to the previously posed question, we would need to quantify the research and practitioner community perception, which is a hard problem in itself. Yet, given the large body of literature on the topic, it is safely assume that the usual answer would be "Yes, Dense Training the typical solution maximize the model robustness.". Nevertheless, the reader can answer to this question by themselves, while further this manuscript we would like to bring more clarity and some new perspectives on this topic.
With respect to Dense Training, recent studies have demonstrated that over-parameterized neural networks, achieve impressive generalizability test data even without explicit regularizers Zhang et al. (2021). This phenomenon is partly explained by the implicit regularizatior induced by the optimization algorithm used during training, such stochastic gradient descent (SGD) ZZhng et al. 2021); Keskar et al. (2017); : Li & Liang (2018). Despite ongoing efforts to understand the robustness behavior of deep neural network models, researchers always favor larger models, trained
Equal contribution.
In older neural network literature, regularization refered specifically penalty term added to the loss function to prevent overfitting. Recently, as in Goodfellow et al, (2016), regularization is broadly defined as "ny modification to learning algorithm intended reduce test without increasing training error."

---

Published as a conference paper at ICLR 2025
densely, due their perceived robustness to unexpected variations in input data, such additional noise, contrast changes, and weather conditions, to a certain extent Hendryekk & Dietterich (2019);
Hendryckk al. (2021). However, larger werrparameterized models do not necessarily gain more
robustness Algan & Ulusoy 2021) Wang et al. (023)).
With respect to sparse training, there static sparse training Liu & Wang (022) and Dynamic
Sparse Training (DST) Mocanu et al.] (2018); [ellec et al. (2018); Mostafa & Wang 2019); : Evei
al. (2000); Yuan et (2021); Liu al. (2021a), which maintain just fraction of the parameters' budget throughout the entire training process. Static sparse training involves sparsifying the network before training and keeps the topology fixed during training. DST. recent and efficient sparse-to- sparse training approach, starts with a sparse neural network and simultaneously optimizes both its weight values and connectivity. This optimization process is known as dynamic sparsity. Conse- quently, it is mainly seen an approach that can maximize deep learning scalability and efficiency, at least in theory, as in practice,
a difficult research problem to fully utilize sparsity during training Nowak et al. (2023). Nevertheless, it is often overlooked that DST frequently achieves performance comparable e to, or even surpassing, that of its dense or static sparse counterparts Chen et al.. (2022b); Xiao al. (2022); Graesser et al. 2022 Sokar et al. 2022 Atashgahi et al. (002 Liu et al. (2023); Sokar et al. (2023).
DST shares many similarities with dense training, including its adaptability to range of learning paradigms, such as continual learning Sokar et al. (2023) and reinforcement learning Graesser et al. (2022) as well as tasks like feature selection| Atashgahi et al. 20022) time series classification Xiao et al. (222)). It is compatible with both convolutional and transformer-basee
architectures and uses core techniques such as dropout, weight decay, and standard optimizers Dey et al. (2024). Despite these parallels, DST utilizes hard sparsity-on on top of the soft sparsity introduced by weight decay or other regularization methods-whie provides additional regularization. This naturally leads to an intriguing question: Is somehow DST implicitly improving model robustmess, and no one knew about this yet?
Although several studies have investigated the impact of pruning model from Dense Training, lot- tery ticket hypothesis methods Frankle & Carbin| 2 Frankle et al. (2020); Renda et al. (2020) on image corruption robustness Liebenwein et al. (2021): i): Diffenderfer et al. (2021) and adversar- ial robustness Liao et al. (2022), other research shows that static sparse networks, when increased width and depth to maintain capacity, can also improve robustness Timpl et al. (2022). However, a closer look at current DST research reveals a literature gap: the overlooked evaluation of DST models under various unexpected image corruptions. In attempt respond to this, examined one of the classical DST algorithms, Sparse Evolutionary Training (SET) Mocanu et al. (2018), as illustrated in Figure [ (Left). We find that SET (with sparsity ratio of 0.5) can markedly improve model robustness against 18 out of 19 image corruptions, compared to Dense Training. Intuitively, adding regularization (e.g., weight decay or dropout) during training is commonly believed to help improve the generalization of neural networks. In light of the strong performance of the DST method (e.g.. SET) against common corruptions, , we propose the Dynamic Sparsity Corruption Robustness (DSCR) Hypothesis.
DSCR Hypothesis. Dynamic Sparse Training at low sparsity levels helps to improve model robust- ness against image corruption in comparison with Dense Training.
We summarize our main contributions as follows:
This study questions the typical approach of using Dense Training to maximize model robustness and proposes Dynamic Sparse Training as a more capable alternative.
Formally, we propose the DSCR hypothesis which is validated in a wide range of corrup- tion scenarios (i.e., corruption type and severity), using different types of data (i.e., image and video), DST algorithms (i.e., SET, Rigl., MEST and GraNet) and deep learning archi- tectures (i.e., VGG, ResNet, EfficientNer, DeiT and 3D ConvNet) (Section
We give insights, from both spatial and spectral perspectives, on the underlying factors that explain the superior performance of DST. Our analysis shows that the dynamic spar- sity process in DST acts automatically focus on more important features (e.g., low- www-fregeeccy components) while reducing its attention to less important ones (e.g., high-frequency components) (Section 5
implicit regularization mechanism, enabling the model to
2

---

Published as a conference paper at ICLR 2025
memory bound
dersty
Waryy (itelar
deriity
ring
baining
d
traking Tin
3.5 0.0 10 3.5
(b) Righ
(0) MEST
(4) Gratet
iobustraee ACC gan
Figure 1: (Left) Robustness accuracy gain (%) in sparsified ResNet34 trained with SET (sparsity ratio=0.5) compared to its dense counterpart, tested on CIFARI00-C with various corruption types shown on the Y-axis. Positive values indicate better performances of SET method. (Right) concep- tual representation of model density during training across different DST algorithms: (a) SET, (b) Rigl. (Note: In the regrow step, Rigl. requires full gradient calculations.), (c) MEST and (d) GraNet.
2 RELATED WORK
2.1 RELEVANT METHODS FOR CORRUPTION ROBUSTNESS
Deep neural networks (DNNs) are susceptible to unseen changes in their inputs, a scenario com- monly encountered in practical applications, such as common corruptions like Gaussian noise, defo- cus blur, etc Hendrycks & Dietterich (2019). To enhance robustness against these common corrup- tions, many methods have been developed, primarily focusing on data augmentation, for example, AugMix Hendrycks* # et al. 2020). AugMax Wang et al. (2021), PRIME Modas al. (2022). These approaches aim to reduce the distribution gap between the training and testing datasets. Alterna- tively, other strategies to enhance corruption robustness concentrate on developing resilient learning approaches, such contrastive learning Chen al. (2020), or designing specialized network components, like the push-pull layer SStrisciuglio et al. (2020). Recent research has revealed that LTH-based sparse models can provide enhanced robustness common corruptions compared to their denser counterparts Diffenderfer et al.] (202T): Chen et al. (2022a). Additionally, |Bair et al.) (224)) introduces an adaptive sharpness-awasd awaree pruning method, which regularizes the model wards flatter regions, thereby improving robustness against image corruption. However, these meth- ods are typically built on dense models either the start or throughout training, often overlooking the aspect of resource-award training.
2.2 DYNAMIC SPARSE TRAINING
Dynamic Sparse Training (DST) Mocanu et al. (2018)) Bellec et al. (2018); Mostafa & Wang (2019)
Evei et al. (2020): Chen al. (2021) Yuan et al. (2021); Wu et al. (2025), sparse-to-sparse train- ing paradigm, starts from scratch and maintains partial parameters throughout training. providing a
promising approach to significantly improve training and inference efficiency while still preserving accuracy. Recent research has focused on deciphering the mechanisms of DST [Luu et al. (2021);;
Evei al. 2022; 2019), and has broadened its application in various domains, such as continuous
learning Sokar et al. (2023). ) reinforcement learning Graesser et al. (2022); Tan et al. (2023). feature
selection SSkkar al. (2022); ; Atashgahi al. (2022). time series classification Xiao al. 2022
network architecture design Liu et al. (2023), and vision transformers Chen et al. (2021).
Moreover, recent studies have concentrated on developing improved DST algorithms Dettmers & Mostafa
Zettlemoyer Mostaaa & Wang (2019); Yuan et al. (202I): ); Jayakumar et al.) (2020): Liu et al. (2021a). For instance, Dettmers & Zettlemoyer (2019) introduced technique to regrow weights based on the momentum of existing weights, while Yuan et al. (2021) proposed a method that utilizes both weight and gradient information for pruning. More recently, has been demonstrated that DST can enhance the performance of DNNs on biased data Zhao et al. (2023) and adversarial samples Chen et al. (022bb). However, the majority of DST research relies on pristine benchmark datasets, leaving the robustness of DST models against image corruption relatively unexplored. This paper aims to systematically investigate the robustness behavior of DST when exposed to various image corruptions.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
3.1 PROBLEM DEFINITION
Given Tir i.i.d training samples {(0]*, v/")) and The test samples {(x) , ) we aim to evaluate the model's robustness in scenarios where the test inputs are corrupted relative to the benchmarked training datasets, (i.e., Ptr(2) + Pte (a)) while the conditional input-label relation- ship remains consistent (i.e., Ptr (y x) = y x)). To achieve this, we utilize standard image (EFAR100/Kizhee
et al. (2009), TinyImageNet Le & Yang (2015) and ImageNet[Deng et al. (2099) for training. Then, we evaluate the performance (robustness accuracy) of the models on the corresponding corruption benchmark datasets, including CIFAR 10-C, CIFAR100-C, TinyImageNet-C. and ImageNet- C/C/3DCC. More details of datasets can be found in Appendix A.2
benchmark datasets such as CIFAR 10 Krizhevsky et al. (2009),
3.2 DYNAMIC SPARSE TRAINING
Consider a dense neural network fa. the sparsification of fe can be formulated as: feoM H for where M is set of binary masks and & denotes element-wise wise multiplication. The sparsity ratio S of the resulting subnetwork, fer, is defined as counting the number of non-zero elements. The target memory budget (or density) ratio is given by jo'lle 1
1e Tol where || llo represents the Lo norm,
The topology of the sparse network is dynamically updated during training every A/ iterations through a parameters remove-regrow process. Specifically, connections with low importance are removed, and new connections are regrown based criteria like gradient information Evei et al. (2020) or randomly Mocanu et al. (2018). Dynamic sparse models are the resulting models from DST. In our study, we explore various DST approaches as follows:
DST with fixed memory budget in forward: To the best of knowledge, SET is pio- neering work in DST. SET follows a magnitude- baase removal and random regrow strategy, maintaining fixed the memory budget, that is ||0'
lello b. throughout the entire training process, as shown in Figure [ (right (a)). RigL Evei et al. (2020) regrows weights based on the gradient information of all weights, while maintaining constant memory budget during the forward phase of training, as shown in Figure (right (b)). In both approaches, the mask M is updated every AL iterations.
Sparse Training (MEST) Yuan et al. (202I), with Soft Memory Bound Elastic Mutation, starts with a sparse model having memory budget of lo = b- b,, where b, > 0 represents the soft memory bound that decays during training. Subsequently, at intervals of At, weights are removed based on magnitude and gradient information, reducing the memory budget to b, and then regrown it to b + by as illustrated in Figure [ (right (c)). The soft memory constraint b,. gradually decays to zero, allowing the model to reach the target memory budget
DST with soft memory budget in forward:
DST with flexible memory budget in forward: GraNet Liu et al. (202 starts with a ||0'|lo < 1). During training, at intervals of Ai, it removes denser neural network (i.e., T00
weights based magnitude and regrows connections using gradient information, similar to RigL. The main difference is that fewer connections are regrown than those removed. This process dynamically updates the mask M throughout training, gradually reaching the target memory budget ratio b by the end, as shown in Figure (right (d)).
In this paper, for simplicity in analysis, we use unified magnitude- based removal approach and implement either random (())) or gradient-based (i.e., re-growing the weights with the highest magnitude gradients)
regrow strategy across different DST methods. Unlike their original papers Yuan et al.] (2021): Liu et al.) (2021a), where MEST utilized random regrow and GraNet employed gradient- based regrow, regrow DST, our analysis includes SET, MEST,, and GraNet,, while for gradient- based regrow DST, we examine RigL, MEST g. and GraNet,. This approach allows to consistently evaluate the robustness of DST models under varying memory budget constraints, such as fixed, soft, or flexible, during training.
explore both strategies for each model. Thus, for random
4

---

Published as a conference paper at ICLR 2025
4 CAN DST IMPROVE ROBUSTNESS AGAINST IMAGE CORRUPTION?
In this section, we aim to carry out extensive experiments to verify the DSCR Hypothesis across a wide variety of DST methods, models and data. To this end, we conduct comprehensive set of experiments to evaluate the robustness accuracy of DST models with different sparsity ratios in the presence of image corruption.
To comprehensively validate the DSCR Hypothesis, our study embarks on an extensive analysis evaluating the robustness behavior of sparsely trained models. We train using several representative DST algorithms, including SET, Rigging the Lottery (RigL) Evci et al. (2020), Memory-Economie Sparse Training (MEST) Yuan al. (2021), and Gradual Pruning with zero-cost Neuroregenerrtioo (GraNet) Liu et al. (2021a) la), across wide range of model architectures (i.e., VGG Simonyan &
Zisserman (013). 2D/3D ResNet He et al. (016)) EfficientNet TTan Le (2019). DeiT' Touvron
et al. 2021 and Two-Stream Inflated 3D ConvNet (I3D) Carreira & Zisserman (2017)).
Our analysis spans a range of image corruption benchmarks CIFARIO-CHHendryck Dietterich
(201)) CIFAR 100-C Hendrycks & Dietterich (2019), TinylmageNet-C [Hendrycks & Dietterich
Dietterich (2019), ImageNet- Mintun al. (2021). ImageNet- 3DCC Kar et al. (2022) and corrupted UCF101 F101 [Soomro al. (2012) -covering 19 types of im- age corruptions for CIFAR 10/100-C, 15 types for TinylmageNet-C and ImageNet-C, 10 types for ImageNet- C and 11 types for ImageNet-3DCCC with each corruption tested across five severity lev- els. The corrupted UCF101 dataset includes 16 corruption types with four severity levels. This anal- ysis provides a comprehensive perspective the robustness evaluation of sparsely trained models.
2209 ImageNet-Ci Hendrycks
Detailed dataset descriptions and implementation setup can be found in Appendix A. and A.3.
dense
SET
MEST.
GraNet,
donse
RigL
MEST,
GraNet
74.6
17
74.4
52.3
73.9
$1.8
73.6
57.3
73.2
$1.3
72.0
$1.8
72.9
0.3 044 0.3
0.6
0.7
0.4
0.5
0.5
0.7
0.3
0.4 0.5
0.6
0.7
17200
0.4
0.5
0.6
$1.3
0.5
6.4
0.S
6.6
0.7
0,3 0.4 03 0.6 0.7
sparsity rablo
sparsity retio
sparsity raño 10)
Spansity rato
(a)
[c)
(8)
19)
Figure 2: Robustness accuracy (%) for (a) VGG16 CIFARIO-C, (b) ResNet34 CIFAR100- and (c) EfficientNet-BB BO on TinylmageNete comparing different DST algorithms with dense training. Left: Results for DST algorithms with random regrow strategy, including SET, MEST,, and GraNet,. Right: Results for DST algorithms with gradient-basee regrow strategy, such as RigL, MEST and GraNetg-
4.1 OVERALL PERFORMANCE
Figure 2 showcases the average robust accuracy across various types of corruption, with severity levels ranging from to 5, on the 10AAR00 and IinyImageNet-C datasets. We can find that DST models outperform their Dense training counterparts (as indicated by the blue line Figure 2 in terms of robustness accuracy across several sparsity ratios. This trend is evident in both the dom regrow (Figure 2 left) and gradient- based regrow (Figure 2 right) strategies. Specifically, for CIFAR 100-C and Imageeee C, DST models with sparsity ratios ranging from 0.3 0.7 gen- erally show encouraging improvements in robustness. For CIFAR 10-C, at certain sparsity ratios, such 0.4, the overall DST methods achieve decent robust performance against image corruptions. Moreover, based on our experimental setup, at least 40% of both training computational and memory costs can be saved while simultaneously enhancing corruption robustness. Details on FLOPs and parameter counts are available in the Appendix A.3.2] This finding substantiates our DSCR Hypoth- esis across different DST methods, varying sparsity ratios, and architectures (i.e., VGG, ResNet, and EfficientNet).
4.2 PERFORMANCE ON CORRUPTION TYPES
We further evaluate the DST methods across various corruption types with different severities. Fig- ure provides detailed relative robustness gains for each type of corruption in ImageNet-C and
Defined as (Acce.01 Acca..1) /ACc..a. where Acc,,si and Acca.11 represent the accuracies of models trained using DST and dense training, respectively, severity level of al.
5

---

Published as a conference paper at ICLR 2025
ImageNet- C Mintun et al. (2021) compared to their dense model counterparts. It is worth mention- ing that the X-axis in Figure B represents different types of corruption, which are ordered according to the amount of high-frequency information they contain, as in Saikia et al. (202I). Intriguingly, on ImageNet-C and ImageNer-C DST models show significant potential in enhancing robustness against specific corruptions, such as impulse, Gaussian, and shot noise ImageNet-C, as well as brownish, Perlin, plasma, and blue noise on ImageNet-C. It is worth noting that these corruptions are typically characterized by relatively high-frequency information. This finding is consistent on CIFARIO-C, CIFARIOO 100-C, and IinylmageNet-C. and due to page limitations, the results for these datasets are detailed in Appendix A.4
Robustness Acc: 38.70 %
Robustness Acc: 38.64 %
Robustness Acc: 38.72 %
25.00
17.00
17.00
1.00
5.00
9.00
100
1.00
1.00
3.00
7.00
brighneng
dlsti fro fracston
and tlar thur
15.00
mastcc transter
men dar
gain Maig rolus
brightm
destinn elsst sigeetine
and
solce
Robustness Acc: 41.44 %
Robustness Acc: 40.50 %
Robustness Acc: 41.07
20.00
9.00
4.00
10.00
-1000
4.00
20.00
6.00
diggte covelic sofo rocmtric
ceectopicn cortorad
frvesse stara spankio
acaveee brownsth Jolie pertin moide
paxmm couse
bloe, moio
dingle coostic cala ceecerrzond conerttc invense sharies taer
brseees, roise perii volie
paamm polit
Nike, poge
diigle chustic refa medeeriall of
rocertnc sige
inverse inariee
Immondt, nalle pentes moler
ponsiee
iieame rolae
thse
Figure 3: Relative robustness accuracy gain (%) on ImageNet-C (top) and ImageNet-C (bottom) for ResNet50, trained using DST with gradient-based strategies (i.e., (a) RigL, (b) MEST, (c) GraNet,) at sparsity ratio of 0.1, compared to dense baseline (which has a mean robustness accuracy of 38. 38.38% on ImageNet-C and 40.38% on ImageNet-C), Positive values reflect better performance by the sparse models compared to the dense baseline. The five bars, ranging from light to dark shades, represent corruption severity levels from 1 to 5 for each type of corruption. The title indicates the mean robustness accuracy for each corresponding DST method.
Additionally, we observe that DST models exhibit more pronounced advantage in robustness over Dense Training models under high severity levels, particularly with noise corruptions. Specifically, as shown in Figure 33 (top (b)), the robustness performance of MEST, at severity level 5 for impulse noise and Gaussian noise demonstrates nearly 25% improvement. This finding is also evident in CIFARIO-C, CIFAR 100-C, and TinyImageNet-C; more details can found Appendix [A.4] This observation highlights the potential of models trained with DST to tackle high-frequency cor- ruptions, especially harsher scenarios, offering the potential to enhance DNNs' robustness in complex corruption-pront environments.
4.3 ROBUSTNESS OF DST MODELS: IS TRUE FOR OTHER DATASETS, TASKS, AND ARCHITECTURES?
Herein, we evaluate the robustness of DST models using a more realistic corruption dataset (e.g., ImageNet-3DDCCKa et al.) (2022)). and across different tasks (e.g.. video classification), with mod- em architectures ttransformer-based e.g.. DeiT-base Touvron et al. (202T)).
Evaluation on ImageNet-3DDC.. We evaluate DST models on the ImageNet-3IDCC, a challenging dataset with realistic corruptions. The ImageNet-3DCC dataset introduces corruptions aligned with 3D scene geometry, simulating real-world shifts including camera motion, weather conditions, oc-
3 "Our experiments primarily focus on DST with gradient- regrow strategy when training on ImageNet. As our implementation is based on PyTorch, this regrow approach tends to achieve strong performance on ImageNet more quickly with this framework Liu et al. (2021a). However, it is worth noting that the random- based approach has also shown significant improvements when implemented in JAX Lee al. (2024), which be future exploration.

---

Published as a conference paper at ICLR 2025
clusions, depth of field, and lighting variations. The improved average test accuracy on ImageNet- 3DCC, as shown in Table indicates that DST models excel not only at handling synthetic dis- tortions, discussed in Section 4.1 and Section 4.2 but also at adapting practical, real-world corruptions.
Evaluation video classification. We ex- tended our exploration to video data using the UCF101 dataset (Staset(Soomro et al. (2012), an action recognition dataset containing 101 categories and approximately 13K videos. Following the settings described in [Schiappa et al. (2023), we compared the performance
Table 1:
The robustness accuracy (%) of dense and DST models on ImageNet-3DCC and UFC101 datasets.
Dataset
Model
Dense
Rigt.
MEST,
GraNet,
ImagoNet-3DDC
ResNe150
43.62
44.08
43.87
44.46
dense training and DST at sparsity level of 0.1 on two archi- tectures: 3D ResNet50 and Two-Stream Inflated 3D ConvNet (I3D) Carreira Zisserman (001
UCFI0I
3D ResNett0
D3D
51.14
53.58
52.71
$5.00
53,00
$6.30
53.57
55.21
Further implementation details are provided in Appendix [A.3. Both models were trained from scratch for 100 epochs on UCFI0L, Our analysis, based on test set featuring 16 corruptions across severity levels to 4, shown in Table I revealed that DST models consistently achieved superior robust accuracy compared to their dense counterparts, even within the context of video data.
Robustness Acc: 54.73
Robustness Acc: 55.15 %
Robustness Acc: $4.94
10.00
10:00
5.00
3.00
0.00
0.00
5.00
dccee elaste contrgt
fidt plat
dez conple ceteeiiin coatiit
bripttmant
5.00
snoed etast contrant
wttaee juet cooeee Beth
bagetneet
Coet elastic prrttio
dorg comph
Figure 4: Relative robustness accuracy gain (%) on ImageNet-C for a DeiT-base, dynamically trained with gradient-based methods (i.e., (a) RigL, (b) MEST,, (c) GraNet,) at sparsity ratio of 0.1, compared to a dense baseline (with a robustness accuracy of 54.68% on ImageNet-C). Posi- tive values indicate superior performance by the sparse models.
Evaluation on transformer-basee architecture. We evaluate the relative robustness accuracy gain of various sparse and dense DeiT-base models on ImageNet-C across different corruption types, shown in Figure 4 Following the experimental settings in Chen et al. (202T), we conducted training from scratch over 200 epochs. The results show that DST models generally outperform their dense counterparts, with a particularly better advantage in handling noise corruptions. This finding, consistent with our observations in CNNs discussed in previous sections, suggests that the robustness benefits of DST models are applicable alternative architectures, including transformer- based model.
In Appendix A.9 we compare the robustness of dense trained and DST models on an image segmen- tation task and find that DST models are robust not only in image and video classification but also image segmentation. In summary, our empirical findings indicate that DST methods can surpass Dense Training models in robustness performance across various datasets specifiic sparsity tios, without (or even reduce) extra resource cost [ against different types of corruption, particularly those with more high-frequency information. The mystery behind these observations will be further explored in the following sections.
Furthermore, DST models tend to perform better
WHAT IS BEHIND THE CORRUPTION ROBUSTNESS GAIN IN DST?
Building on the previous empirical results, where DST showed improved robustness against com- mon corruptions, we are motivated to explore the inner reasons for the robustness of DST models. Our focus is on analyzing how DST and Dense Training models utilize their parameters in the spatial domain and process the input in the spectral domain.
We use binary masking operation simulate the sparisty, the computation cost of binary masking oper- ation is negligible even reduced when the hardware support sparse matrix product.
7

---

Published as a conference paper at ICLR 2025
5.1 ANALYSIS THROUGH SPATIAL DOMAIN
Traditionally, the constraints on weight values, such as L2 regularization, during dense training have been an effective method to mitigate the problem of overfitting, In contrast, DST imposes sparsity constraints, enabling the selective removal of parameters. In this section, we aim to intuitively investigate how DST affects the location of non-zero weights. To achieve this, we performed visualization analysis to pinpoint the locations of sparse parameters for DST models. Figure 5 showcases the zen-zero weight counts and the sum of weight magnitudes (i.e., absolute values) for each kernel in specific layer.
supptt dhaput
100
25g
150
200
200
250
50
100
150
202
250
250
100
150
200
250
50
100
150
200
250
50
100
150
200
250
100 150
200
250
ourp Intex
A3
50
50
50-
100
100
100
200
100
150
U50-
150
230
150
200-
200
200
200
250
250
230
250
130
250
Input channe Index
Input chonnel Index
Inout channel index
Imput charnel Index
irputt chennel index
(4) Dense
(b)
(c)
(c) MEST,
(c) GraNet,
Figure 5: Visualization of the non-zero weight count (1st row) and the of weight magnitudes
(2nd row) within a 3x3 kernel from the #layer3.3.com convv. of ResNet34 after training on the CIFAR100 using dense training different DST methods (with a sparsity ratio of 0.5). Each value in the figure is derived from calculations within the kernel, with lighter colors indicating larger values.
Interestingly, the heatmaps in F show dark vertical and horizontal lines, indicating a concen- tration of either sparsified weights weights with smaller magnitudes in the filters corresponding to specific input or output channels (features). While both dense and DST models exhibit such a concentration phenomenon, in the case of the dense model, this concentration is solely due to the adjustment of weight magnitudes. These magnitudes tend to converge near, but not exactly at, zero, indicated by the consistently non-zero weight count of 9 in 3 (1st row (a)). In contrast, the dynamic sparse model attributes this phenomenon to both the adjustment of weight magnitudes and the allocation of sparsified weights, as shown in Figure S (Ist row (b, c, d. e)), where the non-zero weight count approaches zero. The findings are consistently evident in other layers, particularly in deeper layers, and across different models. More visualizations can be found in the Appendix [A.5] Compared to dense training, we suggest that dynamic sparsity in DST introduces form of implicit regularization. This leads to the prominence of sparsified weights on particular filters, indicating a reduced connectivity to specific features that are determined to be less crucial.
Furthermore, based on the empirical results Section 4.2 prisingly more effective at countering corruptions rich in high-frequency information compared to their dense training counterparts. We assume that the improved robustness of DST models against these corruptions stems from their inherent tendency to de-emphasize high-frequency fea- tures. This selective focus on specific frequency ranges enhances the network's ' robustness against high-frequency corruptions, thereby providing a distinct advantage in handling noise corruptions rich high-frequency information. Further analysis will be provided in the following section.
we observed that DST models are sur-
5.2 ANALYSIS FROM SPECTRAL DOMAIN
Motivated by previous work Yin et al. 2019), which examines ImageNet accuracy after low-pass and high-pass filtering to understand how neural networks utilize frequency information for image recognition, and building on the observations and assumptions from the previous section, will further explore dynamic sparse models from a spectral perspective.
In this section, explore how the test accuracy of both DST and Dense Training models is affected by the removal of high and low- www-freqenccy components from the test data after training. In detail, the test images first transformed into the Fourier spectrum, followed by frequency attenuation
8

---

Published as a conference paper at ICLR 2025
operation. The attenuation is applied in two forms: high-frequency attenuation thigh and low- frequency attenuation alows each controlled by an attenuation radius r. A larger r leads to more information being filtered out, depicted in Figure
We define ACChigh (M,, and ACC1os (M, r) as the test accuracies of model M E M, when tested datasets processed with high- frequency attenuation dhigh and low-frequency attenuation Alow re- spectively, using attenuation radius Here, M represents the set of all considered models. The Radius-Accuracy (RA) curves for ahigh and flow illustrate the relationship between frequency atten- uation range and test accuracy:
Figure 6: Illustration of high-frequency (high and low-frequency Utow attenua- tion. Gray areas indicate the removed frequency com- ponents, with low frequencies near the center.
Car. = {(r, AcCnigh (M,r)) M, Car,ia= == [(r, AcCio (M, M M.,MEER,
(I)
where R denotes the set attenuation radius. Figure 7 (left) il- lustrates toy example of ImageNet with attenuated high and low- frequency information at different attenuation radius, and provides the RA Curves C.ig and Cy, colored with orange and blue.
respectively. In detail, with high-frequency attenuation, images appear blurred but remain recogniz- able to the human. contrast, low-frequency attenuation leaves only the basic outlines of objects visible. Figure 7 (right) shows the detailed RA curves for both dense and dynamic sparse models across different datasets. These models are tested on the CIFAR 100, TinyImaeeNet, and ImageNet test sets, each processed with high- and fow-frequency attenuation at varying radius.
%
40
75
Accuracy
67
Accuracy
30 26
28
73
10
45
40
80
40
nigh tequenty
hrquuetyy
Accuracy
55
65
Accuracy
90
Accuracy
20
20
20
10
10
15
aftenuation range
6
10
12
14
4R
20
40
60
80
Attenustionn
attenuation
attenuation range
(8)
(b)
(c)
Figure 7: (Left) A toy example demonstrating the impact of high- and low- low-frequency attenuation on dense ResNet50 model using the ImageNet set. (Right-top) High-frequency attenuation and (Right-bottom) Low-frequency attenuation performance: (a) ResNet34 (sparsity 0.5) on CIFAR 100, (b) EfficientNet-B0 (sparsity 0.5) on TinylmageNet, and (c) ResNet50 (sparsity 0.1) on ImageNet. The x-axis represents the attenuation radius, and the y-axis shows test accuracy.
Observation Equal attention for low-frequency information. Given models M M and the set of attenuation radius E R, it is observed that Cs, lie below the CM This reveals that attenuating low-frequency information significantly degrades classification accuracy, highlighting the crucial role of low-frequency information in image recognition tasks, consistent with findings in Yin al. (219)). Under low-frequency attenuation atow. both DST and Dense Training models exhibit similar performance, as shown in Figure 7 (right, 2nd row), suggesting that DST models allocate attention to low-frequency information manner comparable their Dense Training counterparts.
Observation 2: Less attention to nigh-frequency information: When subjected to high- frequeryy attenuation dnigh. DST models consistently outperform Dense Training models across all evaluated datasets, as shown in Figure| (right, 1st row). This suggests that feguffreyencc attenuation has less impact DST models, indicating they less sensitive high-frequency information. This finding further supports the robustness of dynamic sparse models against common image corruptions rich in high-frequency content, discussed in Section 5.1

---

Published as a conference paper at ICLR 2025
Based on the above observation, conclude that DST introduces a form of implicit regularization, reducing the focus on high-frequens information. This selective attention in dynamic sparse models enhances their robustness, especially in handling corrupted images with high gegh-freuencc content, such as noise corruption.
Previous works Li et al. (202T); Grabinski et al. (2022) have pointed out that High-frequencr compo- nents can cause frequency aliasing when downsampling operations (e.g., AveragePooling and Max- Pooling) are performed in the spatial domain of neural networks, as explained by the sampling theo- rem in information theory. Specifically, (Li et al. (02T) demonstrated that relying on low-frequency components of wavelets improves noise robustness. attributing this to the suppression of aliasing effects. Furthermore, (Grabinski et al. [2022] 2023) showed that removing high- tigh-frequency compo- nents to reduce frequency aliasing enhances the robustness of networks. These studies validate that reducing high-frequency components mitigates frequency aliasing and improves robustness, align- ing with our findings on the relationship between DST models' tendency to pay less attention to high-frequency information and their improved robustness against common corruptions.
FURTHER ANALYSIS AND DISCUSSION
Our study questions the conventional approach of using "de facto" dense training for model robust- ness in favor of sparse training. Due to the page constraint, we also analyze the complementarity between DST and data augmentation (c.g.. Mixup Zhang et al. (2018) 8)) in Appendix AA.7| and fur- ther explore DST's robustness with Out-Of-Domain (OOD) data and Grad-CAM Selvaraju et al. (217)) in Appendix A.8 and IO, respectively. Furthermore, we extend and validate our DSCR hypothesis with more recent DST methods, including Cannistraci- Hebb soft training (CHT's Zhang et al. (2024b]a), a). which employs more complex growth criteria and topology initialization, as de- tailed in Appendix A.6 In the future, it is worth moving beyond the basic fact revealed in this paper, i.e. vanilla DST outperforms vanilla Dense Training for model robustness, and to devise new DST methods for model robustness especially in the high sparsity regime, and to replace Dense Training with DST in the current state-of- the-art techniques for model robustness. Additionally, while empirically analyze robustness behaviors between dense and sparse training and provide insights from both spatial and frequency domains, further work on the theoretical understanding of this phe- nomenon, along with a fine-grained exploration of different components of DST (e.g., topology initialization, regrowth methods and regrowth weight initialization), could provide deeper insights for the development of more robust and efficient networks. This would be a promising direction for future research.
7 CONCLUSION
This paper questioned the typical approach of using "de facto" Dense Training to maximize model robustness in deep learning, and revealed an unexpected finding. That is: "Dynamic Sparse Training at low sparsity levels can train more robust models against image corruption than Dense Training.". This finding has been confined formally in the DSCR Hypothesis and rigorously validated on nine scenarios (as sum- marized in Table where a striking observation can be made all DST algorithms studied out- perform Dense Training in all scenarios). To understand what factors contribute to this com- pelling and intriguing DST behaviour, to the end, we present a deeper analyze. We show that the dynamic sparsity process performs a form of implicit regularization which makes the model to automatically learn to focus on the more informative features (c.g., rich in low-frequency), while paying less attention to the less informative ones (e.g., rich in high-frequency).
Table 2: Summary of the main results present- ing the 9 scenarios studied in this paper. The table takes a snapshot at a particular sparsity level and presents the robustness accuracy (%) of the gradient-baser regrow DST methods. The bold values represent the best performer between Dense Training and each DST method taken sep- arately. Tiata type
Modil
Tonu
Byrsmii Spante Triining
Trainins
Sparaat
Nigh.
SIET
GraNEE
CHPAR 00-E
51.65
014
$3.56
$3.56
$1.75
15:42
01.
16.46
16.5
36.72
Iragelie C
40 14
01
61.64
48.50
$1.07
Trayehou 1DOY
65.67
01
41/
41.87
41.4%
vakiee
SCFF00
01
17
3377
53,00
37787
06.1
57577
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Qiao Xiao is supported by the research program 'MegaMind Measuring, Gathering, Mining and Integrating Data for Sell-mangeemntt in the Edge of the Electricity System', (partly) financed by the Dutch Research Council (NWO) through the Perspectief program under number P19-25. Elena Mo- canu partly supported by the Modular Integrated Sustainable Datacenter (MISD) project funded by the Dutch Ministry of Economic Affairs and Climate under the European Important Projects of Common European Interest Cloud Infrastructure and Services (IPCEI-CIS) program for 2024- 2029. This research used the Dutch national e-infrastructure with the support of the SURF Cooper- ative, using grant no. EINF-7499.
REFERENCES
Görkem Algan and Iaka Ulusoy. Image classification with deep learning in the presence of noisy Systemsssss,,s Systems, 215:106771, 2021.
labels: A survey. Knowledge-Based
Mike Ashby, Christiaan Baaij, Peter Baldwin, Martijn Bastiaan, Oliver Bunting, Aiken Cairncross, Christopher Chalmers, Liz Corrigan, Sam Davis, N. van Doom, Fowler, Graham Hazel, Basile Henry, David Page, Jonny Shipton, and S. C. Steenkamp. Exploiting unstructured sparsity on next-generation datacenter hardware. 2019.
Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning, pp. 1-38, 2022.
Anna Bair, Hongxu Yin, Maying Shen, Pavlo Molchanov, and José M. Álvarez. Adaptive sharpness- aware pruning for robust sparse networks. In International Conference on Learning Representa- tions (ICLR), May 2024.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In 6th International Conference on Learning Representations, 2018.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724 4733, 2017.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988 2021.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Yang Zhang, Shiyu Chang, and Zhangyang Wang. Data- efficient double-win lottery tickets from robust pre-training. In Proceedings of the 39th Interna- tional Conference on Machine Learning, pp. 3747-3759, 2022a.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generalization from more efficient training. In The Tenth International Conference Learning Representations, 2022b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, 2020.
Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(2):292 -08, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing 2019.
performance. arXiv preprint

---

Published as a conference paper at ICLR 2025
Nolan Dey, Shane Bergsma, and Joel Hestness. Sparse maximal update parameterization: A holistic approach to sparse training dynamics. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems, 2024.
James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve out-of-distribution robustness. In Ad- vances Neural Information Processing Systems, 2021.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 10732, 2019.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Utku Evei, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6577-6586, 2022.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 3259-3269, 2020.
lan Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Learning. Adaptive computation and machine learning. MIT Press, 2016.
Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencyloweut pooling plug and play against catastrophic overfitting. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022 17th European Conference, 2022.
Julia Grabinski, Janis Keuper, and Margret Keuper. Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling. CoRR, abs/2307.09888 2023.
Laura Graesser, Utku Evei, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo. Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021.
generalization. In 202/
Dan Hendrycks* Norman Mu Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Laksh- minarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021.
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems, 2020.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pp- 18941-18952. IEEE, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations. OpenReview.net, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Joo Hyung Lee, Wonpyo Park, Nicole Elyse Mitchell, Jonathan Pilault, Johan Samir Obando Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, et Jaxpruner: A concise library for sparsity research. In Conference Parsimony and Learning, pp. 515-528. PMLR, 2024.
Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavecnet: Wavelet integrated cnns suppress aliasing effect for noise-robust image classification. IEEE Trans. Image Process., 30:7074-7089, 2021.
Yuanzhi Li and Yingyu Liang. Learning overparameterrized neural networks via stochastic gradient descent on structured data. Advances neural information processing systems, 31, 2018.
Ningyi Liao, Shufan Wang, Liyao Xiang, Nanyang Ye, Shuo Shao, and Pengzhi Chu. Achieving adversarial robustness via sparsity. Mach. Learn., 111(2)685-711, 2022.
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost pruning: The effects of pruning neural networks beyond test accuracy. CoRR, abs/2103.03014, 2021.
Shiwei Liu and Zhangyang Wang. Ten lessons we have learned the new "sparseland": A short handbook for sparse neural network researchers. CoRR, abs/2302.02596, 2023.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi. Lu Yin, Huanyu Kou. Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908 9922, 2021a.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neu- commodity hardware. Neural Computing and Applications, 33:2589-2660 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parann parameterization? in-time over-parameterization in sparse training. In International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021c
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Constantin Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. In The Eleventh International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and cor- ruptions in natural corruption robustness. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems, pp. 3571- - 3583, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 911):3383, 2018.
Apostolos Modas, Rahul Rade, Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfoolii and Pascal Frossard. Prime: A few primitives can boost robustness to common corruptions. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, 2022.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference Machine Learning, pp. 4646-4655. PMLR, 2019.
Aleksandra Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor. Fantastic weights and how to find them: Where prune in dynamic sparse training. Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2023, 2023.
Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. AC/DC: alternating com- pressed/decoompressed training of deep neural networks. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, pp. 8557-8570, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Tonmoy Saikia, Cordelia Schmid, and Thomas Brox. Improving robustness against common corrup- tions with frequency biased models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10211-10222 2021.
Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh S. Rawat. A large-scale robustness analysis of video action recognition mod- els. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14698-14708. IEEE, 2023.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In IEEE International Conference Computer Vision, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 2015.
Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Where to pay attention in sparse training for feature selection? In Advances in Neural Information Processing Systems, 2022.
Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Avoiding forgetting and al- lowing forward transfer in continual learning via sparse networks. Machine Learning and Knowledge Discovery in Databases: European Conference, pp. 85-101. Springer, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.00 212.0402, 2012.
14

---

Published as a conference paper at ICLR 2025
Nicola Strisciuglio, Manuel Lopez-Antequera, and Nicolai Petkov. Enhanced robustness of convo- lutional networks with a push-pul -uul inhibition layer. Neural Computing and Applications, 32(24): 17957-17971, 2020.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 6105-6114, PMLR, 2019.
Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. RLx2: Training sparse deep inforcement learning model from scratch. The Eleventh International Conference on Learning Representations, 2023.
Lukas Timpl, Rahim Entezari, Hanie Sedghi, Behnam Neyshabur, and Olga Saukh. Understanding the effect of sparsity on neural networks robustness. CoRR, 10ss/2200.10915. 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 10347-10357. PMLR, 2021.
Haotao Wang. Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, 2021.
Shunxin Wang, Raymond Veldhuis, and Nicola Strisciuglio. The robustness of computer vision models against common corruptions: a survey. arXiv preprint arXiv:2305.06024 2023.
Boqian Wu, Qiao Xiao, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Decebal Constantin Mocanu, Maurice Keulen, and Elena Mocanu. E2enet: Dynamic sparse feature fusion for accurate and efficient 3d medical image segmentation. Advances in Neural Information Processing Systems, 37:1 0011888888181111 2025.
Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Dece- bal Constantin Mocanu. Dynamic sparse network for time series classification: Learning what to "see". In Advances Neural Information Processing Systems, 2022.
Dong Yin, Raphael
Gontijo Lopes, Jonathon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems, 2019.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances Neural Information Processing Systems, 14:20838-2083 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In 6th International Conference on Learning Representations, 2018.
Yingtao Zhang, Jialin Zhao, Ziheng Liao, Wenjing Wu, Umberto Michieli, and Carlo Vittorio Can- nistraci. Brain-inspired sparse training in mlp and transformers with network science modeling via cannistraci-hebt soft rule. Preprints, June 2024a. doi: 10.00944/prepinns202200.0 136.v1.
Yingtao Zhang, Jialin Zhao, Wenjing Wu, Alessandro Muscoloni, and Carlo Vittorio Cannistraci. Epitopological learning and cannistraci-hed network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. The Twelfth International Conference on Learning Representations, 2024b.
15

---

Published as a conference paper at ICLR 2025
Jiaxu Zhao, Lu Yin, Shiwei Liu, Meng Fang, and Mykola Pechenizkiy. REST: robustness in dnns through reweighted sparse training. In Machine Learning and Knowledge Discovery in Databases: Research Track European Conference, ECML PKDD 2023, Turin, Italy, September 18-22, 2023, Proceedinggs, Part 11, 2023.
enhancing group
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning nam fine-ge tne-grained structured sparse neural networks from scratch. In Interna- tional Conference on Learning Representations, 2021.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DATASET INTRODUCTION
A.1.1 CORRUPTION DATASET
Image corruptions refer to visible distortions in images that lead to data distribution shifts from that of the original data. They common in practical applications when models deployed in the real world. Typical corruptions include Gaussian noise, impulse noise, defocus blur, etc. In Figure 8 we display the common image corruptions on the CIFAR-10 dataset.
glass blur
gussian blur
motion blur
defoucs blur
zcom blur
speckle noise
gussian nosie
impluse nosie
shot nosie
snow
frog
frost
clastic transform
pixelate
saturate
jpeg compression
contrast
Brightness
spatter
Figure 8: Visualization of the common image corruptions on CIFARIO dataset.
A.2 RELATED BENCHMARKS
CIFAR-10/100- Hendrycks & Dietterich (2019) is synthetically generated top of the test set CIFAR- 10/100 dataset. It includes 19 sub-datasets, each corrupted with a type of image corruption (Gaussian noise, impulse noise, shot noise, speckle noise, defocus blur, Gaussian blur, glass blur, motion blur, zoom blur, brightness, fog. frost, snow, spatter, contrast, elastic transform, JPEG compression, pixelate, saturate). Each corruption dataset contains five subsets, which have images corrupted with five severity levels. The higher the severity, the influence the corruption has the test images.
Iiny-ImageNNe-CC Hendrycks & Dietterich (2019) consists of 15 types of common im- age corruptions, and is synthetically generated from Tiny-ImageNet dataset. The types of corruption Gaussian noise, impulse noise, shot noise, defocus blur, glass blur, motion
17

---

Published as a conference paper at ICLR 2025
blur, zoom blur, brightness, fog. frost, snow, contrast, pixelate, elastic transform, and JPEG compression, with 5 severity levels.
Hendrycks Ceend-yyckss & Dietterich (0019) consists of 15 types of common image cor- ruptions, and is synthetically generated from ImageNet dataset. The types of corruption are Gaussian noise, impulse noise, shot noise, defocus blur, glass blur, motion blur, zoom blur, brightness, fog, frost, snow, contrast, pixelate, elastic transform, and JPEG compression, with severity levels.
ImageNet-CC
ImageNet-CC Mintun et al. 2021 are perceptually dissimilar to ImageNet-C in our trans- form feature space, consists of 10 types of common image corruptions. The types of corrup- tion are blue noise, brownish noise, caustic refraction, checkerboard cutout, cocentric sine waves, inverse sparkles, perlin noise, plasma noise, single frequency greyscale, sparkles, with 5 severity levels.
ImageNet-3DCC Kar et al. (2022): addresses several aspects of the real world, such as camera motion, weather, occlusions, depth of field, and lighting. It consists of 11 types of common image corruptions: near focus, far focus, fog 3d, flash, color quant, low light, xy motion blur, motion blur, iso noise, bit error, h265 abr, with 5 severity levels.
Corrupted UCF101 Soomro et al. (2012): realistic action recognition dataset featuring videos collected from YouTube, spanning 101 different action categories. It consists of 16 corruptions: motion blur, compression, defocus blur, gaussian noise, shot noise, impulse noise, zoom blur, rotate, static rotate, speckle noise, sampling, reverse sampling, jumble, box jumble, freeze, translate, with severity levels.
A.3 EXPERIMENT SETUP
A.3.1 IMPLEMENTATION DETAILS
For CIFARIO, CIFARIO0, and ImnylmageNet, the codes were implemented in Python using PyTorch and executed on a single NVIDIA A100 GPU. Each was run three times using three fixed random seeds. The ImageNet experiments were run using distributed computing on four NVIDIA A100 GPUs. A batch size of 64 was utilized for each task, resulting in a combined total batch size of 256. For the UCF101 dataset, we followed the settings described in |Schiappa et al. (2023), utilizing the codebase For the FashionMNIST dataset, we followed the training settings detailed in Zhang al. 2024b using the codehase
Table 3: The experimental hyperparameters settings. The hyperparameters include Learning Rate (LR), Batch Size (BS), LR Schedule (LRS), Optimizer (Opt), Weight Decay (WD), Sparsity Dis- tribution (Sparsity Dist), Topology Update Interval (AT), Pruming Rate Decay Schedule (Sched), Initial Pruning Rate (P).
Model
Data
Epoch
BS
LRS
Opt
WD
Sparsity Dist
A7
Schod
VGG16
CIFARIO
160
0.1
100
Cosine
SGD
5c-4
Unifarm
500
polynomial
0.1
ResNet34
CIPAR100
160
0.1
100
Cosine
SGD
5c-4
Uniform
500
polynomial
0.1
EfficienNeetB00
TinyInageeet
128
Cosine
SGD
5e-4
Uniform
100
polynomial
0.1
RosNet50
ImageNet
90
0.1
256
Step'
SGD
le-4
ERK
2000
polynomial
0.1
DeiT huse
ImageNet
200
0.0005
512
Cosine
AdamW
5c-2
Uniform
7000
polynomial
0.5
ResNe150/33
UCF101
Cosine
SGD
16-3
300
polynomial
0.1
MLP
Fushion-MNIST
100
0.025
8
Cosine
SGD
Se-1
ERK
300
polynomial
0.1
FCN-VGG16
Pascal VOC 2012
0.0001
polynomial
SGD
500
Step denotes schedule
learning rate decayed by every epochs
epochs.
For the uniform distribution, the spansity ratio each layer the same (.e. S, 1- (0]1s/((0,lo). where A, the parmmeters in i-th layer. FRK
distribution allocates higher sparsitics to layers with more parameters, while assigning lower sparsitics smaller layers. In ERK, the number of parameters of the sparse convolutional layers is scaled propottionally to 1 and output channels in i-th layer, wy and it, are the width and the height of the kernel.
1h, where "i.-, and are the number of input channels "i- (i-_x (M))))
decayed according a polynomial strategy: px MaP total number training steps. SGD optimizer with momentum 0.9.
The and regrow
where stepaznat the step and step
18

---

Published as a conference paper at ICLR 2025
A.3.2 THE COMPUTATIONAL AND MEMORY CONSUMPTION
In this section, we present the computational cost in terms of Floating-Point Operations (FLOPs) for training and inference, as well as the memory cost in terms of the number of parameters, for different models across different sparsity ratios.
CIFAR-10: For the CIFARIO dataset, we train VGG16 for 160 epochs. The soft memory bound for MEST is set to 10% of the target density. For example, the sparse network's density 0.3, then the soft memory bound is 0.03, resulting in an initial model density of 0.33. This soft memory bound is decayed to 0 using a polynomial strategy. The initial density of GraNet set at 0.8. Every 500 training steps, it prunes weights to achieve density described by the formula: d; (d; de) nAt ).² where d, is the initial density, de is the target density, and nAt is set to half of the total training time.
Table 4: Training FLOPs (TrFLOPs, x10')), Inference FLOPs (InFLOPs, x109), and the number of model parameters (Param, x x10%) for VGG16 CIFARIO at various sparsity ratios (S) using different DST algorithms.
TRPLOPS (x10"))
InFT IIFLOP (x.0))
Param (x10")
5
0.7
0.6
0.3
0.0
0.7
0.6
0.5
0.4
0.3
0.0
0.7
0.6
0.5
0.4
0.3
0.0
Dense
1.51
6,30
15.25
SET RigL
0.46
0.76
091
1.06
1.92
2.55
3.17
3.80
4.42
4,95
6.43
7.89
9.37
10.81
MEST, MEST,
6.43
7.89
GraNet, GriNet
0.68
0.73
0.5
0.97
1.09
192
2.55
3.17
3.80
4.43
4.95
6.43
7.89
9.37
1081
CIFAR-100: For the CIFAR100 dataset, we train ResNet34 for 160 epochs. The soft memory bound for MEST is set to 10% of the target density. For example, if the sparse network's density is 0.3, then the soft memory bound is 0.03, resulting in an initial model density of 0.33. This soft memory bound decayed 0 using polynomial strategy. The initial density of GraNet set at 0.8. Every 500 training steps, it prunes weights to achieve a density described by the formula:
d, - (d, de) (1 t-to>> where d; is the initial density, de is the target density, and nAL set half of the total training time.
Table 5: Training FLOPs (TiFLOPs, 1 Inference FLOPs (InFLOPs, x10*), and the number of model parameters (Param, x x10%) for ResNet34 on CIFAR 100 at various sparsity ratios (S) using different DST algorithms.
OrFLOPs (x10"))
InFLOPs (xc)))
Param (x110)
0.5
SET/RigL.
1.68
2.24
2.79
3.35
3.90
0.67
0.93
1.18
1.40
1.62
6.45
8.57
10.30
12.82
14.95
MEST. MEST,
1.85
2.46
3,07
3.68
4.29
0.67
0.93
L.8
1,40
1.62
6.45
8.57
10.70
12.82
14.95
GriNet, GraNer,
2,51
2.68
3,13
3,63
4.02
0.67
0.93
1.18
144
1.62
6.45
8.57
10.70
12.82
14.95
Tiny Iinylma::Net: For the TinylmageNet dataset, we train EfficientNet-BO for 100 epochs. The soft memory bound for MEST is set to 10% the target density. For example, if the sparse network's 's density 0.3, then the soft memory bound is 0.03, resulting in initial model density of 0.33. This soft memory bound is decayed to 0 using a polynomial strategy. The initial density of GraNet is set 0.8. Every 500 training steps, it prunes weights to achieve density described by the formula: d, - (d, - d)) (1- t-to 3 where d, the initial density, de is the target density, and nAt set to half of the total training time.
nát
Table 6: Training FLOPs (TrLLOPs, x x101)), Inference FLOPs (InFLOPs, x x10*), and the number of model parameters (Param, x x10") for EfficientNet-BO on TinylmageNet at various sparsity ratios (S) using different DST algorithms.
TIFLOPs (x10%%)
InFLOPs (x10P)
Param (x110))
0.7
0.6
0.5
0.4
0.3
0.0
0.7
0.6
0.5
0.4
0.3
0.0
0.7
0.6
0.5
0.4
0.3
0.0
Dense
1.98
6.63
4.26
Rigl.
0.66
/ /MEST,
0.72
0.92
1.13
1.34
1.55
2.21
2.85
3.48
4.12
4.75
1.89
2.29
2.68
2.68
3.08
GraNet, / /GrNNe
0.89
1.02
1.15
1.32
1.46
2,21
2,85
3.48
4.12
4.75
1.89
2.29
2.68
2,68
3.08
19

---

Published as a conference paper at ICLR 2025
ImageNet: For the ImageNet dataset, we train ResNet50 for 90 epochs. The soft memory bound for MEST is set to 0.2, and the sparse network's 's density is 0.9, then the soft memory bound is 0.92, resulting in an initial model density of 0.92. This soft memory bound is decayed to 0 using polynomial strategy. The initial density of GraNet is set at 0.95. Every 2000 training steps, it prunes
weights achieve a density deseribed by the formula: d, (d. de) (1 - t ) where d, is the initial density, de is the target density, and nAt is set to half of the total training time.
Table 7: Training FLOPs (TrFLOPs, x101)), Inference FLOPs (InFLOPs, x10*), and the number of model parameters (Param, x10%) for ResNet50 on ImageNet at sparsity ratios (S = 0.1) using different DST algorithms.
TrFLOPs (x110*)
InfL.OFs (x1l')
Param (x (x1)))
Init density -+ Final dencity
Dense
2.28
$.21
25.56
1.0
Righ
2.17
7.80
23,00
0.9 -a 119
MEST,
2.18
7,80
33.00
0.92 - 0.9
GmNNr
2.70
7,80
33.00
0.95 0.9
The Resource Consumption and Robustness. Even if is outside of the scope of this paper, in the traditional DST style, we provide an overview of the relationship between resource efficiency and the model's robustness against common corruption. Figure g showcases the computational cost (i.e. training FLOPs and model sizes) and robustness of dense and dynamic sparse models on TinylmageNet-C (results for other datasets can be found in the above.
We can find that the dense model, with the highest resource usage (e.g., FLOPs and parameter count), does not necessar- ily lead to better robustness. In the end, we also investigate other training paradigms: iterative dense and sparse training, represented by AC/DC Peste et al. (202I) algorithm. AC/DC begins with dense warm-up, then starts alternating dense and
Reburron Accenoo
16.2
bog
16.0
15.8
sparse training, returning accurate sparse-dense model pairs at the end of the training process. We find that AC/DC also
15.4
exhibits decent robustness, as in Figure 9 However, in most cases, AC/DC tends to require more training FLOPs compared with DST methods.
15.2
0.75
1.00
1.25
1.50
1.75
2.00
Training
Figure 9: The comparison of train- ing FLOPs, parameter count (rep- resented by the of each cir- cle), and robustness accuracy of the dense and sparse EfficientNet- BO on TinyImageNet. Each point represents the result for specific sparsity ratio.
It is worth noting that the computational costs this work are theoretical and not fully realized gains on current hardware in- frastructures. This a common limitation the literature on sparse training methods, as most current hardware and soft- ware systems are optimized for dense matrix operations rather than sparse ones. However, recent advancements in model sparsity are increasingly aligning with hardware and software developments to fully leverage the benefits of sparsity. For ex- ample, NVIDIA's A100 GPU supports 2:4 sparsity Zhou et al. (2021), I, and other innovations are making strides toward efficient sparse implementations Chen al. (2019); Ashby et al. (2019). Si- multaneously, software libraries such as Liu et al.) (2021b); Mocanu et al. (2018) are emerging to enable truly sparse network implementations. These advancements are paving the way for future deep neural networks to achieve greater efficiency in terms of computation, memory, and energy use.
A.4 THE ROBUSTNESS ACCURACY FOR DIFFERENT CORRUPTION TYPES AND SERVERITIES
and 15 showease the relative gains in robustness each of these severity levels. We observe that dynamic sparse models exhibit more pronounced advantage in robustness over dense models, particularly under high severity levels of high-frequency corruption.
Figure
Defined as (Accs. Accsense) Acca,, where Accs, and Acca. represent the accuracies of dynamic sparse and dense models, respectively, under severity level
20

---

Published as a conference paper at ICLR 2025
0.2
0.3
0.
0.0
0.
ind
04 MSST,
Grraee.
Figure 10: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) SET, (b) MEST, (c) GraNet, under different severities of CIFARIO-C.
0.3
04
rdlate shtrnnnh gan
62
0:3
0.2
0.2
-01-
brigdement
mostel cereel cresee
iryallic motle onter and croset
lovg corry
elsute
161 Righ.
[4]
GraNet,
Figure 11: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) RigL, (b) MEST g, (c) GraNet, under different severities of CIFARIO-C.
0:0
0.00
-305
10
incuiit cute aolld
(l MIST,
Figure 12: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) SET, (b) MEST, (c) GraNet, under different severities of CIFAR100-
325
020
adamn mawoereeeeeooo gin
2.55
0.1
3.0
-3.6
grout sistom
RigL
MEST,
GraNes,
Figure 13: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) RigL, (b) MEST, (c) GraNet, under different severities of CIFAR100-C
21

---

Published as a conference paper at ICLR 2025
06
pecertit yould geerin
in postin gane
hightne
pooe eostred
lnd gourt potic
AmBST,
fo GraNer.
Figure 14: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) SET, (b) MEST,. (c) GraNet, under different severities of TinylmageNet-C.
0.5
320
5:35
230
308
2.0
0.0
gong
(x) Righ.
MEST,
(e) GraNit,
Figure 15: Compared with the dense model, the relative robustness gain of different dynamic sparse models with sparsity 0.5 trained by (a) Rigl., (b) MEST g. (c) GraNet, under different severities of TinylmageNet-C.
A.5 ANALYSIS THROUGH THE LENS OF FILTER
Figure 16 and showcases the non-zero weight counts and the sum of weight magnitudes (i.e., absolute values) for each kernel in a specific layer of dynamic sparse VGG16 and ResNet34 trained on CIFAR1O and CIFAR100, respectively.
Additionally, we investigated the accuracy of test data subject to high-frequency or cowffrequency information attenuation during dense or dynamic sparse training. In Figure 18 the results indi- cate that at the beginning of training, the superiority of dynamic sparse models in handling high- frequency information removal starts to appear, becoming more evident as training progresses. Considering the phenomenon of sparse weight concentration the filter after training, it becomes evident that dynamic sparse training dynamically allocates computational resources to prioritize rel- evant low- frequency features during the learning process.
A.6 DSCR HYPOTHESIS IN RECENT DST ALGORITHMS
In this paper, we primarily investigate the robustness of fundamental DST mod- els initialized from random sparse networks with the basic removal and regrowth criteria, to validate our DSCR hypothesis in commonly used settings. With the emergence of other recent and promising DST methods, we also include an initial exploration to further validate our hypothesis these methods, such as Habb soft training (CHTs) Zhang et al. (2024b a), which employ more complex growth criteria and topology initialization. We compare dense training with CHTs using Correlated Sparse Topological Initialization (CHTs+CSTD) and bipartite small- world networks (CHTs+BSW) on robustness accuracy, using the Fashion-MNi corrupted dataset with MLP model, as shown in Table g More training details are provided in Appendix A.3.
Table Robustness accuracy (%) of MLP on Fashion MNIST-C.
Dense Training
Dynamic Sparse Training
Sparsity
SET
RigL
CHTsACSTI CHTs+BSW
97%
23.44 : 1.79
25.23
1.39
24.50 1
1.03
26.64
1.06
19.82 1 0.32
95%
25.55
0.32
2503
LIT 27.38
0.25 27.3 F1.33
22

---

Published as a conference paper at ICLR 2025
feoturc.20
Chepee Charrels
Cuppu Charnels
Outet Cherses
Ou
200
690
200
200
20C
200
2013
Trput Charnes
CHnpue Cheree
Carpue Creree
Cheree
200
215
0.00
Iepul 250
Inout Channed
Ieput Channed
Vouu Charnils
Layer Teature.30
Coppu Crateek
-00
600
200
200
620
6.0
400
0.15
6.35
0.15
Oreenee
200
300
Inps: Charten
Ieput Chanseh
Incal Chansit
Inpur ChenneR
Layer feature.
Outea Currrh
200
100
2013
Irpuc
T:put Charnel:
Tpput Chomsess
5.15
0.53
Outpst Charmes
100
200
+00
200
6.00
Iepuu Chassen
200
400
Inpu Charves
6 Dane
(N SEE
(4) MIST,
60) GraNee
Figure 16: Visualizing non-zero weight count (Ist row) and the sum of weight magnitudes (2nd row) in filters of VGG16 (#layer feature.20, #layer feature.30 and #layer feature. 40 with kernel size 3 x 3) after training on CIFARIO using different DST algorithms: (b) SET, (c) RigL, (d) MEST, and (e) GraNet,. compared with (a) dense counterpart. Each point represents the corresponding value of filter.
The results indicate that the recent DST method (i.e., CHTs) achieves even more inspiring robustness performance, further validating our DSCR hypothesis on the robustness of DST.
A.7 HOW DATA AUGMENTATION INTERACTS WITH DST?
We extend our analysis to explore how data augmentation (c.g.. Mixup Zhang et al. (2018) interacts with dynamic sparse train- ing in comparison to dense training. From Table 9 we observe: In general, data augmentation helps improve model generalization by providing increased input variability. For DST, combining data augmentation with dynamic sparse training can lead to even greater robustness compared to dense training, suggesting that dynamic sparse training (e.g., SET) interacts with data augumentation in a complementary way.
Table 9: Robustness accuracy (%) on CIFAR100 with and
without Mixup.
mixup Dense training
SET
$1.68
52.60
54.75
54.93
23

---

Published as a conference paper at ICLR 2025
layer2.0.comv2
Di Charness
30
100
100
100
100
100
50
175
103
50
1000
160
DN Charnes
-0.
OeEEA Charees
50
100
100
100
100
100
50
103
50
103
50
120
300
leyer2.1.oom 1ccov1
Cutruc Chanped
20
60
50
300
100
50
100
Charnelt
Channeit
Ouree Dutre Chansee
50
30
20
30
300
170
0.0
90
505
20
100
0.0
10
1.00
0.0
1A0
layer3.3.comm conv2
Olttdd Charrel
250
Imput Charmela
Trput Channes
Inpus Channs
Impat Charrila
1sp4t Charnela
8.3
0.5
0.3
03
Outood Oxprebb
Qurque chanse
850
130
150
250
Ingar crannec
irput (nannec
Inpar: Crownse
Impar Crannet
1spar Crannic
(a) Dense
(b) SET
(c) RigL
(d) MEST,
(e) GraNet,
Figure 17: Visualizing non-zero weight count (Ist row) and the sum of weight magnitudes (2nd row) in filters of ResNet34 #elayer2.0.com. #layer2.1 1coovr2.1.coovl .convl and #layerlayer3 c.nov23 with kernel size x 3) after training on CIFAR 100 using different DST algorithms: (b) SET, (c) RigL, (d) MEST, and (e) GraNet,, compared with (a) dense counterpart. Each point represents the corresponding value of filter.
A.8 THE PERFORMANCE OF DST ON -UO-OF-DOMAIN (OOD) TEST DATA
ImageNet-R et al. with 30,000 images of art, cartoons, and other 14 renditions from 200 ImageNet classes, presents a notable domain shift from the original dataset. We report the average test accuracy on ImageNet-R for dense and DST models trained on the original ImageNet.
ImageNet-v2 Recht al.) is valuable dataset for evaluating the generalization of models trained on orig- inal ImageNet dataset. ImageNet-v2 contains three test subsets: Top-images, Threshold-0.7, and Matched- frequency. Each subset comprises ten images per class, selected from a pool of candidates according to various selection frequencies.
Table 10: Average classification accu- racy (%) ImageNet-R dataset. The bold highlights result that are better than the dense model.
Dense RigL. MEST, g GraNet,
37.84
40.04
38.33
38.46
24

---

Published as a conference paper at ICLR 2025
16
04
04
Attenuation
02
00.0
-0.10
299 883688
22
0.10
Attenuation
32
36
03
63
0.03
68
61
188 3888
09 0 R
08 R
Epoch
Epoch
Epoch
(b)
(c)
Figure 18: The impact of high- -reeueeyyy (top) and low low-frequency attenuation (bottom) on Tiny Im- ageNet during training. Test accuracy for (a) dense model, (b) dynamic sparse model from SET. (c) The accuracy difference between sparse and dense models, where positive value indicates that the sparse models perform better.
As shown in Table 10 and [1]] the results from ImageNet-R and ImageNet-v2 correspond to the evaluation domain adaptation and gen- eralization, respectively. The results show that DST models reliably surpass dense models on ImageNet-R and exhibit superior performance in the majority of scenarios on ImageNet-v2. This suggests that models trained with DST not only handle common data corruptions ef- fectively but also excel in the face of significant domain shifts.
Table 11: ImageNet-v2 dataset. The bold highlights result that are better than the dense model.
Average robustness accuracy (%)
Dense
RigL
MEST,
GraNet,
Top-images
77.80
77.88
78.33
78.21
0.reshold-0.7
72.93
12.81
73,23
73.58
Matched Matched-frequency
64.62
64.23
64.14
64.60
Avg. accuracy
71.79
71.64
71.90
72.13
A.9 THE PERFORMANCE OF DST ON SEGMENTATION TASK
We compared the performance of a dense trained fully convolutional network (FCN) with VGGI6 backbone on the Pascal VOC 2012 Everingham et al. clean dataset, then test on corrupted Pascal VOC 2012 (generated following the ImageCorruption library which includes 19 types of corrup- tion at 5 severity levels). The results are presented in Table 12
Unlike classification tasks CIFAR ImageNet, where the sparse topology is randomly initialized, first prune the backbone to obtain the sparse model, then perform sparse fine-tuning on the train- ing set of the Pascal VOC 2012 clean dataset, and fi- nally test it on both the clean and corrupted versions
Table 12: Clean metric (%) on Pascal VOC 2012 and Robustness metric (%) on cor- rupted Pascal VOC 2012.
Menic
Desse training
SET (ow012)
SET (s-0.1)
clean PiAAc (?)
35.65
85.63
83.00
clean mlol (T)
16.31
46.99
47.10
robusiness pis
7630
77.30
77.10
of the Pascal VOC 2012 test set, as the VGG16 back- bone for the segmentor is typically pretrained on ImageNet. The results show that DST models are not only robust in image and video classification but also in image segmentation tasks.
22.15
2309
A.10 GRAD-CAM
Grad-CAM (Gradient-weighted Class Activation Mapping) [eelarajuu et al. (2017) generates heatmap highlighting important regions by weighting the feature maps based on the gradient val- ues. provides practitioners with visual indication of the areas on which the network focuses when making predictions. We visualize and compare the Grad-CAM outputs of both dense models and dynamic sparse models, as shown Figure 19 We present different cases, including clean background ((a) and (b)), and complex background (c). Interestingly, we observe that the highlight- ing region of dynamic sparse models is smaller than that the dense model. In Figure 19 (a), the dynamic sparse models focus the face of the cat for prediction, while the dense models pay atten- tion to both the face and body of the cat. In Figure T9 (b), the dynamic sparse models specifically
25

---

Published as a conference paper at ICLR 2025
emphasize the wings of the eagle. In contrast, the dense models primarily concentrate single wing while also capturing irrelevant contexts, leading to larger attention regions. In the case more complex background (In Figure I9 (c)), the dynamic sparse models tend to allocate some at- tention to the surrounding elements. However, the attention regions are still relatively smaller and primarily focused the face of the bear, especially in the sparse model sparsified by SET.
These visualizations provide further support for our hypothesis that dynamic sparse models effec- tively allocate limited resources to the most crucial features. As result, dynamic sparse models can be robust against image common corruptions, making them a promising approach for addressing real-world challenges.
B IMPACT STATEMENT
In an era dominated by perrpaaaaetteried models and the pervasive presence image corruption, the design of resource-awar and robust Al models is of increasing importance. Gaining insight into the robust behavior of dynamic sparse models against image corruption paves the way for em- bracing these environmentally friendly models in challenging environments. This holds significant implications across various domains, including medical diagnosis, robotics, autonomous vehicles, and other AI applications. Furthermore, our insights into the inner workings of sparse models help us understand the reasoning behind their robust decisions. Overall, we advance our fundamental understanding of dynamic sparse training and provide future perspectives for scalable, efficient, and trustworthy AL. We do not anticipate any negative socictal impacts resulting from this research.
26

---

Published as a conference paper at ICLR 2025
brightness
contrast
fog
gaussian noise
motion blur
pixelate
shot noise
snow
Input
Dense
SET
RigL
(a)
Input
Dense
SET
RigL
(b)
Input
Dense
SET
RigL
(c)
Figure 19: Grad-CAM for TinylmageNet-C. The images corrupted by different corruptions with clean background (a) and (b), and complex background (c).
27


Reviews:
[{"review_id": "YnytoFaGxi", "reviewer": "Reviewer_MR2K", "summary": "The following paper explores the effect of conducting Dynamic Sparse Training (DST) within a neural network towards its robustness to adversarial samples. First, the paper introduces the Dynamic Sparsity Corruption Hypothesis, which postulates that performing DST on artificial neural networks at a low sparsity regime improves its robustness against image and video corruption tasks compared to dense training. Next, The paper proceeds to validate this hypothesis by conducting experiments on various DST algorithms under different deep learning architectures, data modalities, and data corruption scenarios and their severity and comparing their robustness and accuracy with dense training. Then, the authors analyze spectral and spatial domains to show that DST works as an implicit regularizer for neural networks by reducing its attention to high-frequency components or less meaningful features.", "questions": "1. In Figure 3 within the main paper, what do these five bars represent in each corruption type? Please indicate this in a legend within one of the graphs for clarity.\n2. Regarding analyses conducted in Section 5.1 and Section 5.2, what are the motivations for using the random regrowth procedure for both MEST and GraNet instead of their gradient-based regrowth variants for comparison in Figure 5 and Figure 7? I am asking because the results displayed on the paper (Figure 3, Figure 4, Table 1, and Table 3) use the latter instead of the former.\n3. What causes each DST method to achieve worse performance in a particular type of corruption (i.e., single_freq (single frequency greyscale) in ImageNet-$\\bar{\\mathbf{C}}$)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Writing is easy to follow, and the content organization is clear, making the paper easy to read and comprehend.\n2. A comprehensive set of evaluations is conducted to show Dynamic Sparse Training (DST) achieves better performance in terms of robustness accuracy than its dense counterparts.\n3. While this paper does not propose a new method that improves state-of-the-art performance at a specific benchmark, it does provide new insights into the unexpected benefit of performing DST on image and video corruption tasks in favor of dense training on varying neural network architectures.", "weaknesses": "I hope the authors can provide revision for the paper based on the following points:\n\n1. There is no detailed explanation for several experiments conducted in the main paper. Such details are as follows: \n- The experiment was conducted on the ImageNet-3DCC dataset in Section 4.3. One needs to find Table 3 at the end of page 10 to see the model's sparsity ratio employed for the experiment. Even then, there is no detailed explanation of how to train the model for experiments at both the ImageNet-3DCC dataset and the UCF101 dataset for video representation.\n- There are lack of information regarding the details of the experiment conducted in Section 6.1. It only briefly mentions that it uses an MLP model without explicitly mentioning its detailed architecture. In addition, I fail to find details/recipes on how you train the model to achieve results in Table 2.\n\n2. I also don't get why the sparsity ratio differs for experiments in different Sections (namely Section 4.1 and the rest of the subsections in Section 4). Namely, in section 4.1, the authors are trying to evaluate the robustness accuracy in various sparsity ratios ranging from 0.3 to 0.7, whereas the rest of the experiments are conducted at a low sparsity regime of 0.1. I also want to ask the same question regarding the motivation in setting the initial density of GraNet to be 0.8 and the motivation for setting the soft memory bound for MEST to 10% of the target density. I hope the author can address the reasoning for doing so in their rebuttal.\n\n3. The main claims that this paper is trying to address are a bit problematic for me. For example, the DSCR hypothesis mentions a \"low sparsity regime\", but the authors fail to quantify which range of values would be suitable to be defined as a \"low\" sparsity regime and one that is not in principle. This issue, I think, connects with issue 2 pointed out above, which makes the sparsity constraint $S$ for different experiments about different datasets different.\n\n4. (Minor) These are things that might need to be revised in the paper and the appendix:\n- \"Y-axis in Figure 3\" should be \"x-axis in Figure 3\". [Section 4.2]\n- **ImageNet-C** -> \"generated from Tiny-ImageNet\" should be \"generated from ImageNet\" [Appendix A.2]\n- **CIFAR-100** -> ReNet-34 should be ResNet-34 [Appendix A.3.2]\n- \"(c)\" should be \"(a)\" [Figure 7 Right-Top] (I think it's better to put (a), (b), and (c) below the bottom figure and then explain the right figure without mentioning the top and bottom part so that (d), (e), and (f) don't get wasted there).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ThFLt8R9cD", "reviewer": "Reviewer_EZM6", "summary": "This paper explores whether Dynamic Sparse Training (DST) results in more OOD Robust Models than Dense Training.\nTo this extent, the paper reports its findings using various DST methods over multiple architectures and different datasets.\nThe findings are consistent across experiments that DST methods lead to more OOD robust models than Dense Training.", "questions": "**Q1- Section 3.1**\n\nI fail to understand the significance of $p_{tr}(y|x)=p_{te}(y|x)$, is it supposed to mean that training and test samples are iid? If yes, then why is mentioned explicitly? What is the significance of explicitly mentioning this detail?\n\n**Q2 - Appendix A.3.2**\n\nTo the best of my current understanding, the FLOPs calculated are mere estimations, right? These are not real gains in FLOPs that are materialized by the hardware infrastructure of today? If so, then please mention that in Appendix A.3.2, otherwise, it misleads the reader.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The structure of the paper is quite different from usual submissions and is a welcome change, the style of having research questions for section headings and related experiments, findings, and discussion in those sections is interesting and keeps the reader engaged.", "weaknesses": "**W1- Lack of Proper Literature Review.**\nThe biggest weakness of this work is that it only analyzes its findings from the narrow lens of \"Dynamic Sparse Training\" because almost all of the questions asked in this work have been asked before from a larger context of Deep Neural Networks and Sparse Deep Neural Networks and have been successfully answered. \nThe related work section of this paper is very poorly written and researched.\nHad significant effort been invested into finding true related works done by the community this paper would have been significantly different. \nFor example, in section 5.1 the paper attempts to find a correlation between inherent properties of dynamic sparse training and increased OOD robustness, however in their work [1] already showed that sparse models can be very OOD Robust, and while this proposed work on an average looks only up to 50% sparsity, [1] showed that 90% sparse models can be more robust than their dense counterparts. While [1] does not specifically look into DST, the overall observations from [1] and this work are not very dissimilar, and thus maybe there is more at play here than \"inherent properties of DST\", that is these observations are not unique to DST and have been explored before and shown that models can be trained to have **\"reduced connectivity to specific features that are determined to be less crucial\"**.\n\nAnother example is in section 5.2 the paper does an analysis using a high-pass filter and a low-pass filter. Here the conclusion in the paper is that low-frequency information is more important, high-frequency information is less important and DST methods learn some kind of inherent regularisation to focus on \"more information i.e. low frequencies\" and thus are more robust. \nHowever, the claim that \"low-frequency information\" is \"more important\" information and \"high-frequency information\" is \"less important\" has not been supported by studies from this paper or from any other paper cited here.\nThis is essential because another exists to explain the observations that have nothing to do with \"less important\" and \"more important\" but finds its basis in signal theory and is therefore more plausible. This explanation is provided by [2] and [3] that show using only \"low-frequency information\" one can avoid aliasing and avoiding aliasing leads to a reduction in spectral artifacts which leads to increased OOD robustness. So, maybe DST is inherently leading to less aliasing due to the sparsity and is therefore increasing OOD Robustness. \nTherefore, despite the experiments in the paper, the conclusions presented in the paper do not hold without doubt as other plausible explanations exist.\n\nW2 - **Poorly Written.**\nWhile the structure of the paper is quite good, the content is very poorly written and has many mistakes. Just to prove the point I will point out a few, so that similar mistakes can be avoided in the future. \n\n1. Figure 7 subcaptions are wrong, (a) has been written as (c). \n2. Math variables are not defined, example before eqn 1, it is written M $\\in \\mathcal{M}$, however $\\mathcal{M}$ has not been defined. And in Preliminaries 3.1, $p_{tr}$ and $p_{te}$ are decomposed, but no reason has been given for it, a reason should be provided for doing an operation such as decomposition, moreover there exists $i$=1 in subscripts but $i$ is never defined.\n\n\nW3- **Missing Important citations**\nWhile I have covered some of the missing important related works in W1, it is paramount to note the missing citation for [4]. The frequency properties of common corruptions are referenced multiple times in this paper however these properties are not general knowledge but findings from [4] and they must be cited for their work.\n\nW4 - **Outdated Data Augmentation techniques used in Appendix A.6**\nMixUp is an old data augmentation technique, significantly newer data augmentation techniques exist now. Some of the most common ones while not very new are AugMix and DeepAugment [5], it might be prudent to rather (or additionally) ablate with these in in Appendix A.6.\n\n\n**References** \n\n[1] Hoffmann, Jasper, et al. \"Towards improving robustness of compressed cnns.\" ICML Workshop on Uncertainty and Robustness in Deep Learning (UDL). Vol. 2. 2021.\n\n[2] Grabinski, Julia, et al. \"Frequencylowcut pooling-plug and play against catastrophic overfitting.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Grabinski, Julia, Janis Keuper, and Margret Keuper. \"Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling.\" arXiv preprint arXiv:2307.09804 (2023).\n\n[4] Yin, Dong, et al. \"A Fourier perspective on model robustness in computer vision.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[5] Hendrycks, Dan, et al. \"The many faces of robustness: A critical analysis of out-of-distribution generalization.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yNJnh8z4V", "reviewer": "Reviewer_ncJc", "summary": "This paper raises an intuitive research question through a thorough review of dense training, static sparse training, dynamic sparse training, and their counterpart model robustness, \"Is somehow DST implicitly improving model robustness\". Then, propose the Dynamic Sparsity Corruption Robustness (DSCR) Hypothesis \"Dynamic Sparse Training at low sparsity levels helps to improve model robustness against image corruption in comparison with Dense Training.\" Finally, it validates the DSCR Hypothesis on two types of data, images, and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. The findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art. I think this is a good paper and worth accepting.", "questions": "same as the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Problem statement. This paper clearly narrates the robustness from dense training, static sparse training to dynamic sparse training.\n2. Necessary and sufficient experiments. From convolution-based network to transformer-based network, this paper makes a thoroughly empirical experiment to justify the robustness of dynamic sparse training.\n3. Besides the above, section 5 explores the inner reasons for the robustness of dynamic sparse training models. Analyzing how DST and Dense training models utilize their parameters in the spatial domain and process the input in the spectral domain.", "weaknesses": "The research direction of robustness only focus on the classification task (with dataset cifar10/100-c, ImageNet-c), why not extend to new tasks/datasets like segmentation, pose estimation, and image matching et al?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2NCT1tdUx", "reviewer": "Reviewer_Khz6", "summary": "This paper challenges the conventional use of dense training for model robustness in deep learning. Through extensive experiments across multiple datasets, architectures, and Dynamic Sparse Training (DST) methods, the authors demonstrate that DST can consistently outperform dense training in image corruption robustness. Analysis reveals that DST's superior performance stems from its implicit regularization mechanism, which naturally focuses on low-frequency features while reducing sensitivity to high-frequency components. The findings are validated across both image and video tasks, suggesting new directions for improving neural network robustness.", "questions": "More experiments and comparisons with SOTA training methods or robust models would be beneficial.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental evaluation is thorough and well-structured, covering both image and video domains, multiple architectures (CNNs and Transformers), and various corruption types and severity levels.", "weaknesses": "1.\tThe core contribution appears to be an empirical observation, while the technical contributions are limited.\n2.\tWhile the results show improved robustness against corruption, the paper doesn't extensively analyze whether this comes at the cost of other aspects of model performance (e.g., clean accuracy).\n3.\tThere is a clear error in describing Figure 3's axis. The text states \"the Y-axis in Figure 3 represents different types of corruption\" when it appears to be the X-axis that shows corruption types.\n4.\tThe considered datasets to evaluate the proposed method are limited. There are a lot of robustness benchmarks, including ImageNet-A/R/P. It is unclear whether the proposed method is able to consistently improve robustness across diverse benchmarks.\n5.\tIt would be interesting to compare with SOTA robust training methods, rather than only comparing with the standard dense training.\n6.\tThe comparisons against SOTA robust training methods and robust models should be included, including RVT and FAN.\n7.\tThe comparison of training cost should be reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)