Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training
Authors: Geng Yuan
Yanyu Li
Sheng Li
Zhenglun Kong
Sergey Tulyakov
Xulong Tang
Yanzhi Wang
Jian Ren
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sparse training
model compression
efficient training

Abstract: Recently, sparse training has emerged as a promising paradigm for efficient deep learning on edge devices. The current research mainly devotes the efforts to reducing training costs by further increasing model sparsity. However, increasing sparsity is not always ideal since it will inevitably introduce severe accuracy degradation at an extremely high sparsity level. This paper intends to explore other possible directions to effectively and efficiently reduce sparse training costs while preserving accuracy. To this end, we investigate two techniques, namely, layer freezing and data sieving. First, the layer freezing approach has shown its success in dense model training and fine-tuning, yet it has never been adopted in the sparse training domain. Nevertheless, the unique characteristics of sparse training may hinder the incorporation of layer freezing techniques. Therefore, we analyze the feasibility and potentiality of using the layer freezing technique in sparse training and find it has the potential to save considerable training costs. Second, we propose a data sieving method for dataset-efficient training, which further reduces training costs by ensuring only a partial dataset is used throughout the entire training process. We show that both techniques can be well incorporated into the sparse training algorithm to form a generic framework, which we dub SpFDE. Our extensive experiments demonstrate that SpFDE can significantly reduce training costs while preserving accuracy from three dimensions: weight sparsity, layer freezing, and dataset sieving. Our code and models will be released.

Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training
Geng Yuan 1.2.f Yanyu Li Sheng Li Zhenglun Kong", Sergey Tulyakov',
Xulong Tang', Yanzhi Wang", Jian Ren
Snap Inc., NNrtheesttern University, Universitty of Pittsburgh
yuan. www.gggoottthatteemmmmmmm jrenGsnapchat.com
Abstract
Recently, sparse training has emerged as a promising paradigm for efficient deep learning edge devices. The current research mainly devotes the efforts reducing training costs by further increasing model sparsity. However, increasing sparsity is not always ideal since it will inevitably introduce severe accuracy degradation at an extremely high sparsity level. This paper intends to explore other possible directions to effectively and efficiently reduce sparse training costs while preserving accuracy. To this end, we investigate two techniques, namely, layer freezing and data sieving. First, the layer freezing approach has shown its success in dense model training and fine-tuning, yet it has never been adopted in the sparse training domain. Nevertheless, the unique characteristics of sparse training may hinder the incorporation of layer freezing techniques. Therefore, we analyze the feasibility and potentiality of using the layer freezing technique in sparse training and find it has the potential to save considerable training costs. Second, we propose data sieving method for dataset-efficien training, which further reduces training costs by ensuring only partial dataset is used throughout the entire training process. We show that both techniques can be well incorporated into the sparse training algorithm to form a generic framework, which we dub SpFDE. Our extensive experiments demonstrate that SpFDE can significantly reduce training costs while preserving accuracy from three dimensions: weight sparsity, layer freezing, and dataset sieving")
Introduction
Sparse training, promising solution for efficient training edge devices, has drawn significant attention from both the industry and academia [[] Recent studies have proposed various sparse training algorithms with computation and memory savings to achieve training acceleration. These sparse training approaches can be divided into two main categories. The first category is fixed-mask sparse training methods, aiming find better sparse structure in the initial phase and keep the sparse structure constant throughout the entire training process 2 3 These approaches have a straightforward sparse training process but suffer from higher accuracy degradation. Another category is Dynamic Sparse Training (DST), which usually starts the training from a randomly selected sparse structure 5.. DST methods tend continuously update the sparse structure during the sparse training process while maintaining an overall sparsity ratio for the model. Compared with the fxed-mask sparse training, the ofate-of-the-ar DST methods have shown their superiority in accuracy and recently become more broadly adopted sparse training paradigm [6]
'Theee authors contributed equally.
Code will be available at http:://ssiubb..oo//mapprreee//r
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, although the existing sparse training approaches can reduce meaningful training costs, most of them devote their efforts to studying how to reduce training costs by further increasing sparsity while mitigating accuracy drop. As a result, the community tends to focus on the sparse training performance at an extremely high sparsity ratio, e.g., 95% and 98%. Nevertheless, even the most recent sparse training approaches still lead to severe performance drop at these high sparsity ratios. For instance, on the CIFAR-I0 10 dataset 77. MEST [T] has 2.5% and 4% accuracy drop at 95% and 98% sparsity, respectively. In fact, the network performance usually begins to drop dramatically at the extremely high sparsity, while the actual gains from weight sparsity, i.e., savings of computation and memory, tend to saturate. This indicates that reducing training costs by pushing sparsity towards extreme ratios at the cost of network performance may not always the desirable methodology when a certain sparsity level has been reached. Towards this end, we raise fundamental question that has seldom been asked: Are there other ways that can be seamlessly combined with sparse training to further reduce training costs effectively while maintaining network performance?
To answer the question, we first take a step back to understand whether all layers in sparse networks are equally important. Recent studies reveal that not all the layers in dense Deep Neural Networks (DNNs)
Table 1: The key features of SpFDE com-
pared to representative sparse training works.
need to be trained equally 88 9 [0]. Generally, the
Only
Only sparse
Dataset efficient training
Layer
early layers in DNNs are responsible for extracting
sparse
freeze
low-level features and usually have fewer parameters
weight
BP
SNIP, GraSP
No
No
No
No
than the later layers. These make the early layers have
RigL, ITOP
Yes
No
No
No
higher representational similarity and converge faster during training 8 11. Therefore, layer freezing tech-
SET. DSR
Yes
Yes
No
No
niques are proposed, which stop the training (updating) of specific DNN layers early in the training process to
MEST
Yes
Yes
Partial
No
SpFDE (Ours)
Yes
Yes
Yes
Yes
save the training costs. The early work attempts the layer freezing technique in dense model training (8) while many following works focus on layer freezing in the fine-tuning process of the large transformer based models [12] I0 [4] Even with the progress, the layer freezing technique has never been explored in sparse training.
Layer freezing seems like a promising solution for sparse training that further reduces train- ing costs. However, the conclusion is still too early to draw, given that sparse training has two critical characteristics that make it a unique domain compared with dense DNN training and fine-tuning. This might impede the incorporation of the layer freezing technique in sparse training. D The superiority of the DST method is attributed to its continuously changed sparse structure, which helps it end up with better result (6). This could inherently contradict the layer freezing that requires the layers to be unchangeable early in the training process. @ The impact of the sparsity for each layer is unknown in terms of the convergence speed. These two characteristics directly affect the feasibility and potentiality of using the layer freezing techniques in sparse training.
Back to the question we raise, whether there other directions to effectively reduce sparse training costs besides increasing sparsity. To tackle the question. in this work, we propose two techniques that can be well incorporated with sparse training, namely layer freezing and data sieving.
We explore the feasibility and potentiality of leveraging layer freezing in the sparse training domain by carefully analyzing the structural and representational similarity of sparse models during sparse training. We find the layer freezing technique is suitable for sparse training and has the potential to save considerable training costs (Sec. 3 Based on this, we propose a progressive layer freezing method, which is simple yet effective in saving training costs and preserving accuracy (Sec. 4.2)
Shrinking the size of the training dataset is another possible dimension to reduce training costs. Studies have shown that the importance of each training sample is different during DNN train- ing [16] 17. Toneva et al. (18] distinguish the importance of each training sample by counting the number of times each training sample is forgotten by the network during training. Later, Yuan er al. ① propose dataset efficiency in the sparse training domain with two phases. The first phase uses the whole dataset to count the forgotten times of each sample, and the second phase removes the unimportant samples. Though they prove the feasibility of dataset efficiency training in the sparse domain, it only obtains limited cost savings from its second phase. To fully exploit the potential of dataset efficiency, we propose the data sieving-a a circular sieving method to dynamically update the shrunken training dataset, ensuring high dataset efficiency throughout the entire training.
2

---

Putting it all together, we propose a generic and efficient sparse training framework SpFDE, that achieves significant reduction in the training computational and memory costs through three- dimension: weight Sparsity, layer Freezing, and Dataset Efficiency. The comparison of key features between SpFDE and other representative sparse training works, i.e., SNIP 2 GraSP 33, Rigl. [5], ITOP [6]. SET [9]. DSR 4 and MEST [1, is provided in Tab. I Extensive evaluation results show that the SpFDE framework consistently achieves a significant reduction in training FLOPs and memory costs while preserving higher similar accuracy. Specifically, SpFDE maintains the highest 71.35% accuracy at 90% sparsity and 76.03% accuracy at 80% sparsity on the CIFAR-100 and ImageNet dataset, respectively, and achieves 18% and 29% training FLOPs reduction compared to the most recent methods such as MEST . and IТOP 66. Moreover, SpFDE can further save 20% 25.3% average training memory, and 42.2% 43.9% minimum required training memory compared to the prior sparse training methods.
2 Related Work
Background for Sparsity Training. Sparsity scheme and training strategy are two important components for defining sparse training pipeline from literature.
Three main sparsity schemes introduced the area of network pruning consist of unstructured (20 27 28
22] structured 23
B3 34 35 (3),, and fine-grained structured 46). Though network pruning initially proposed for inference acceleration, it is widely adopted in sparse training to achieve satisfactory trade-offss between network performance, e.g., classification accuracy, memory footprint, and training time. Most of these works follow the training pipeline of generic sparsity training framework that works for edge devices by focusing on sparse networks trained from scratch, instead of training dense networks, which is not feasible resource-limited devices M.
24
25
26
29
30 31 32
pruning 37
38
39
40 41
Instead, we consider
can be categorized into fixed-mask and sparse- mask sparse training, where the former aims to make feasible that the training of the pruned models and the latter studies how to reduce memory 5) The sparsity scheme and training strategy in this work follows MEST D as it does not involve dense computations, making it desirable for edge device scenarios. Unlike previous works on sparse training that mainly reduces computation by increasing the sparsity ratio at the cost of network accuracy, we investigate a new method-layer freezing-in sparse training, which can reduce training costs for an arbitrary sparsity ratio.
Research on the sparse training strategies 47
48 49
can be implemented on edge devices [2] 3 50 51
52
costs along with the computation during training 53
Layer Freezing during Training, The study on accelerating the training of dense neural networks has shown that not all the layers need to be trained equally and the decreasing of training iterations for certain layers can reduce training time with only minimal performance drop observed [5] g 12]
13
56
57
14 58 Liu et al. [10] and He et al. [15 calculate network gradients for automatically
layer freezing during training. Wang et al. [59] use knowledge distillation [60] to guide the layer freeziig schedule. However, these works require the network be static or an extra dense network during the training, which is not eligible for the sparse training scheme. In this work, we show layer freezing can be incorporated into sparse training with detailed analysis. Additionally, we propose a flexible and hybrid layer freezing strategy that can be fitted into sparse training.
3 Analysis of Layer Freezing in Sparse Training
Existing works have shown the success of the layer freezing technique in dense model training, especially for the model fine-tuning, which effectively saves training costs terms of computation and memory and thus accelerates the training [8) 10 [4] [5]. However, sparse training has two critical charucteristics, i.e., dynamically changed architecture and different sparsity per layer, that makes it a unique domain compared to dense DNN training and fine-tuning, which might impede incorporating the layer freezing technique into the sparse training methods. To investigate the best way to leverage layer freezing in sparse training, we need first to answer the following two questions.
3

---

Representatioon similarity (%)
Structural similarit (%5
100
80
40
layr
demse
---
layer 30
50%
80% sparsiyy
100 130 140
10
50
60
70
30
100
110
120
130
140 150
Training Epoch
Training Epoch
(b)
Figure 1: (a) Analysis of structural similarity:difffrenn layers with the same sparsity (90%). (b) Analysis of representational similarity: the same layer (10th) with different sparsity. All results collected using ResNet-32 on the CIFAR-100 dataset during the sparse training process.
3.1 Is Layer Freezing Compatible for Dynamically Changed Network Structure?
The Dynamic Sparse Training (DST) method shows its superior performance by continuously changing its sparse model structure during training search for better sparse architecture, making a desirable sparse training paradigm [5]. However, the dynamically changed network structure seems essentially contradictory to the layer freezing technique, given the weights of frozen layers fixed and not further trained. Thus, these layers cannot keep searching for better sparse structures, which may compromise the quality of the sparse model trained, leading lower model accuracy.
Assumption, Inspired by the convergence speed for various layers is different in conventional dense model training, we conjecture that, in DST, the early layers may also find desired sparse structure faster than the later layers. If true, we may be able to introduce the layer freezing technique in sparse training without compromising the sparse training accuracy.
Experimental Setting. We investigate the assumption by tracking the structural similarity of the sparse model along the sparse training process. Specifically, we select the well-trained sparse model as the reference model and compare the intermediate sparse model obtained each epoch with the reference model. We define structural similarity as the percentage of the common non-zero weight locations, i.e., indices, both the intermediate sparse model and the final sparse model. For instance, the structural similarity of 70% indicates that 30% of the current intermediate sparse structure will be altered during the rest the training process and will not be presented the final model. The 100% structural similarity shows the sparse model structure is fully stabilized.
Analysis Results. Fig. I (a) shows the trend of structural similarity of different layers within model along the same sparse training process. We adopt the DST method from MEST [ with 90% unstructured sparsity and evaluate the results using ResNet-32 on the CIFAR- 100 dataset. Note that we check the structural similarity by choosing the locations of the 50% most significant non-zero weights from the intermediate models. The reason is that sparse training algorithms (e.g.. MEST and Rigl.) may force less important weights/locations to be changed during sparse training, regardless of whether the sparse structure has already been stabilized. Additionally, the most significant weights play the most important role in the model's generalizability. Therefore, tracking the structural similarity using 50% most significant non-zero weights reasonable and meaningful.
Observation. From the results, we can observe that the structural similarity of the first layer converges at the very early stage of the training, i.e., 40 epochs, and the early layers' structural similarity converge faster than the later layers. The structural similarity of sparse training follows a similar pattern as the dense model training. This indicates that the changing/searchiin of the early layers from the sparse models can be stopped earlier without compromising the quality of the final sparse model, providing the feasibility of applying the layer freezing technique in sparse training.
3.2
What Is the Impact of Model Sparsity on the Network Representation Learning Process?
With the above exploration giving evidence that layer freezing is compatible with sparse training (Sec. B.I. another critical question is to understand the impact of model sparsity the neural

---

Inital stage
Active training stage
Progressive layer freezing stage
DNN model
Active layers
Activa layers
Frozen
Active layers
Frozen
Active layers
Frozen
Active
+
Inttialize sparse
Frozen structure and weights
in dynamic sparts toininn
with layer freesing
Falasel every dr opachs
ddbart
edkact
training
Figure 2: Overview of the SpFDE framework. The sparse training process consists of three stages. The initial stage creates a sparse model with a random sparse structure and randomly partitions the training dataset into a partial training dataset and a removed dataset. The active training stage conducts sparse training using the selected sparse training algorithm and periodically updates training dataset via data sieving. The progressive layer freezing stage starts to progressively freeze layers in sequential manner and the frozen layers will not change their sparse structure and weights values.
network representation learning process. In other words, what is the convergence speed for the same layer under different sparsity? If the convergence speed is different under different sparsity, applying layer freezing would be much more complicated since deciding the stop criterion would be very challenging, and the potential gain we can have by using layer freezing could be limited.
Previous work shows that layer freezing can be used to effectively reduce training costs due to the ability of fast representation learning of the early layers in the network [8]. Another work observes that wider network is easier to learn the representation to saturated level [... Therefore, whether the sparsity would significantly slow down the representation learning or the convergence speed of the layers is unknown since the width of the layer be changed during sparse training.
Experimental Setting. To explore the impact of sparsity on network representation learning speed, we adopt the centered kernel alignment (CKA) [I]] as an indicator of representational similarity. We track the trend of the CKA between the final and intermediate model at each training epoch.
Analysis Results and Observation. The results are shown in Fig. (b). We compare the CKA trends of the same layer, i.e., 10th layer in ResNet-32. in dense model training and sparse training under different sparsity. Surprisingly, we find that, in sparse training, even under a high sparsity ratio (i.e., 90%), there is no apparent slowdown in the network representation learning speed. Similar observations can also be found in other layers (more analysis results in Appendix A). This indicates that the layer freezing technique can potentially be adopted in the sparse training process as early as in the dense model training process, thereby effectively reducing the training costs.
Takeaway. Considering the unique characteristics of sparse training, we first explore the feasibility and potentiality of using the layer freezing techniques the sparse training domain. By investigating both the structural similarity and representational similarity of the sparse training, we tentatively conclude that the layer freezing technique is also suitable for sparse training and has the potential to save considerable training costs.
SpFDE Framework
In this section, we introduce our generic sparse training framework, with training costs saved through three dimensions: weight sparsity, layer freezing, and data sieving. Fig, showw the overview of our SpFDE framework, and we introduce more details as follows.
4.1 Overview of Sparse Training Framework
The overall end-to-end training process can logically divided into three stages, including the initial stage, the active training stage, and the progressive layer freezing stage.
5

---

Initial Stage. As the first stage in training, we initialize the sparse network and partial training dataset. The structure of the sparse network is randomly selected. The partial training dataset obtained by randomly removing a given percentage of training samples from the whole training dataset, which differs from prior work that starts with whole training dataset []. We ensure that only parts of the whole training dataset will be used during the entire training process.
Active Training Stage. Following the initial stage, all layers are actively trained (non-frozen) using a sparse training algorithm. We apply DST from MEST [ as the sparse training method due to its superior performance, while other sparse training algorithms are compatible with our framework. We use the proposed data sieving method to update the current partial training dataset during the training periodically (more details in Sec. (43)). Besides the computation and memory savings provided by the sparse training algorithm, our SpFDE can benefit from the data sieving method to further save computation and memory costs. Specifically, the computation costs are reduced by decreasing the number of training iterations in each epoch, and the memory costs are reduced by loading the partial dataset.
Progressive Layer Freezing Stage. At this stage, we begin to progressively freeze the layers in a sequential manner. The sparse structure and weight values of the frozen layers will remain unchanged during the sparse training. The computational and memory costs of all gradients of weights and gradients of activations in the frozen layers can be eliminated. which especially crucial for resource- limited edge devices. More details are provided in Sec. 4.2
4.2 Progressive Layer Freezing
Algorithm 1: Algorithm of SpFDE
Motivated by the observation that the struc- tural and representational similarity of early layers converges faster than later lay- ers in sparse training (Sec. D. we propose the progressive layer freezing approach to gradually freeze layers sequentially. Specif- ically, layer can be frozen only if all the layers in early of this layer are frozen. The progressive manner brings the bene- fits for maximizing the saving of training costs since the entire frozen part of the model does not require computing back- propagation.
Input: Network with randomly initialized weight W under sparsity sp, number of blocks N, target training FLOPss wurget_flpps total training epochs T, starting freeze epoch T Trr,, and DST structure changing frequency Ar.
Output: The final sparse model.
Initialize train__ flops as the total sparse training FLOPs without freezing and put all blocks in the
active_layers for i 0, ON..,,N-1 B generate freeze config do
if train >> target_ flops then lii.jisttuushbbaee
saved_flops BpFlops(blocks) (T - Tirs AT+ + i) train_flops = train_flops saved_
4.2.1 Layer Freezing Algorithm
for epoch o,....,
DST training 1oop
Alg. shows the training flow of SpFDE and the algorithm of progressive layer if (epoch mod A-= -- 0) then
do
if epoch > Trr then
freezing. For a given DNN model with L layers, we divide it into N blocks, with each block consisting of several consecu- tive DNN layers, such as bottleneck block in the ResNet (61]. We denote T as the total training epoch, AT as the sparse struc- ture changing interval of dynamic sparse training, and Tyr (0- < Trr < 7) the epoch that we start the progressive layer freezing stage and freeze the first block. Then, for every AT epochs, we sequentially freeze the next block until the expected overall training FLOPs satisfy the target_flope We consider the frozen blocks still need to conduct forward propagation during training. Therefore, we compute the training FLOPs reduction of freezing a block as its sparse pro-ppropggttion computation FLOPs (calculated by BpFlops(.) in Alg. 1 multiplied by the total frozen epochs of the block. For detailed implementation, given the target training FL.OPs largel_flops and the total number of layers to freeze, we empirically choose to freeze 2/3 layers of the model, and the I fre can be casily calculated.
block = block bidoccklist.ooplf
FreezeLayers (block))) active active_layers. wayerr.rerooeeeeee. (block)
for each layer weight tensor W in active layers do
W' +- PrunekGrox( (eaeiGrrorWWW..... ssp)
Update training dataset
Collect sample status

---

fraining process
35.0%
difference of gradient norm
byee
layers blocks
25.0%
fast decrene
searching stopped
layer
layer
20.0%
fluctuating
layer 25
10.0%
5.0%
single-shot
periodically
0.0%
gradient norm
300
250
150
single-shot & resume
delayed periodically
active epoch
frozen epoch
97
100
115
127
133
139
epoch
(a)
(b)
Figure 3: (a) Illustration for different layer freezing schemes. (b) The trend of layer gradient norm and the difference of layer gradient norm during dynamic sparse training.
To better combine with the DST and make sure the layers/blocks are appropriately trained before being frozen, we synchronize the progressive layer freezing interval to the structure changing interval, i.e., Ar, of the sparse training, and adopt layer/block-wiie cosine learning rate schedule according to the total active training epoch of each layer/block.
4.2.2 Design Principles for Layer Freezing
There are two key principles for deriving a layer freezing algorithm, the freezing scheme and the freezing criterion. Here we discuss the reasons that the proposed progressive layer freezing is rational.
Freezing Scheme. Since sparse training may target the resource- limited edge devices, is desired to have the training method as simple as possible to reduce the system overhead and strictly meet the budget requirements. Therefore, we follow a cost-saving-orts -rriented freezing principle guarantee the target training costs and derive the layer freezing scheme, which can include the single-shol, single-shot resume, periodically freezing, and delayed periodically freezing (as illustrated in Fig. (a)). We adopt the single-shot scheme since it achieves the highest accuracy under the same training FLOPPs saving (detailed results Appendix B). The possible reason that the single-shot freezing scheme has the longest active training epochs the beginning of the training, which helps layers converge to a better sparse structure before freezing.
Freezing Criterion. With the freezing scheme decided, another important question is how to derive the freezing criterion, i.e., choosing which iterations or epochs freeze the layers. Existing works have explored adaptive freezing methods by calculating and collecting the gradients during the fine-tuning of dense networks [10]. However, from our observations, the unique property of sparse training makes these approaches not applicable. For example, as shown in Fig. 3 (b), the difference of gradients norms from different layers decreases very fast at the beginning of the sparse training while keeps fluctuating after some epochs because of the prune-and-grov weights. Abstracting the freezing criterion based on the gradient norm would inevitably introduce extra computation and system complexity since the changing patterns of the gradient norm difference volatile. Therefore, our strategy of combining the layer freezing interval with the DST interval is more favorable.
4.3 Circular Data Sieving
We propose the data-sieving method achieve true dataset-efficient training throughout the sparse training process. As shown in Fig. at the beginning of the training, we randomly remove given portion of total training samples (e.g., 20%) from the training dataset depending on the target training cost reduction ratio and create a partial training dataset and a removed dataset. During the sparse training, for every AT epoch, we update the current partial training dataset by removing the easiest p% of the training sample from the partial training dataset and adding them to the removed dataset. Then, we retrieve the same number of samples from the removed dataset and add them back to the partial training dataset to keep the total number of training samples unchanged.

---

Table 2: Comparison of classification accuracy (on CIFAR-100) and training FLOPs (xe15) between the proposed SpFDE and the most representative sparse training works using ResNet-32.
Method
Spursity
90%
95%
98%
FLOPs (1)
Acc. (f)
FLOPs (L)
Acc. (t)
FLOPs (.)
Acc. (1)
LTH
62
N/A
68.99
N/A
65.02
N/A
57.37
SNIP 2
1.32
68.89
0.66
65.22
0.26
54.81
GraSP
1.32
69.24
0.66
66.50
0.26
58.43
DeepR [53
1.32
66.78
0.66
63.90
0.26
58.47
SET 19
1.32
69.66
0.66
67.41
0.26
62.25
DSR
1.32
69.63
0.66
68.20
0.26
61.24
MEST
1.54
71.30
0.96
70.36
0.38
67.16
1pFDE
1.26
71.35=00228
0.66
70.43++0.22
0.30
67.04:+0.20
SpFDE +20%
1.12
71.25:10.23
0.58
70.14:10.17 17
0.26
66.37:100.4
SpFDE +25%
0.96
71.0220.39
0.52
69.48:10.19
0.24
65.04:10.19
We adopt the number of forgetting times as the criteria to indicate the complexity of each training sample. Specifically, for each training sample, we collect the number of forgetting times by counting the number of transitions from being correctly classified interval. We recollect this number for each interval to ensure the newly added samples can be treated equally. Additionally, we use the structure changing frequency Åt in sparse training as the dataset update frequency to minimize the impact of the changed structure the forgetting times.
being misclassified within each År
We treat the removed dataset as a queue structure, retrieving samples from its head and adding the newly removed sample to its tail. After all the initial removed samples are retrieved, we shuffle the removed dataset after each update, making all the training samples can be used at least once. As result, we can gradually sieve the relatively easier samples out and only use the important samples for dataset-efficient training. More analysis results are in the Appendix C.
5 Experimental Results
In this section, we evaluate our proposed SpFDE framework on benchmark datasets, including CIFAR- 100 (7] and ImageNet (63]. for the image classification task with ResNet-32 and ResNet-50. Note that we follow the previous work [ 2
using the 2x widen version of ResNet-32. We compare the accuracy, training FLOPs, and memory costs of our framework with the most representative
19 3 I at different sparsity ratios. Models are trained by using PyTorch [64] on an 8 100 GPU server. We adopt the standard data augmentation and the momentum SGD optimizer. Layer-wise cosine annealing learning rate schedule is used according to the frozen epochs. To make a fair comparison with the reference works, we also use 160 training epochs on the CIFAR-100 dataset and 150 training epochs on the ImageNet dataset. We choose MEST+EM&S M as our training algorithm for weight sparsity since it does not involve any dense computations, making it desirable for edge device scenarios. We apply uniform unstructured sparsity across all the convolutional layers while only keeping the first layer dense. More experiments on setting are provided in the Appendix D.
sparse training works 12 3 53
54
other datasets and the detailed
5.1 Comparison on Model Accuracy and Training FLOPs
Tab. 2 shows the comparison of accuracy and computation FLOPs results on CIFAR-I00 dataset using ResNet-32. Each accuracy result averaged over 3 runs. We denote the configuration of our SpFDE using v% + y%, where x indicates the target training FLOPs reduction during layer freezing and y is the percentage removed training data. Our SpFDE can consistently achieve higher or similar accuracy compared to the most recent sparse training methods while considerably reducing the training FLOPs. Specifically, at 90% sparsity ratio. SpFDE20% +20% maintains similar accuracy MEST 11 while achieving 27% training FLOPs reduction. When compared with DeepR (53). SET 19 and DSR [4], SpFDE achieves 27% FLOPs reduction and +1.36% ~ +4.24% higher accuracy. More importantly, when comparing SpFDE at 90% sparsity with the MEST at 95% sparsity, we can see the two methods have the same training FL.OPs, i.e., 0.96, while 2AFDE25%+2255 has clear higher accuracy, i.e., +0.66%. This further strengthens motivation
8

---

900
ory of DST methods and the average and mini- Table 3: Comparison results on ImageNet using
Memory (MB)
800
saving
76.95%
ResNet-50 with unstructured sparsity scheme.
700
600
Method
Training
Inference
Top-1
FLOPS (xel8) FLOPS (xe9) Accuracy
500
saving
400
43.05%
Dense
3.2
8.2
76,9
300
Spersity ratio
80%
200
SNIP
0.74
1.7
72.0
GraSP
0.74
1.7
72.1
DeepR 53
nia
n/a
71.7
SNFS
n/a
n/a
74.2
dense 9p0.8 sp0.9 sp0.95 sp0.98
DSR 9
1.28
3.3
73.3
#DST
SpFDE Avg. SpFDE
SET [9
0.74
1.7
72.9
RigL
0.74
1.7
74.6
Figure 4: The comparison of the training mem-
SpFDFag211202
0.74
1.7
75.35
ory costs between the minimum required mem-
MEST [
1.17
1.7
75.70
SpFDE
0.84
1.7
75.75
RigL-ITTOP
1.34
1.7
75.84 75.84
mum costs of SpFDE under the configuration
SpFDE 10%
0.95
1.7
76.03
saving 20% training FLOPs via layer freezing.
that pushing sparsity towards extreme ratios is not the only desirable direction for reducing training costs. SpFDE provides new dimensions to reduce training costs while preserving accuracy.
Tab. Lab.3/pmm provides the comparison results the ImageNet dataset using ResNet-50. At each training FLOPs level, SpFDE consistently achieves higher accuracy than existing works. is interesting to see that SpFDE outperforms the original MEST, in both accuracy and FLOPs saving. The FLOPs saving attributed to layer freezing and data sieving for end-to-end dataset-efficient training. Moreover, compared to the one-time dataset shrinking used in MEST, data sieving dynamically updates the training dataset, mitigating over- over-fitting and resulting in higher accuracy. We also conduct ablation studies on the impact of only applying layer freezing technique or data sieving technique and the results are provided in Appendix E.
5.2 Reduction on Memory Cost
the superior memory saving of our SpFDE framework. The memory costs indicate the memory footprint used during the sparse training process, including the weights, activations, and the gradient of weights and activations, using 32-bit floating-poinl representation with batch size of 64 on ResNet-32 using CIFAR-100. The "SpFDE Min." stands for the training memory costs after all the target layers frozen, while the "SpFDE Avg." is the average memory costs throughout the entire training process. The baseline results of "DST methods Min." " only
From Fig.
consider the minimum memory costs requirement for DST methods 3 53 54
4 19 [ [ [].
which ignores the memory overhead such as the periodic dense back pack-propagation in RigL [5]] dense sparse structure searching at initialization under this condition, our "SpFDE Avg." can still outperform the "DST methods Min." with large margin (20% 25.3%). The "SpFDE Min." results show our minimum memory costs can be reduced by 42.2% ~ 43.9% compared to the "DST methods Min." at different sparsity ratios. This significant reduction in memory costs is especially crucial to edge training.
(3, and the soft memory bound MEST 1 Even
5.3 Discussion and Limitation
The reduction in training FLOPs of our method comes from three sources: weight sparsity, frozen layers, and shrunken dataset. However, the actual training acceleration depends on different factors, e.g., the support of the sparse computation, layer type and size, and system overhead. Generally, the same FLOPss reduction from the frozen layers and shrunken dataset can lead to higher actual training acceleration than weight sparsity (more details in Appendix F). This makes our layer freezing and data sieving method more valuable in sparse training. We use the overall computation FL.OPs to measure the training acceleration, which may be considered a theoretical upper bound.

---

Conclusion
This work investigates the layer freezing and data sieving technique in the sparse training domain. Based on the analysis of the feasibility and potentiality of using the layer freezing technique in sparse training, we introduce a progressive layer freezing method. Then, we propose a data sieving technique, which ensures end-to-end dataset-efficient training. We seamlessly incorporate layer freezing and data sieving methods into the sparse training algorithm to form a generic framework named SpFDE. Our extensive experiments demonstrate that our SpFDE consistently outperforms the prior arts and can significantly reduce training FLOPs and memory costs while preserving high accuracy. While this work mainly focuses the classification task, future direction is further investigate the performance of our methods on other tasks and networks. Another exciting topic is studying the best trade-off between these techniques when considering accuracy, FLOPs, memory costs, and actual acceleration.
Acknowledgment
This work is supported in part by National Science Foundation CNS-1909172.
References
[1] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34, 2021.
[2] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
[3] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
[4] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning (ICML), pages 4646-4655. PMLR, 2019.
[5] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning (ICML), pages 2943-2952, PMLR, 2020.
[6] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense veerpparramterizzation?? in-time overrparaaeeterizztion in sparse training. In International Conference on Machine Learning, pages 6989-7000. PMLR, 2021.
[7] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[8] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint arXiv: PrXivv/706...r 770.004833, 2017.
[9] Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances Neural Information Processing Systems, 33: 13:14111- 14023, 2020.
[10] Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning, arXiv preprint arXiv:2102.013888 2021.
[11] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[12] Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Learning, pages 2337-2346. PMLR, 2019.
10

---

[13] Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint ArX.v:1911.0888 1.03900. 2019.
[14] Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive bert training. arXiv preprint arXiv:2010. /2562, 2020.
[15] Chaoyang He, Shen Li, Mahdi Soltanolkotabi, and Salman Avestimehr. Pipetransformer: Automated clastic pipelining for distributed training of transformers. arXiv preprint arXiv:2102.03161, 2021.
[16] Yoshua Bengio. Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5): 2007.
[17] Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie-Yan Liu. Learning what data to learn. arXiv preprint arXiv:
2017.
[18] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05115 2018.
[19] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1-12, 2018.
[20] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations (ICLR), 2016.
[21] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems (NeurIPS), pages 1379-1387, 2016.
[22] Ning Liu, Geng Yuan, Zhengping Che, Xuan Shen, Xiaolong Ma, Qing Jin, Jian Ren, Jian Tang. Sijia Liu, and Yanzhi Wang. Lottery ticket preserves weight correlation: In International Conference on Machine Learning, pages 7011-7021 PMLR, 2021.
desirable or not?
[23] Hengyuan Hu, Rui Peng. Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv: 160.v/607..00 03350, 2016.
[24] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 2074-2082, 2016.
[25] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning (ICML), pages 2498-2500 PMLR, 2017.
[26] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5058-5066, 2017.
[27] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV),
pages 1389-1397, 1397, 2017.
[28] Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Structured bayesian pruning via normall multiplicative noise. In Advances in Neural Information Processing Systems (NeurIPS), pages 6778-6787, 2017.
[29] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching- Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9194 2018
[30] Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), 2018
[31] Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In International Conference on Machine Learning (ICMI., pages 1135-1144 PMLR, 2018.
11

---

[32] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In Advances in Neural Information Processing Systems (NeurIPS), pages 759-770, 2019.
[33] Tuanhui Li. Baoyuan Wu, Yujiu Yang. Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3977-3986, 2019.
[34] Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4340-4349, 2019.
[35] Tianyun Zhang, Shaokai Ye, Xiaoyu Feng, Xiaolong Ma, Kaiqi Zhang. Zhengang Li, Jian Tang, Sijia Liu, Xue Lin, Yongpan Liu, et al. Structadmm: Achieving ultrahigh efficiency in structured pruning for dnns. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[36] Xiaolong Ma, Sheng Lin, Shaokai Ye, Zhezhi He, Linfeng Zhang, Geng Yuan, Sia Huat Tan, Zhengang Li, Deliang Fan, Xuehai Qian, et al. Non-structured dnn weight pruning- is it beneficial in any platform? IEEE Transactions on Neural Networks and Learning Systems, 2021.
[37] Maurice Yang, Mahmoud Faraj, Assem Hussein, and Vincent Gaudet. Efficient hardware realization of convolutional neural networks using intra- regular pruning. In 2018 IEEE 48th International Symposium on Mulippee-Vlued Logic (ISMVL), pages 180-185. IEEE, 2018.
[38] Xiaolong Ma, Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. Pconv: The missing but desirable sparsity in dnn weight pruning for real tall-imme execution on mobile devices. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34,
pages 51 5117-5124, 2020.
[39] Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn exccution on mobile devices with pattern- based weight pruning. In Proceedings of the Twenty-Fifh International Conference on Architectural Support for Programming Languages and Operating Systems, pages 907-922, 2020.
[40] Xiaolong Ma, Wei Niu, Tianyun Zhang, Sijia Liu, Sheng Lin, Hongjia Li, Wujie Wen, Xiang Chen, Jian Tang. Kaisheng Ma, et al. An image enhancing pattern-based sparsity for real-time inference on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 629 645. Springer, 2020.
[41] Peiyan Dong, Siyue Wang, Wei Niu, Chengming Zhang, Sheng Lin, Zhengang Li, Yifan Gong, Bin Ren, Xue Lin, and Dingwen Tao, Rimobile: Beyond real-time mobile acceleration of mns for speech recognition. In 2020 57th ACM/IEEE Design Automation Conference, pages 1 6. IEEE, 2020.
[42] Masuma Akter Rumi, Xiaolong Ma, Yanzhi Wang, and Peng Jiang. Accelerating sparse enn inference gpus with performance-aware weight pruming. In Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques, pages 267-278, 2020.
[43] Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang. and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13600-13611, 2021.
[44] Chengming Zhang, Geng Yuan, Wei Niu, Jiannan Tian, Sian Jin, Donglin Zhuang, Zhe Jiang, Yanzhi Wang, Bin Ren, Shuaiwen Leon Song, and Dingwen Tao. Clicktrain: Efficient and accurate end-to-end deep learning training via fine-grained irchtteture-prresevvvi pruning. In Proceedings of the ACM International Conference on Supercomputing, ICS '21, page 266-278, New York, NY, USA, 2021. Association for Computing Machinery.
[45] Wei Niu, Zhengang Li, Xiaolong Ma, Peiyan Dong, Gang Zhou, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Grim: A general, real-time deep learning inference framework for mobile devices based on fine-grained structured weight sparsity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[46] Hui Guan, Shaoshan Liu, Xiaolong Ma, Wei Niu, Bin Ren, Xipeng Shen, Yanzhi Wang, and Pu Zhao. Cocopie: enabling real-time ai on off-the-shelf mobile devices via compression- compilation co-design. Communications of the ACM, 64(6):62 68, 2021.
12

---

[47] Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-13, 2019.
[48] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard G. Baraniuk. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations (ICLR), 2020.
[49] Mohit Rajpal, Yehong Zhang, Bryan Kian, and Hsiang Low. Balancing training time vs. performance with bayesian early pruning. In Open 2020.
[50] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
[51] Paul Wimmer, Jens Mehnert, and Alexandru Condurache. Freezenet: Full performance by reduced storage costs. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020.
[52] Joost van Amersfoort, Milad Alizadeh, Sebastian Farquhar, Nicholas Lane, and Yarin Gal. Single shot structured pruning before training. arXiv preprint arXiv:2007.00389 2020.
[53] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep net-works, In International Conference on Learning Representations (ICLR), 2018.
[54] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 19Xiv: 040400 2019.
[55] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016.
[56] Sheng Shen, Alexei Baevski, Ari S Morcos, Kurt Keutzer, Michael Auli, and Douwe Kiela. Reservoir transformers. arXiv preprint arXiv:2012.1 15045, 2020.
[57] Wenchi Ma, Miao Yu, Kaidong Li, and Guanghui Wang. Why layer-wise learning is hard to scale-up and possible solution via accelerated downsampling. In 2020 IEEE 32nd International Conference Tools with Artificial Intelligence (ICTAI), pages 238-245 IEEE, 2020.
[58] Pooneh Safayenikoo and Ismail Akturk. Weight update skipping: Reducing training time for artificial neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 11(4):563-5) -7,45 2021.
[59] Yiding Wang. Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowdhury. Efficient dnn training with www.wllde-guiieee. layer freezing. arXiv preprint vrXiv:2201.06222 2022.
[60] Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7350-7357, 2020.
[61] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[62] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
[63] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition (CVPR), pages 248 255, 2009.
[64] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "sihVffhcmVJ", "reviewer": "Reviewer_EhKj", "summary": "This paper brings a new perspective in sparse neural networks training with dynamic sparsity (or, on short, DST) by studying layer freezing and data sieving. Moreover, it highlights and brings understanding on why DST is fundamentally different than dense training. Consequently, it proposes a new framework to improve DST performance by considering three dimensions, i.e., weight Sparsity, layer Freezing, and Dataset Efficiency (SpFDE). The empirical validation shows good performance for SpFDE.", "questions": "Structural similarity. Line 163 – 50% seems an arbitrary choice? How other values would affect the structural similarity findings? \n\nRepresentational similarity. Line 188 – Why you find this “surprisingly”? As long as connections are added and removed from the networks in DST during learning, the representation is somehow “forced” to change.\n\nTraining set. Lines 288-289 – Is there any relation with bagging?\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n_ Originality. Up to my best knowledge, this paper is the first to study layer Freezing in Sparse Training. The study itself is novel, and the findings on structural and representational similarity bring a new perspective in DST.\n\n_ Quality. The novel proposed methodology and the well-designed empirical validation seem sound. Good qualitative discussions, reflecting a good understanding of DST and its challenges. \n\n_ Clarity. The paper is well written.\n\n_ Significance. The paper may have some impact. Particularly, the knowledge unveiled on structural and representational similarity can lay the ground for impactful future research.\n\n**Weaknesses**\n\n_ Clarity. A careful proofread is recommended. There are still typos (e.g., on line 305)\n\n_ Statements. Some statements are too strong. Please consider revising. For instance, line 326, *reducing training costs by pushing sparsity towards extreme ratios is no longer a desirable methodology* prunes valid research directions.\n\n_ (not really a weak point) The improvement in performance over the well-chosen baselines is relatively small, but quite constant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hNuD-9dkcnI", "reviewer": "Reviewer_y6sL", "summary": "The paper studies two concepts, namely, layer freezing and data sieving, in conjunction with sparse training for DNN architectures. The layer freezing methodology is not new, but it has been studied for the first time in this context. The newly proposed data sieving methodology relies on first removing a % of the training instances and then at each epoch, a similar % of the easiest instances are removed and replaced with instances removed from the original set. Both ideas are studied in the context of sparse training, and extensive experimentation shows promising improvement in reducing computational and memory costs with good accuracy.", "questions": "I enjoyed reading the paper as it's well-written and the two concepts studied are properly introduced and motivated by experiments to demonstrate their flaws and importance.\n\nThe experimental results support the claims, and they are extensive.\n\nThe code is provided, which can help in the validation of the results as well as in the adaptation of the proposed solution.\n\nMy main concern is that the paper is lacking in terms of technical depth. The ideas explored are relatively simple, and the idea of sieving reminds me of ideas of perturbation analysis (we don't perturb but instead remove instances and replace the easily classified cases). That being said, sometimes we don't know over-complicated solutions, and this approach seems to work well.\n\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1. Important and timely problem\nS2. Well-written paper, nicely motivated \nS3. Extensive experimental results\n\nWeaknesses:\n\nW1. The technical depth is somewhat low", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g3PV4k9zajq", "reviewer": "Reviewer_AURL", "summary": "The authors propose to modify an existing dynamic sparse training (DST) method, MEST, by (a) freezing the training of early layers in a CNN  sequentially early in training to remove the requirements to calculate/maintain backwards pass, and (b) use a “data sieving” method to train in a more data efficient manner. The freezing method is proposed under the intuition that early layers converge to both weights and sparse structure earlier than later layers, and the authors present experimental results claiming to demonstrate this. The authors compare their method (SpFDE) to existing sparse training methods, including SNIP, GrasP, SET and MEST on CIFAR-100 with ResNet-32.", "questions": "Questions:\n* The experiment presented in 3.1/fig 1(a) comparing the masks of two models trained with a DST method seems to assume that the mask found by DST methods is the same for a DNN trained with different initializations - this is well known not to be the case (as also seen with lottery tickets)! How is your experiment finding masks that are substantially similar, up to 100% in some cases? Are the initializations the same? What is the saliency measure used when choosing the “50% most significant weights”?\n* The experiment presented in 3.2/fig 1(b) claims that sparsity levels do not affect sparse training convergence time with MEST. Is this a known result? At least with other DST methods, such as RiGL, I believe this may not be the case.\nHow exactly do you calculate the theoretical FLOPSin general, and what are your assumptions on the FLOPS of a frozen layer? On a sparse layer?\n* You state that you focus on sparse training from scratch rather than pre-training as it’s not feasible to use pre-training on edge devices. Why? Intuition suggests the exact opposite if anything.\n\nSuggestions:\n* I actually find the research direction(s) presented by this paper to be very interesting and intuitive, especially the freezing direction, however the current paper doesn’t have sufficient experimental analysis, results or focus to make for a convincing story. I would encourage the authors to take the feedback of the reviewers and attempt to correct this for a future submission.\n* I would recommend focusing on each of these methods as it’s own paper (or only on the freezing), they are not related enough to write together as one method, and it weakens the story and clarity of your research motivation and findings. There are significant experimental details missing likely as a result.\n* Your work is not the first work to explore “frozen” layers (e.g. fine-tuning, or freezing weights) & sparsity, I would remove this claim as it’s hard to justify. Also, FreezeNet in particular should be cited here as other work looking at freezing weights in pruning/sparsity.\n* “front layers” -> “early layers” in all instances, to match the common terminology in the field.", "limitations": "The authors do not discuss any limitations or potential impacts of their work. I would note specifically that the data sieving method could affect the learning of biases from datasets since it is sub-sampling the dataset during training, and the authors do not identify this.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Questions the wisdom of using the same amount of data/training time to train different layers in a sparse DNN training given the common intuition that early layers converge on weights faster.\n* Interesting and intuitive hypothesis that the early layers in a CNN may converge on sparse structure early in training with a dynamic sparse training method.\n* Analysis of working memory footprint of model is interesting, and rare to find in papers.\n\nWeaknesses:\n* Only theoretical FLOPS shown, and the calculations of theoretical FLOPS for frozen layers is flawed in that it doesn’t consider the real-world scenarios of usage of these models. Forward passes are still required for training through the frozen layers, and this analysis seems to ignore this. Theoretical FLOPS are at best misleading and often completely unrelated to real-world efficiency (e.g. unstructured sparse masks of 99% have a theoretical FLOPS that is low, but hard to accelerate), and any paper claiming to demonstrate real-world speed up of training/inference must include real-world computational measurements of timings. Unfortunately, this fact alone means this paper cannot be accepted as-is. It is not explained what FLOPs these are (e.g. multiply-accumulate), or how they are calculated in general, or specifically for frozen layers.\n* Results showing decreased FLOPS with good generalization only demonstrated on a single model (ResNet-32) and on a single dataset (CIFAR-100). Results on CIFAR-10 and ImageNet are in the appendix, but seem to show increased FLOPS for similar/better generalization.\n* Missing substantial experimental details, making some of the experiments non-reproducible. In fact, some of the results of experiments presented contradict prior work/well-known issues in sparse training (see questions).\n* The first few layers in a CNN are actually upwards of 90% of the compute of the whole model (while being a small fraction of the weights as the authors ). In this sense, the intuition presented by the authors on why they focus on the latter layers doesn’t make sense.\n* By trying to present two very different approaches to improving sparse training as one method, the paper reads as a confusing story — these two methods are not related enough to roll into one paper/method IMHO.\n* SpFDE incorporates both the frozen layers and data sieving methods, and results are only shown as such. To evaluate the individual contribution of each of these methods the authors need to provide an ablation analysis, however there is none. It is impossible to evaluate the claims of the authors unfortunately on each of the individual proposed methods without this. Furthermore, the results comparing the proposed method with other DST and sparse training methods are not directly comparable, since they are not using data sieving.\n* Experiments only focus on one specific dynamic sparse training method (MEST), but conclusions are made over all dynamic sparse training methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IP5tKYAZb7k", "reviewer": "Reviewer_363A", "summary": "This paper proposes a framework for efficient sparse training. Instead of focusing on pushing the sparsity levels, the authors investigate the effect of two known methods, including layer freezing and data sieving, in the dynamic sparse training framework. The authors show why layer freezing can be applied to sparse training. Then, they described a framework that reduces both memory requirement and FLOPs count while achieving a comparable performance or outperforming state-of-the-art sparse training algorithms. The authors performed an extensive evaluation to support their claims.", "questions": "- I suggest the authors add a brief explanation about the CKA metric and how it is calculated in the background information.\n- If enough time and resources are available, it would be nice to see how much the layer freezing and data sieving can be increased without a significant drop in accuracy. E.g., to present in a 2d-plot where the x-axis shows the change in one of these components and the y-axis shows the accuracy.\n-  Minor: typosà lines 288 (,), 305 (sparse) ", "limitations": "Yes, the authors have discussed the limitations and adequately addressed them.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper is very well-written and has decent organization, and it was a pleasant read.\n-  This paper provides insights on other approaches to push further the efficiency of sparse training algorithms instead of only focusing on increasing the sparsity level.\n- Extensive evaluation that supports the initial claims.\n-  The results on the ImageNet dataset are significant in decreasing the training FLOPs while increasing the accuracy and having different trade-offs between efficiency and accuracy. \n\nWeakness:\n\n- The effect of each component (layer freezing and data sieving) on the accuracy is not separately clear. It would be nice to add an ablation to analyze the effect of each component.\n- It is unclear how the authors determine the value of T_{frz} (starting freezing). Is it difficult to choose a value for this hyperparameter for different sparsity levels/models/datasets? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept