Get More at Once: Alternating Sparse Training with Gradient Correction
Authors: Li Yang
Jian Meng
Jae-sun Seo
Deliang Fan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sparse training
Alternating training scheme

Abstract: Recently, a new trend of exploring training sparsity has emerged, which remove parameters during training, leading to both training and inference efficiency improvement. This line of works primarily aims to obtain a single sparse model under a pre-defined large sparsity ratio. It leads to a static/fixed sparse inference model that is not capable of adjusting or re-configuring its computation complexity (i.e., inference structure, latency) after training for real-world varying and dynamic hardware resource availability. To enable such run-time or post-training network morphing, the concept of `dynamic inference' or `training-once-for-all' has been proposed to train a single network consisting of multiple sub-nets once, but each sub-net could perform the same inference function with different computing complexity. However, the traditional dynamic inference training method requires a joint training scheme with multi-objective optimization, which suffers from very large training overhead.  In this work, for the first time, we propose a novel alternating sparse training (AST) scheme to train multiple sparse sub-nets for dynamic inference without extra training cost compared to the case of training a single sparse model from scratch. Furthermore, to mitigate the interference of weight update among sub-nets, we propose gradient correction within the inner-group iterations to reduce their weight update interference. We validate the proposed AST on multiple datasets against state-of-the-art sparse training method, which shows that AST achieves similar or better accuracy, but only needs to train once to get multiple sparse sub-nets with different sparsity ratios. More importantly, compared with the traditional joint training based dynamic inference training methodology, the large training overhead is completely eliminated without affecting the accuracy of each sub-net. 


Get More at Once: Alternating Sparse Training with Gradient Correction
Li Yang", .Jan Meng", Jae-Sun Jae-Sun Seo, Deliang Fan School of Electrical. Computer and Energy Engineering Arizona State University, Tempe, AZ 85287
{1yang166, jmeng15,, jseo28, dfan @asu. edu
Abstract
Recently, new trend of exploring training sparsity has emerged, which removes parameters during training, leading to both training and inference efficiency im- provement. This line of works primarily aims to obtain single sparse model under a pre-defined large sparsity ratio. leads to a static/fixed sparse inference model that is not capable of adjusting re-configuring its computation complexity (i.e., inference structure, latency) after training for real-world varying and dynamic hardware resource availability. To enable such run-time post-training network morphing, the concept of 'dynamic inference' or 'training-once-for -frlll has been proposed to train single network consisting of multiple sub-nets once, but each sub-net could perform the same inference function with different computing com- plexity. However, the traditional dynamic inference training method requires a joint training scheme with multi-objective optimization, which suffers from very large training overhead. this work, for the first time, we propose a novel alter- nating sparse training (AST) scheme to train multiple sparse sub-nets for dynamic inference without extra training cost compared to the case of training a single sparse model from scratch. Furthermore, to mitigate the interference of weight update among sub-nets without losing the generalization of optimization, we pro- pose gradient correction within the oner-group iterations to reduce their weight update interference. We validate the proposed AST on multiple datasets against state-of-the-an art sparse training methods, which shows that AST achieves similar better accuracy, but only needs to train once to get multiple sparse sub-nets with different sparsity ratios. More importantly, comparing with the traditional joint training based dynamic inference training methodology. the large training overhead is completely eliminated without affecting the accuracy of each sub-net. Code is available at /http:: h/g::/iiiii gighub...... com/ conm/menngia00502/AA1
Introduction
For Deep Neural Networks (DNNs), sparsification (i.c., pruning) has been widely explored in the last decade aiming reduce the computational and memory cost by removing unimportant parameters. Early works of DNN pruning focused on exploring the post-training sparsity [10. 7 11 to improve the inference efficiency, which prunes well-trained dense model followed by additional fine-tuning to recover accuracy. Recently, exploring in-training sparsity has emerged as a promising technique
improve the training efficiency by pruning parameters during training 3 4
9 19 [4]. The
success of these in-training sparsification methods is mainly rewarded from the prune-and-reyee
technique, which periodically and aggressively eliminates the unimportant non-zero weights from the
'These authors contributed equally to this work
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Disabled connection
Active connection
+ Pruned Channel
loss
Y
Y.
Loss
Loss
Y;
Inner group
Loss
Grad
Loss
X,
X
Switch
Switch
X
X
X
X
Mini-Batches
Mini-Batches
Mini-Batches
Training Direction
(a)
(b)
(c)
Figure| I: (a) Individually train multiple sub-nets from scratch. (b) Jointly train multiple sub-nets for dynamic inference. (c) The proposed alternating sparse training (AST) scheme.
model, and then regrows certain portion of the pruned candidates back. Until now, such in-training sparsity works primarily focused training single sparse model under pre-defined sparsity ratio, generating a static and fixed sparse network. In other words, the sparsity ratio has to be pre-calculated before training depending on the target hardware resources and application requirement. However, obtaining multiple sparse models (towards storage/energy vs. accuracy trade-off) from the same model architecture requires extra fine-tume tne-tuning or training overhead.
To enable run-time or post-training network morphing, the concept of ddnamiic inference' or 'training- for-llo-- has emerged recently [24] 2). Slimmable Neural Network (S-Net) (4 and its optimized counterpart (US-Net) [22] allow the model to switch between different architectures among the pre-defined sub-net candidates with different channel widths. Such run-time dynamics allows users to tune the tradeoffs between model complexity and inference accuracy. To mitigate the weight update interference, S-Ne//US- Net forces the smaller sub-net be completely nested by the larger sub-net. Based on that, OFA [2] and BigNAS (23) are extended to support a much larger number of sub-nets across more dimensions (e.g., depth, width, kernel size, and resolution). However, all prior methods rely on a joint training scheme with multi-objectivy optimization, which requires minimizing the loss of all sub-nets collectively. Specifically, all sub-nets have to perform forward and backward passes in each mini-bat batch iteration, resulting in much larger training overhead compared to the individual network training.
To address these concerns, we propose novel alternating sparse training (AST) method for dynamic inference which aims train multiple sparse sub-nets one training procedure without extra training cost, compared to training a single sparse model independently [3 4.
9 19 14]. To generate and sparsely train each sub-net, we adapt the prune-and-regrow scheme from the in-training sparsity method 14]. as it achieves high training sparsity among other schemes. AST alternately trains sub-nets with different sparsity values for different mini-batch iterations, as illustrated in Figure I Inspired by the implicit regularization of SGD that was revealed by [88 16]. we further demonstrate such alternating training scheme for dynamic inference helps to maximize the inner product of gradients between the consecutive mini-batch iterations (i.e., sub-nets in our case). However, even with the help of the implicit regularization, we find that negative inner products still remain, which represents the conflicting gradient direction between each two sub-nets. Such phenomenon causes the interference of weight update, while helpful to escape from local minima and improve generalization of optimization for individual network training by mini-batch SGD (6] []] To mitigate the interference weight updates between sub-nets but without losing the generalization, we further propose gradient correction remove the confficting gradient direction between sub-nets only within iterations, where is the
inner-group group iterations, which is defined as consecutive number of sub-nets (c.g., N = 2 shown in Figure D
Overall, compared the sparse training and dynamic inference works, AST achieves both training efficiency with sparsity (against dynamic inference) and inference dynamics with multiple sparse sub-nets (against individual sparse training). We conduct comprehensive experiments on CIFAR-10. 10, 100AR-100 and ImageNet datasets with various DNN models and sparsity granularities. Compared to the SoTA sparse training on CIFAR, the proposed AST algorithm achieves similar or better accuracy while training multiple sparse sub-nets all at once (reduced training cost). Furthermore, compared to the dynamic inference method, AST consistently achieves better accuracy on all the experiments.

---

2 Related Works and Background
2.1 Sparse Training
As summarized in [8]. sparse training works mainly can be categorized into three groups according to
the time point the sparsity is applied: 1) Post-training sparsification 3 4 9 19
14,, which removes
the weights by fine-tuning based on a pre-trained model; 2) Before-training I 20 sparsification, which obtains sparse model before the main training procedure: and 3) In- In-training sparsification,
which removes weights during the training process from scratch [14] 26 4 B).
In-training sparsification Different from post-training and reforigtraining sparsification, exploit- ing sparsity during training only requires single training process from scratch. The pruning topology is gradually perfectized along with the weights optimization. Compared to the before-training sparsifi- cation, pruning the model during training makes the gradient visible to the algorithms, so the reflection of the gradient can be used to correct the final pruning decision, leading to better accuracy compared to post-training. Motivated by this, the prune-and-regron technique unimportant non-zero weights from the sparse model and regrows certain pruned weights back during each mini-batch iteration of training process. As a representative work, RigL [4] first prunes certain ratio r based on weight magnitude as:
3 periodically removes the
= Topoo((w)) rr,,
where TopK (v, k) returns the weight/activation tensor retaining the top k-proportion of elements from e, and s is the targeted sparsity ratio. After that, the proportion of new connections are re-generated based on the gradient magnitude in the same mini-batch iteration:
w + Topk (0t==",str
(2)
By doing so, such grune-and-regwe scheme optimizes the sparse connection with fixed sparsity ratio during the entire training process. Based on this, recent works develop different weight importance criterion to perform prune-and-regrow. SNFS (3] uses the momentum magnitude of weight to do pruning. MEST (26) uses the sum of weight and gradient magnitude to indicate the importance of weights for pruning, and further randomly select weights to grow back. In addition, GraNet [4] employs the same rule as Rigl but adopts a smaller sparsity as initialization (c.g.. 0.5) and gradually prunes the model to the target sparsity (c.g.. 0.8, 0.9) by following a cosine decay. In addition, more similar to our work, [9 16] propose to alternately train a dense model and one of its sparse variant (i.e., sub-net) returning both accurate dense model and sub-net. However, the rationality behind it is not clearly described. Overall, the objective of our work is obtaining multiple sparse sub-networks with the same amount of training effort the individual network sparsification. More recently. based on the in-training sparsification mechanism, [13] proposes dynamic sparse training ensemble method to independently generate multiple sparse sub-nets for ensemble. Such method is orthogonal to our work which could be further combined to improve accuracy.
2.2 Joint Training for Dynamic Inference
21 primarily aim train a single network which consists of multiple sub-nets to perform inference independently. Since the weights of the sub-nets are partially shared between each other, they can be switched at run-time permitting dynamic inference accuracy- complexity trade-off. Such concept is firstly proposed by Slimmable Neural Network (S-Net) [24] and its optimized counterpart (US-Net) [22 which train single neural network executable different widtnsl-widths. Inspired by S-Net, OFA 2 and BigNAS 23 are proposed to construct a dynamic DNN that includes a much larger number of sub-nets across many more dimensions (e.g.. depth, width, kernel size, and resolution). Note that, as revealed by (2], to enable sub-nets switching with un-compromised individually inference accuracy, the structures of sub-nets are defined by a subset rule: the smaller sub-net must be the completely subset of the larger sub-net. Specifically, this line of works train network a joint training fashion, which can be expressed as:
Dynamic inference works 24
22 23
min Ew C C(X(, (W.)):))
where X is the mini-batch of inputs with corresponding targets Y. and N is the number of sub-nets. ((::-) calculates the entss-yn loss of DNN output and target. f(X, (W:)) computes the output of sub-net parameterized by [. However, such method consumes much larger training time compared to training a single individual model, since all sub-nets have to perform forward and backward once in each mini-batch iteration as shown in Eq.
3

---

Disabled connection
Active connection
Loss,
LOSS
LOS
Los
Swithy
Switch
Switch
Initialization
[
Iterations
Sub-Net 1
Sub-Net 2
Sub-Net 3
Sub-Net 1
Figure 2: Alternttng sparse training (AST): The subset network (sub-net) are iteratively activated throughout the training. The model only learns the active connections of each sub-net, leading to the one-time-traiie trained multiple sub-nets.
3 Sparse Train Once Get More
3.1 Rationale of Alternating Sparse Training (AST)
22] which Different from joint training for dynamic inference such as S-Net [24] and US-Net
updates all sub-nets in each mini-batch iteration, we propose the alternating sparse training (AST) scheme which trains multiple sub-nets in an alternating fashion over time, where a single sub-net trained in one iteration. Note that, the sub-net is defined part weight of the network, which can run inference independently. All the sub-nets are partial shared with each other within a single network. As shown in Figure 2, assuming
have three sub-nets with different sparsity ratio (e.g., sub-net I < sub-net 2 < sub-net 3), such training scheme will start to train the sub-net in the first iteration followed by the training of the sub-net 2, and sub-net 3 in the second and third iteration, respectively. only updates the active connections while ignoring the disabled weights (i.e., pruned for current sparse sub-net), then repeatedly switches sub-nets every three consecutive iterations until the training completes.
The rationale behind AST is inspired by the findings in Reptile [8]] which is meta-learning algorithm originally aiming to learn an initialization of model from multiple tasks for fast adaption on tasks. It shows that mini-batch SGD imposes an implicit regularization to maximize the dot-produc of consecutive mini-batches. the following, we will show how this proof adapts to our case. Assuming that we alternately train two sub-nets, sub-net and sub-net 2 with two consecutive mini-batches B₁ and B2 respectively, according to the Taylor Series, the gradient of sub-net g2 calculated by SGD can be expressed by:
L' (w2) = L' (wI) + L" (ww) - w1) + O(1 will3)
c' (w1) Hu( - w1) + (aa2)
= L' (W1) aH + O(a)) (using w
ag2)
Where H1 1 is Hessian of the sub-net 1 and a is the current learning rate. Similar to Reptile, the term aH 192 serves maximize the dot-pne not-product of the consecutive sub-nets, where the expectation can be expressed as:
F1 = - 112 Ow' (g) 92)]
2
Thus, -aH is the direction that serves to maximize the inner product of two consecutive mini- batches, which means to improve the gradient alignment of two sub-nets towards better learning generalization. Such proof can be easily adapted to more than two sub-nets. The detailed explanation of Eq. iss relegated into the Appendix.
3.2 Proposed AST with Gradient Correction on Consecutive Inner-group Sub-nets (AST-GC)
Although the implicit regularization of mini-batch SGD helps to maximize the inner product gradient of consecutive sub-nets as discussed Section 3.1 we indeed observe that there still remain partial number of negative inner product during training, causing conflicting direction of weight updates among sub-nets as shown in Figure However, terms of optimizing network by mini-batch SGD, the noise caused by such conflicting gradient direction helps to escape from saddle

---

Inter group
Inner group Grad Correction
Inner group Grad Correction
Switch
Initialization
t+1 1 Sub-Net
t+2 Sub-Net 3
2+3
t 4
t+ Sub-Net
Iterations
Sub-Net
Sub-Net
Sub-Net 2
Figure 3: The overview of the AST process with gradient correction on consecutive inner-group sub-nets (AST-GC)
points or local minima and improve the generalization well gradient projection method, which is used in multi-task learning propose a gradient correction technique remove the conflicting gradients within the inner-group itera- tions during training, meanwhile allow the gradient to have negative direction for the inter-group sub-nets. As shown in Figure 3] the inner-group iterations is defined as the consecutive sub-nets within N mini-bath itera- tions, where N is the total number of sub-nets (c.g..
6 1
Due to that, inspired by the
and continual learning [12]
Inner product ratio 196)
1.0
Negtive
0.8
Positive
0.6
0.4
0.2
N=3 as shown in Figure 3). In contrast, inter-group rep- resents the relationship between inner-group iterations. Specifically, the gradient correction adapts a simple pro- cedure within the aner-group sub-nets: if the gradients between two consecutive sub-nets are in conflict (i.e., their cosine similarity is negative), we project the gra- dient of current sub-net onto the normal plane of the gradient of its prior sub-net. Otherwise, the original gradient is unchanged. Considering two sub-nets in inner-group iterations with the gradient gi and 9j respectively, the technique can be mathematically illustrated as:
0.0
40
60
100
120
140
160
Epochs
Figure 4: The ratio of negative and positive inner product of two sub-nets during the AST process CIFAR-10 by using wide ResNet-32.
9.gg To.1 10 Sj: if <
9t
(5)
9t.
otherwise
By doing so, the proposed gradient correction within inner-group iterations have two benefits: 1) it could guarantee that all sub-nets within the consecutive amer-group are updated the non-confficting direction, reducing the weight interference between each other; 2) the normal update using the mini-batch SGD between inter-group sub-nets will help to improve the generalization of the model.
3.3 Sparse Sub-net Training
Another important aspect of our AST scheme is generate and train "sparse" sub-nets. Following most sparse training works, we adopt the prume-and- aad---nod-re scheme from GraNet [14] as the backbone technique. Specifically, AST starts from a random initialized sparse model, and then applies prune- and-regrow mechanism as described in Eq. [ and Eq. 2 for each training iteration with the sub-net specific sparsity ratio. The pseudocode of the proposed AST algorithm is relegated into the Appendix. Given to the fact that alternating training scheme switches the sub-net per iteration, the following questions arises: 1) What is the optimal architecture relationship between sub-nets? 2) How the pruning should be scheduled for each sub-net? We address these two questions with the following observations:
Observation 1: Enabling the freedom of exploring unique architectures of each sub-net elevates the learning capacity of AST over the S-Net.
To validate this observation, we generalize the sub-net relations into the following two categories:
Completely subset (CS): As proposed by S-Net 24 the high-sparsity models are fully contained the low-sparsity models (Figure a)). Under the context of prune-and-regroww the regrowing process is only performed in the lowest sparsity model, while the rest of
5

---

Completely Subset (CS)
Non-disjoint (ND)
Table I: Averaged accuracy and standard devia- tions between different AST training schemes on CIFAR- 10 dataset (three times experiments each).
Gk
Gj
G
G
G₁
Dataset
CIFAR-18
ResNer-32 32
Dense Model Acc. 94.88
(a)
(b)
Pruning Ratio
70%
98%
Figure 5: Different relationships between three subnets G. Gj, and Gx: Completely subset (CS) and Non-disjoint (ND).
Completely Suboet
Non Disjoint (ND)
94.47 = 0.10
93.78 = 0.09
92.76
0.21
the sub-nets rigorously extend the sparsity from the previous low-sparsity model by using magnitude-bane pruning only. The one-time regrow guarantees the fully-subset relation- ships among different sub-nets.
Non disjoint (ND): Each sparse sub-net performs prune-and-regrow individually to optimize the overall pruning decision during training. As depicted in Fjg sub-nets can freely exploit their own architectures while the intersections remain non-empty. Compared to the CS scheme, the non-disjoint AST empowers the subset networks with more architecture freedom. Figure 6 shows the layer-wise sparsity gap and non-overlap between two ND- trained sub-nets that target different final sparsity values (sy). The non-overlap is computed by XORing the binary sparse masks between sub-nets, the percentages of "1" in the resultant tensor represents the level of non-overlap. Apparently, the distinction of connections is larger than the sparsity difference, which implies the existence of the unique connections generated by the ND yuuneannd-regrow in each sub-net.
20.0
Sparsity gap: SubNet-2 (s,)8%% to SubNet-1 (sp70%)
18.0
Non-overlap: (8,088%) (s,77%%
Sparsity Non- overtaa
Architecture freedom
14.0
12.0
10.0
8.0
6.0
8
10
16
18
22
24
26
32
Layers
Figure 6: Layer- wise sparsity and the connection dissimilarity between two ResNet-32 sub-nets trained by the non-disjoint (ND) AST scheme.
With the ND scheme, we observe that the small amount of architecture freedom shown in Fig- ure/6|can can lead to the improved overall performance: Table summarizes the accuracy of the wide ResNet-32 [26] trained by different AST schemes on CIFAR-I0 dataset. Assisted by the compre- hensive architecture exploration, the non-disjoint AST scheme achieves higher accuracy compared the conservative completely-subee (CS) training. Thus, in this work, we use ND-scheme for all experiments.
Observation 2: Intermittent sparsity incre- ment among sub-nets stabilizes AST process. As introduced in Section 2.1 the sparsity of the
CIFAR-10 Validation Accuracy bo (%
in-training sparsification method is periodically updated (e.g., 1,000 iterations) based on a pre-
so.
defined sparsity schedule [14). Regarding the
AST scheme, the sub-net model architectures are consecutively switched and trained, but the
successive architecture switching does not im-
Iterations
ply the necessity of consecutive sparsity update of each sub-net. the contrary, the intensive sparsity increment of all sub-nets could desta- bilize the training. this work, the sparsity of each sub-net increases periodically, but the spar-
Ar 300 Iterations
60
70
80
100
110
Training Epoch
Figure 7: Stabilized training of 98% subnetwork rewarded from the extended adjustment period A..

---

Table 2: CIFAR-10/100 accuracy and training cost comparison with SoTA works on wide ResNet-32.
Dataset
CIFAR-10 Acc. (%)
CIFAR-100 Acc. (%)
ResNet-32
Dense Model Acc. 94.88
Dense Model Acc. 74,94
Train.
Pruning Ratio
90%
95%
98%
90%
95%
98%
Individuul Training
Lottery Ticket [5]
92.31
91.06
88.78
68.99
65,02
57.37
3x
SNIP [0
92.59
91.01
87.51
68 68.89
65,22
54.81
3x
92.97
88.46
GraNet a 056)
94, 12
94.02
92.98
73.18
72:56
69 89
3x
MEST 26 (s. 90%)
92.12 0.13
2 0.11
68.78
0.26
69.35 4: 0.36
67.85 =: 0.23
62 58 4: 0.31
3x
MEST+EMI
0.29
0.11
0.26
0.32
3x
MEST (4) 90%)
93.27
0.14
92.
0:13
90.51 + 0.11
71.30 + 0.31
70.36 + 0.05
67.
0.25
3x
Training once for all
JrannlyTTriined [2] = 0%)
92.59 = 0.21
92.48 0.21
70.30 0.14
69.32 = 0.84
06.85 = 0.59
3x
AST(a; 0%8)
93.51
0,06
93.11 0.08
92.44 0.04
73.12 1
72.39 1: 0.14
68.06
:: 0.211
1x
AST = 90%)
92.32
0.06
92. 12.19:::00
91.31 + 0.01
69.82 0.12
09:22 = 0.07
66.37
0.15
1x
AST+GCLs; (1%)
93.88
0.19
93.70
0.28
92.69 0.09
73.41 1: 0.04
72.57
1: 0.15
68.42 1 0...0.11
1x
AST+GCGs 90%)
0.13
0.10
0.18
70.11 0.39
70.01
0.54
67.15
sity increment of each sub-net intermittently performed with the adjustment period Ar. In the meantime, sub-nets are still consecutively switched during the Ar. Figure Fggure]shows shows validation accuracy of an AST-trained ResNet-32 sub-net with final sparsity of 98%. During the early stage of training with high learning rate, the stabilized validation accuracy indicates the effectiveness of the extended adjustment.
the CIFAR-I0
4 Experiments
4.1 Settings
The proposed AST method is thoroughly verified with multiple datasets, including CIFAR-10, CIFAR- 100, and ImageNet. We use similar training scheme as prior works [4] 26). which employed 160 epochs for CIFAR-10/100 and 100 epochs for ImageNet training. The multiple sparse sub-nets trained from scratch and pruned by the proposed AST algorithm. The cosine annealing learning rate schedule is employed with 0.1 initial learning rate for the CIFAR datasets training. For ImageNet dataset, we introduce extra 5 epochs for the initial warmup training. The pruning candidates are globally selected while the first layer remains dense. Regarding the regrowing process, the percentage of regrow candidates gradually decrease from 0.5 to 0.0 with cosine annealing schedule. In the meantime, the extended adjustment period between sub-nets set 100 and 400 for CIFAR and ImageNet experiments, respectively. While we believe a more fine- ine-grained hyperparameter tuning could lead to better accuracy, we choose the above scheme for simplicity and reproducibility. The reported neb accuracy values are obtained from single background model checkpoint. In all experiments, we report the average accuracy with its variation in 3 runs.
4.2 Main Results
CIFAR-10 and CIFAR-100. Table shows the CIFAR-10/100 accuracy of the proposed AST algorithm, where we used the wide ResNet-32 model that was employed in (26). Following the typical high sparsity results reported previously, we train three sub-nets with 90%. 95%, and 98% sparsity from scratch all at once with identical epochs as prior works. Given the fact that the high initial sparsity is beneficial to the overall memory saving of the training process, report the results with both dense (s; 0%) and highly sparse (s; = 90%) initial models. Compared to the prior individually-traine SoTA works [14] 26),, the proposed AST+GC algorithm obtains three highly sparse models through one-time training with negligible accuracy degradation. The resultant 3x total training cost reduction frees the model from mnergy-urged extra training, leading to power and latency benefits. The joint training scheme 24 requires multiple forward pass in each iteration, the increased computational cost leading to the setngedddyy--uubptiimal performance. In addition to the reduced training cost, the proposed AST method outperforms the soint-training scheme (24) by up to 1.3% and 2.9% inference accuracy, while achieving up to 2.63 training cost reduction CIFAR-10 and CIFAR-100 datasets, respectively. We also verify the AST scheme with more sub-nets and large sparsity gaps in Section 4.4
2m2geNet-2012.. We further evaluate the proposed method with ResNet-50 on ImageNet Table 3 It is noticeable that the 0.5x and 0.25x model of the jointly-traines S-Net (24] corresponding to
7

---

Table 3: ImageNet accuracy and training cost comparison with SoTA works on ResNet-50.
Method
ImageNet-2012
ResNet-50
Dense model Acc. 76.8
Prune Ratio
80%
90%
Individual Training
Top-I Acc. (%)
Training Cost
Top-I Acc. (%)
Training Cost (xe11)
SNIP 11]
69.7
1.67
62.0
0.91
SET
72.9
0.74
69.6
0.32
DSR 17/
73.3
1.28
71.6
0.96
RigL
4
74.6
0.74
0.39
MEST EM
236
75.8
0.74
73.6
0.39
GraNet 14
76.0
1.18
74.5
0.80
Training once
Jointly. Trained 24
50%)
71.9p.5x
1.19
65
1.19
AST 50%)
72.6
0,41
AST GC (5; 50%
73.2
0.59
73.1
0.41
AST GC 80%)
72.6
0.37
72.5
0.13
Table 4: Inference acceleration and negligible accuracy drop of the proposed AST algorithm with structured fine-gh ane-grained sparsity on ResNet-18 18 model.
Dataset
CIFAR-IO
(%)
Training Cost
N Sparse Pattern
Dense Model
2:4
3:4
78
15:16
Individually Trained (SR-STF) 27
95.07
94.89
94.47
94.25
93.92
2.33e=16 (3.95x)
AST + GC
94.63
94.26
94.31
93.79
5.91e+15
Inference Time / 10K images (sec)
1.40
1.01
0.67
0.66
0.63
72.98% and 92.23% weight sparsity, the averaged overall sparsity (82.6%) is less than our targeted sparsity (85%). The proposed AST training scheme outperforms the joint-training scheme by 7.5% inference accuracy with 80% highly sparse initial models. Compared to the individual training scheme, the proposed AST scheme achieves a comparable even better performance to SNIP [II] and SET [15), while reduces the total training cost up to 2.38x. In addition, compared to AST without gradient correction (GC), AST+GC improves accuracy by 0.6% and 0.8% respectively. The experimental results with different settings and models are shown in the Appendix.
4.3 Extend to Structured Fine-grained Sparsity
Demonstrating the pparsity-inducee speedup on GPU has now emerged as a feasible solution to depict the effectiivness of pruning algorithms. The recent Nvidia Ampere architecture is equipped with the Sparse Tensor Cores to accelerate the neural network computation on GPU with N: structured fine-grained sparsity (27). Varying the group size M and the number of sparse elements N leads to different overall sparsity values. Motivated by this, we extend the proposed AST algorithm to train multiple sub-nets with different N:M sparsity patterns all once. Specifically, the moune-and-regrow is performed based on the group-wise summed weight or gradient magnitude. Powered by the open-sourced Nvidia-ASP library, the convolution computation weight group size M divisible by 4 (e.g., 4, 8, 16). The proposed AST algorithm collectively trains four ResNet- 18 sub-nets with 2:4, 3:4, 7:8. and 15:16 sparse patterns, corresponding to 50%, 75%, 87.5% and 93.75% overall sparsity, respectively. Starting from scratch, the percentage of the N:M sparse groups is gradually increased from 0% 100% within each sub-net. Compared the individually trained dense model, the proposed AST scheme achieving up to 2.3x inference speed up on GPU with 4x less training efforts and negligible accuracy loss, as shown in Table 14 The inference time is measured on a Nvidia 3090 GPU with FP32 data precision.
be accelerated when the sparse
4.4
Ablation Study
AST with more than 3 sub-nets. We also verify the effectiveness of the proposed AST scheme for training different number of subset networks. Table 5 shows the CIFAR-10 performance of AST by training 2, 3, 4, 5 sparse sub-networks collectively based on ResNet-18. Compared to the individually-
baseline sparse model, training more sub-nets with AST reduces the total training effort with the cost of the marginally degraded accuracy, especially in the high sparsity models (e.g., 98% sparsity). Given the tradeoffs between accuracy and total training cost, training three sub-nets have the best overall performance. The flexibility of training large number of sub-nets all at once enables the proposed AST algorithm to select suitable architecture for different power budgets.
8

---

ABT Sub-Nel-1 34.9
AST Sub-Net-
Bub-Net-3 Grahet Baselins
Grahet Baseline
GraNet Baseline
94.0
Table 5: ResNet- training results of AST (160 epochs) with various sparsity values and num- bers of sub-nets for CIFAR-10 10 dataset.
50.0
CIFAR-10 Inference
34.1
80.1
93.00
$4.0
Dataset
CIFAR-I0 10 Acc. (%)
23.2
Sparsity - 98%
95.99
Sparsity 95%
Sparsity
Sparsity
Indiv. trained
2
4
53.90
52.8
sub-nets
sub-nets
sub-nets
sub-nets
Toining Effor
Minimum 2375-
Training Effort
Minimum
52.5
Training Effort
Minimum
70%
80%
94.94
95.11
94,88
94.75
94,81
94.79
94.73
94,81
20
35
Normalizod
Commalied Toral Training Effort
25
45
90%
94.93
94,85
94.88
94.67 94,26
94.63
94.47
Figure 8: AST with extended training effort on CIFAR-10 with wide ResNet-32 [66].
98%
94.50
94.76
94.22
94.18
Table 6: ImageNet accuracy with different sparsity combinations on ResNet-50.
Method
ImageNet-2012
ResNet-50
Dense Model Acc.
76.8
Prune Ratio
50%
90%
50%
95%
80%
90%
AST+GC (s 0%)
74.68
73.26
AST+GC (s 0%)
74.21
71.07
AST+GC (s = 50%)
73.2
73.1
AST+GC 80%)
72.6
72.5
AST sub-nets with different sparsity difference. Besides the collective training of highly sparse models, learning the largely-varied subset architectures is also essential. As shown in Table aad [ the proposed AST algorithm still able optimize the model performance with the large sparsity gaps (c.g., 50% vs. 95%).
Extended AST training efforts. As shown in Figure AST iteratively selects different sub-net for each bini-batch iteration. Even though the batch shuffling is employed to the training, each sub-net cannot be fully trained by the whole training set inside each epoch. A traaight-fowwrte solution is extending the total training efforts (epochs). Assume the unit training cost is 160 epochs (1x), we evaluate the AST performance with three wide ResNet-32 (26) sub-nets with different training effort, as shown in Figure g Compared to the individually trained GraNet [4] baseline (total=3x)) AST achieves the same accuracy in all three sparse models with only ~2x averaged total training effort,
5 Conclusion
In this work, we first propose Alternttng Sparse Training (AST) scheme to train multiple sparse neural networks all at once. Then, we demonstrate the benefits of gradient correction (GC) with theoretical analysis and experimental results. As one of the carliest research works in this domain, the AST algorithm achieves high accuracy and high training efficiency without any repeated or ensembled training steps [24]. Inspired by the nrune-and-regrow scheme [14), the proposed AST-GC scheme exploits multiple sparse sub-nets simultaneously while achieving comparable or even higher accuracy against the individually-trainn trained SoTA methods. Obtaining multiple models with AST brings practical benefits to the energy-driven hardware computation along with superior performance on accuracy.
Impact and limitation. In the current work, AST can work well on certain number of sub-nets. Extending it to support a larger number of sub-nets is an interesting direction. In addition, to enlarge the practical efficiency for both training and dynamic inference, we will further apply quantization and investigate more structured sparsity pattern upon AST. Currently, we have demonstrated the practical value for dynamic inference on GPU with Nividia Ampere architecture by using N:M structured sparsity pattern. We would like to further demonstrate the actual efficiency of AST on more hardware platforms, such CPU, mobile/loT devices, specific hardware accelerators, etc.
Acknowledgments. We thank the anonymous reviewers for their constructive comments. This work is supported in part by the National Science Foundation under Grant No. 1931871, No.2144751, No. 1652866, No. 1931871, No.2144751, and the Center for Brain-inspired Computing (C-BRIC) in JUMP, SRC program sponsored by DARPA.

---

References
[1] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In International Conference Computational Statistics, pages 177-186. 2010.
[2] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. m. In International Conference on Learning Representations, 2019.
[3] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint wwXivv1907.048/00 2019.
[4] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, 2020.
[5] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2018.
[6] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-onling stochastic gradient for tensor decomposition. In Conference on Learning Theory, 2015.
[7] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
[8] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124 2021.
[9] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. TAS-KAST:
K always sparse training. In Advances in Neural Information Processing Systems, 2020.
[10] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
[1]] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
[12] Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. In International Conference on Learning Representations, 2021.
[13] Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106. 14568, 2021.
[14] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems, 2021.
[15] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1-12, 2018.
[16] Amirkeivan Mohtashami, Martin Jaggi, and Sebastian U Stich. Masked training of neural networks with partial gradients. Proceedings of Machine Learning Research, 2022.
[17] Hesham Mostafa and Xin Wang- Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, 2019.
[18] Alex Nichol, Joshua Achiam, and John Schulman, On first-order meta-learning algorithms. arXiv preprint ArXivv1803.02991 02999, 2018.
10

---

[19] Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. AC/DC: Alternatnn compresseddiecopprssed training of deep neural networks. In Advances in Neural Information Processing Systems, 2021.
[20] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
[21] Li Yang, Zhezhi He, Yu Cao, and Deliang Fan. A progressive sub-network searching framework for dynamic inference. arXiv preprint wrXiv:200900668/ 2020.
[22] Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In IEEE/CVF International Conference on Computer Vision, 2019.
[23] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. BigNAS: Scaling up neural architecture search with big single-stage models. In European Conference on Computer Vision, 2020.
[24] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference Learning Representations, 2018.
[25] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, 2020.
[26] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework the edge. In Advances Neural Information Processing Systems, 2021.
[27] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning M fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2020.


Reviews:
[{"review_id": "rw2U3ZpBz0V", "reviewer": "Reviewer_sm86", "summary": "This paper proposes to alternatively train multiple sparse sub-nets to obtain a supernet that supports dynamic inference with varying sparse rates, termed AST. Experiments on ImageNet and CIFAR-10/100 validated the efficacy of the proposed AST, which is also characterized by efficient training compared with traditional supernet methods.\n", "questions": "1. For the gradient correction part, does Eq.5 work if the overlap of two subnets is very small?\n2. I recommend the authors provide experiments of structured sparsity to demonstrate the efficiency of AST against S-Net, US-Net, OFA.", "limitations": "The authors do not discuss the limitations of their work as well as the negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed grad correction mechanism is well-motivated.\n2. The manuscript is well-organized and the proposed AST is easy to follow.\n\nWeaknesses:\n1. The practical value of AST is highly limited. 1) It is a consensus that unstructured pruning methods hardly gain acceleration on general hardware. In contrast, the advantage of unstructured pruning falls into significant parameter saving (storage reduction) without performance compromise. However, AST trains multiple subnets towards \"dynamic inference\" for sparse networks, which is unreasonable since dynamically running sparse subnets can not save inference costs as structured methods do. Moreover, the resulted supernets even bring more parameter burden than classic sparse training methods. (Although the authors have provided N:M fine-grained experiments, the patterns are selected with 2:4, 3:4, 7:8, 15:16. Here 3:4, 7:8, 15:16 patterns can not be applied on real hardware as far as I can know. Also, the experiments are only conducted on CIFAR-10. Overall, it is not convincing to me.)\n2. Lack novelty. The key component of AST is combining prune-and-regrow and supernet-training. Despite the originality of gradient correction, the limited practical value also degrades the overall contributions of this paper.\n3. The performance of AST is poor, compared with SOTA sparse training methods like GraNet. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "iiSC_0Zbwc2", "reviewer": "Reviewer_x1xA", "summary": "This paper combines two recently emerging methods, sparse training and dynamic inference, to obtain multiple subnetworks with different sparsity levels. Compared with traditional sparse training methods, the proposed method (AST) is able to obtain various sparse NNs with one training run. Compared with dynamic inference,  AST does not need to perform forward and backward for every sub-net. \n\nI believe this paper, albeit short on technical novelty, does provide a sufficiently novel and important empirical evaluation and details analysis of sparse training and dynamic inference, and could be accepted - assuming the authors can address the issues I have detailed in my review.\n\n", "questions": "Please refer to the weakness below. ", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n(1) To the best of my knowledge, the combination of sparse training and dynamic inference is novel, which can further improve the efficiency of sparse training. \n\n(2) The paper is relatively clear and I can obtain the general idea easily after reading the introduction. \n\n(3) The experimental setting is clear and fair, demonstrating the effectiveness of the proposed methods.\n\n(4) The empirical study in this paper is extensive (e.g., Completely subset (CS) vs Non-disjoint (ND) ), providing insights for researchers who want to follow this topic. \n\n## Weakness\n\n(1) Several naive baselines are missing. This paper only compares with several sparse training baselines and Jointly-Trained, while some naive baselines are missing. On the top of my head, for instance,  (1) train a dense model, and prune to various sparsity with a short time of fine-tuning. (2) train a sparse model with sparse training methods (e.g., GraNet), and prune to various sparsity with a short time of fine-tuning. While this procedure involves fine-tuning, we can still keep the overall training time as same as AST by decreasing the pretraining time. \n\n(2) The accuracy of AST are relatively lower compared with the backbone GraNet on ImageNet. I can imagine that the performance of AST is lower than GraNet, since training one network is intuitively easier than training multiple networks. Is the performance degradation caused by the insufficient updates of each subnet? Since the author keeps the overall training time the same as GraNet, the overall training time of each is 1/N (N is the number of subnets) of the GraNet? \n\n(3) A more elaborate description between the backbone technique GraNet and AST is required to help readers understand the proposed method more clearly.\n\n(3) There are quite many typos and inconsistencies in the paper. A significant polishment is required to be published. i> ''s+r'' in Eq.2 should be  capital. ii> RigL and Rigl should be consistent. iii>  line 129, sub-net 1 < sub-net 2 < sub-net 3: the meaning of this comparison is vague. Does this mean sparsity or model size? iv> what is the difference between inner group and inter group? It would be better to add the inter group in Figure3 as well. v> figure2: I assume the sparsity of the subnet with different colors is different from each other, but green and blue have exactly the same number of connections. vi> the readability of the Tables in the paper is low. I suggest arranging different baselines according to their accuracy, e.g., in ascending order. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BWzU4XxcHhH", "reviewer": "Reviewer_MN8L", "summary": "This paper introduces an algorithm for obtaining multiple sparse neural networks with different sparsity levels in a single training run. The authors also propose a gradient correction strategy that contains projecting the gradient of a sub-net to the plane of the prior sub-net; the authors explain that this makes the direction of the gradient updates among various models consistent.", "questions": "Most of the concerns/suggestions have been discussed in the previous section. I add some minor comments here:\n- In some places, descriptions of methods/experiments are not clear. in Section 3 (or 2), I suggest authors add a formal definition of a sub-net. At first read, it is not clear how W_i are related, and they are coming from the same model. Adding a pseudo-code can help to better clarify this.\n- Typos: Lines 139 (,),233 (remains), 290 (iteratively) \n- Line 260, The results from the Appendix are better to be included in the paper. \n- Figure 8. It’s better to include the sparsity levels for each sub-network in the figure.\n- How many runs are performed for each experiment?\n- I believe [1] is a related work in terms of deriving multiple sparse networks at a single run.\n\n[1] Liu, S., Chen, T., Atashgahi, Z., Chen, X., Sokar, G., Mocanu, E., ... & Mocanu, D. C. (2021). Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106.14568.\n", "limitations": "Limitations are not discussed in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is efficient in terms of deriving multiple sparse networks during a single round of training. \n- Figures 2 and 3 describe the proposed methods clearly.\n- While the writing and the descriptions can be improved, the paper is well-organized.\n\nWeaknesses:\n\n- One of the main concerns is that while the method derives multiple sub-networks, the performance of each sub-network falls behind the competitors in several cases. The reported accuracies are mostly compared to a single method [23]. I think the results should be compared with more methods given similar training costs to evaluate the quality of the sub-network. E.g., authors can consider deriving subnetworks with similar sparsity levels during a single run of GraNet and compare the accuracies.\n- The training costs are given in terms of training rounds. However, as different methods can have different training costs, the costs should be reported using another metric, e.g., FLOPs. Memory costs of keeping the masks for each sub-network need to be discussed more clearly.\n- AST-GC requires extra computation at Equation 5, which needs to be considered in the discussions and FLOPs computation.  \n- Some points are referenced in the Appendix; however, they are missing. E.g., Line 147 regarding discussion on Equation 4. Line 388 Limitations of the work \nWhile the general idea is interesting, I believe the paper is not ready for publication yet. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "51Pmo2GceJg", "reviewer": "Reviewer_LUjM", "summary": "This paper presents a sparse neural network training algorithm. The technical highlights include:\n\n(1) The proposed algorithm adopts a scheme similar to Reptile, which sequentially trains several sparse sub-nets. \n\n(2) To handle the conflicting direction of weight updates among sub-nets. Gradient correction is conducted to alleviate the negative gradient inner product.\n\n(3)As for signal sparse subset training, two observations are reported that enabling the freedom of exploring unique architectures of each sub-net elevates the learning capacity of AST over the S-Net, and intermittent sparsity increment among sub-nets stabilizes AST process.", "questions": "1. If we view the training of each sub-network as a learning task. Is the overall framework a special case of Reptile?\n\n2. The effect of gradient correction is not clear. What will the “negative gradient” and “conflicting direction of weight updates among sub-nets ” bring about to the learning process? I haven’t seen sufficient reason that it is necessary to deal with the issue. The experiment comparison is only conducted on two Cifar datasets.  \n \n3. The two observations in Sect. 3.3 are not sufficiently justified in the experiment.\n\n4. Some critical technical details are missing. For example, is the final sparse network an ensemble of all trained sub-nets or the last sparse sub-net?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The idea that learning sparse sub-nets sequentially and recursively is interesting. \n2. The sparse network learned by the proposed algorithm has better performance than previous algorithms.\n3. The paper writing is good, with sufficient related work introduction.\n\nWeaknesses:\nMy major concerns are:\n1. Experiments are kind of weak.\n2. Some of the proposed technical points lack sufficient experiment verification.\n3. The algorithm novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept