Waste not, Want not: All-Alive Pruning for Extremely Sparse Networks
Authors: Daejin Kim
Hyunjung Shim
Jongwuk Lee
Venue:  (2021)
Keywords: Model compression
Network pruning
Iterative pruning
Dead connections

Abstract: Network pruning has been widely adopted for reducing computational cost and memory consumption in low-resource devices. Recent studies show that saliency-based pruning can achieve high compression ratios (e.g., 80-90% of the parameters in original networks are removed) without sacrificing much accuracy loss. Nevertheless, finding the well-trainable networks with sparse parameters (e.g., < 10% of the parameters remaining) is still challenging to network pruning, commonly believed to lack model capacity. In this work, we revisit the procedure of existing pruning methods and observe that dead connections, which do not contribute to model capacity, appear regardless of pruning methods. To this end, we propose a novel pruning method, called all-alive pruning (AAP), producing the pruned networks with only trainable weights. Notably, AAP is broadly applicable to various saliency-based pruning methods and model architectures. We demonstrate that AAP equipped with existing pruning methods (i.e., iterative pruning, one-shot pruning, and dynamic pruning) consistently improves the accuracy of original methods at 128× - 4096× compression ratios on three benchmark datasets.

Under review as a conference paper at ICLR 2021
WASTE NOT, WANT NOT: ALL-ALIVE PRUNING FOR EXTREMELY SPARSE NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Network pruning has been widely adopted for reducing computational cost and memory consumption in low-resource devices. Recent studies show that saliency- based pruning can achieve high compression ratios (e.g., 80-90% of the parameters in original networks are removed) without sacrificing much accuracy loss. Never- theless, finding the well tralntrraieble networks with sparse parameters (e.g.. < 10% of the parameters remaining) is still challenging to network pruning, commonly believed to lack model capacity. this work, we revisit the procedure of existing pruning methods and observe that dead connections, which do not contribute to model capacity, appear regardless of pruning methods. To this end, we propose novel pruning method, called all-alive pruning (AAP), producing the pruned networks with only trainable weights. Notably, AAP is broadly applicable to various saliency- based pruning methods and model architectures. We demonstrate that AAP equipped with existing pruning methods (i.e., iterative pruning, one-shot pruning, and dynamic pruning) consistently improves the accuracy of original methods at -096x compression ratios on three benchmark datasets.
INTRODUCTION
The sate--of-the-ard neural networks have shown remarkable performance gains on various down- stream tasks such as computer vision, natural language processing, and speech recognition. Because neural networks are typically overparameterized, they require high computational cost and memory consumption. Such nature inherently hinders the deployment of models with excessive parameters on low-end devices such as mobile, embedded, and on-device systems.
Network pruning (Reed 1993) is the prevalent technique to compress high-capacity models by removing unnecessary units such as weights/filters while maintaining the performance with minimal accuracy loss. Existing pruning methods can be divided into two categories. 1)
The first group
enforces the sparsity as model regularization (Chauvin 1988
Weigend et al. 1990: Ishikawa 1996.
Molchanov al. 2017a| Carreaaa-rrrppppaa
& Idelbayev 2018
Louizos et al. 2018). is theoretically
well-investigated and does not require network retraining. 2) Another group develops saliency criteria
to prune less important units (Mozer & Smolensky
1988 LeCun al., 1989 Karnin] 1990: Hassibi
1993 Han et al. 2015 Guo et al. 2016 Lee et al. 2019 Park et al. 2020 Evci et al. 2019). ct al.
Because of its simple operation and outstanding pruning performance, magnitude pruning (MP) (Han ct al. 2015 Narang et al. 20177 Zhu & Guptal 2018) is the most popular saliency-based pruning method. Recently, the effectiveness of MP is highlighted by the success of the lottery ticket hypothe- sis (Frankle & Carbin 2019) and learning rate rewinding (Renda et al. 2020), which achieves less than 1% accuracy loss even after pruning 90% of the parameters. However, all pruming methods show that the trade-off between sparsity and accuracy is significantly degraded, especially at the extremely high sparsity (Gale et al.] 2019 Liu et al. 2019 Blalock et al. 2020). Considering the explosive increase in the size of state-of-te ofotthh-rtt models for downstream tasks (c.g., GPT-3 (Brown al. (020) for machine translation and FixEfficientNee-L2 (Tovroon et al. 2020) for image classification), the effective pruning methods extreme compression ratios must be accomplished, and it is particularly crucial for adopting high-capacity models to low-resource devices (e.g. One Laptop per Child).
To break the performance bottleneck high compression ratios of the state-of-the-an pruning methods, we investigate the procedure of existing pruning methods. Surprisingly. it is revealed that all existing studies overlook the existence of dead neurons after pruning dead neurons the nodes

---

Under review as a conference paper at ICLR 2021
IMP Accuracy
dead connections
GeNeet300-100 MNIST
ResiNet-18 CIFAR-10
R5 Tiny-ImageNe
% of dead connections
% dead comnections
% of dead connections
Test Accunacy (3)
Test Accoracy (%)
Accuracy (5)
80
80
45
40
40
40
60
60
30
+
20
377
64%
128k
256k
512x
1034x
128x
20
2566
512x
10240
2040
4096x
324
64x
128%
2066
512x
1024x
Compression ratio
Compression
Compression ratio
Figure 1: Model accuracy and percentage of dead connections under varying compression ratios:
LeNet-300- 100 on MNIST, ResNet-18 CIFAR-10, and ResNet-50 on Imageee.. We used IMP with learning rate rewinding (Renda et al. 2020 as the base pruning method. The percentage of dead connections is calculated by the number of dead connections divided by the total number of connections in a given network.
with no input connections or output connections. That is, dead neurons make all their connected weights useless, called dead connections. Although Han et al. (2015) have already raised the potential issue of dead neurons, they anticipated that the dead neurons could be negligible after multiple iterative pruning. Unfortunately, we find that this problem is not as simple as their expectation.
deppics prediction accuracy (green line) and the percentage of dead connections (red line) over various compression ratios. For this empirical study, iterative magnitude pruning (IMP) with learning rate rewinding (Renda al., 2020) is employed the baseline pruning method. Then, we analyze the correlation between pruning accuracy and the occurrence of dead neurons on three benchmark datasets. We discover two meaningful findings: 1) The dead neurons cannot be successfully removed by iterative pruning, especially at high compression ratios. 2) The severe performance degradation at high compression ratios is strongly correlated with the number of dead connections, i.e., they are inversely proportional to each other.
Based on these valuable findings, we propose all-alive pruning (AAP), which improves the pruning performance by effectively eliminating dead connections. Specifically, we search dead neurons at the pruning stage by inspecting their gradient flows when zero gradients are passing through the node, we regard them as dead neurons. Once identified, dead neurons and corresponding dead connections (any weights linked from or to dead neurons) are removed together during the pruning procedure. Note that detecting gradient-bar based dead neurons be applied for complex model architectures with skip connections. weights with the highest saliency scores. As result, AAP constitutes all-alive subnetwork, ie., all connections in the subnetwork are kept as trainable weights.
we remove more weights than a sparsity threshold, we can revive some
To summarize, the key advantage of AAP is two-fold: AAP is versatile it is broadly applicable to various saliency-based pruning methods and model architectures by minimizing the loss of prediction accuracy. 2) AAP consistently improves the accuracy of the original pruning methods at high compression ratios and breaks the state-of-the-as performance on three benchmark datasets (i.e., MNIST, CIFAR- 10, and ImageNe)....
RELATED WORK
Network pruning utilizes the highly overparameterized nature of modern neural networks to com- press the model. Also, it provides meaningful insight into what leads to the success of neural networks; small subnetworks in the original network can achieve comparable performance and improve generalization effects (Arora et al. 2018).
Recent ad adnacs... In general. network pruning be categorized into two groups The first approach employs the loss function with regularization terms to enforce the sparsity (Chauvin 1988 Weigend al. 1990 Ishikawal 1996). Also, Molchanoo et al. (2017)) proposed variational dropout to produce
2

---

Under review as a conference paper at ICLR 2021
highly sparse networks and Louizos al. (2018) reparameterized network weights the product of weight and a stochastic gate variable. Carreira- Perpiiná & Idelbayev (0118) attempted to minimize the loss using the stochastic version of projected gradient descent.
The second approach utilizes the sensitivity the loss for weights (LeCun et al, 1989 Karnin 1990
Hassibi et al.. 1993). For aaliency-based criteria, |Han et al. (2013) proposed iterative pruning based on the magnitude of weights. Despite its simplicity, magnitude pruning shows significant performance gain at high compression ratios (Guo et al., 2016 Narang et al. 2017 Zhu & Guptal 2018). Recently, Park et al. 2020) proposed lookahead pruning (LAP) that considers the neighbors of connections. While iterative pruning requires pre- praitre networks, SNIP (Lee et al. 2019) and GraSP (Wang et al. 2020) proposed to prune the network at initialization. Also, dynamic pruning methods (e.g.. SET Mocanu et al., 2018), DSR (Mostafa Wang 2019), and Rigl. (Evci al. (019)) adjust the sparsity level with prune-and-gre
cycles. Comprehensive analysis and experimental results for network pruning can be found (Gale et al.) 2019) Liu al. 2019 Blalock et al., 2020p.
Detecting dead connections in network pruning. Pruning neurons/weights may result in unwanted side-effects, producing useless connections in the pruned networks. In structured pruning, Li al. (017) and Liu et al. (2020) attempted to eliminate dead neurons by merely considering the connections between adjacent filters. However, it is not applicable for complex networks with shortcut connections. In unstructured pruning, Han al. (2015) discussed the existence of dead weights but neglected them. Evci et al. (2019) proposed dynamic pruning, which iteratively updates the weights to identify more useful weights under given computing constraints. Although dynamic pruning eliminates the connections with relatively low importance, it still cannot correctly handle dead connections in each pruning stage. In contrast, this paper devises a new pruning method that removes dead connections and keeps all trainable connections in general networks, including shortcut connections and batch normalization.
3 PROPOSED METHOD: ALL-ALIVE PRUNING (AAP)
Given dataset D = (z((9, ( Ti=1 and a sparsity threshold (i.e., the number of non-zero weights), unstructured pruning can be formulated as the constrained optimization problem.
min Lic & ww min 1 Σ ((cow;)) s.t. C (0,1)) and ||c|lo S ST, c.w
c,W n
where m is the number of parameters, and n is the number of samples in D. Also, ((-) is the loss function, O is the Hadamard product, and || llo is the Lg norm. e E {0. 1)mh Th is the indicator variable to determine whether or not w is pruned. In other words, Cj indicates whether wy is active (cz = 1)
or inactive (cg = 0)-
In this work, we aim at improving saliency-bases based pruning, where the importance of weights in the network is measured by a pre-defined saliency criterion. Popular criteria include the magnitude of weights (Han et al.) 2015), the sensitivity of weights (Lee al. 2019), and the neighboring connections of weights Park et al. 2020). Given a saliency criterion, we calculate the importance of weight wy a saliency score sy; the higher the saliency score, the more important the weight is. Then, each element of c is expressed as cj if 8j E S and c, = 0 otherwise, where j E {1, (1,.... m) and S is set of larggstt scores associated with That is, connections are kept in the original network, and the rest of the connections are removed from the network.
3.1 LIMITATIONS OF EXISTING SALIRNCY- PRUNING
Depending training efficiency, prior studies in saliency-based pruning utilize either one-shot pruning or iterative praning. One-shot pruning solely crases selected connections at once and retrains the network. Meanwhile, iterative pruning alternates pruning and retraining cycles by gradually increasing the sparsity levels at multiple steps. Based on this process, iterative pruning can achieve higher accuracy than one-shot pruning for highly compressed networks.
However, pruning weights can lead to unexpected changes in network architecture, and affects the importance of weights. Although the pruned network can constitute the most useful subnetwork in
3

---

Note that, when of (1-1) = b (1-1) is also regarded as a dead neuron since we can reestablish the (1) Under review as a conference paper at ICLR 2021
terms of selected criteria, the subnetwork may include useless connections, called dead connections, as a result pruning.
To describe the emergence of dead connections, consider simple fully-connected network. Here, w/ a weight connecting two nodes dj (1-1) and ag between the (1 1)-th layer and the (1)-th layer.
Based on a backpropagation rule, the gradient of w is rewritten as follows:
OL(w;D) dul au
0a,2 aC(w; a; 07 and ag vp Σ
w 11 (1-1)
a
+ 657)
(2)
da))
where 010h = (L(w; D) /a and (-) is the activation function. Also, oj and b are an activation and bias of the j-th node at the (1)-th layer.
After pruning, the updated network can possibly have a dead neuron which appears when the neuron has zero input connections or zero output connections, i.e., either a; (!-1) or of0) is zero:
1) No input connections: When of (111) O, a. a!-")) apparently a dead neuron. According to Eq. 2 the gradient of all output weights !! are also zeros, and thus can be marked as dead connections.
network by transferring the bias b(-11
1) of a;
(1-1) the bias b.¹ of aj
2) No output connections: When 0 0, all input weights w!? connected ag and the biases Uj can be removed.
When the dead neurons are detected, we can also eliminate all useless connections, including the biases and the weights for batch normalization, associated with dead neurons. Note that the dead neurons in the fully-con culnecco layer can be extended filters in the convolutional networks.
3.2 BUILDING ALL-ALIVE SUBNETWORK
The key idea of our pruning method, namely all-alive praning (AAP), is build all-alive subnetwork during the pruning procedure. Toward this goal, we first sort all weights by saliency-basee criterion. We highlight that our pruning method can be applied various saliency-basee criteria. Once the weights with r-highest scores are determined, we check whether dead connections appear as side effect of removing the connections. That is, additional dead connections lead overly pruned networks. To meet the sparsity threshold т, we can revive some connections with the highest saliency scores among previously pruned connections and update c.
Specifically, the overall process of AAP consists of the following three phases. (In Appendix, Algorithm is the pseudo-code of detecting dead connections in the pruning process.)
1) Eliminating dead connections: Choose set of high-impacted weights according to a saliency- based criterion, and eliminate the rest weights as well as dead connections that are associated with pruned weights.
2) Reviving new connections: If the more number of weights are eliminated than sparsity threshold, we restore some connections with the highest saliency scores, where the restored connections have their original weight values.
3) Repeat steps 1-2 until the pruned subnetwork is converged to the all-alive subnetwork.
Figure F depicts AAP equipped with magnitude pruning (MP). When we prune four weights from the original network, eliminate the least important connections (dashed lines). While existing pruning methods include dead connections, we eliminate all the incoming (or outgoing) weights of a dead neuron dead connections (red color) and then revive alternative connections (green color). Finally, all remaining weights are active, thereby the pruned network being all-alive. Please refer to Algorithm 2 for the detailed procedure of AAP in the Appendix.
We observe that AAP has two key advantages. Firstly, all weights in the subnetwork are kept alive without dead connections. That is, utilize all connections in the subnetwork. Owing to this
4

---

Under review as a conference paper at ICLR 2021
Existing praning method
All-alive praning (AAP)
12
12
12
12
Prune less imporiani weights from the network.
Eliminale
Revive connections
Original netwurk
deud connections
Overall procedure of all-alive pruning (AAP) with a magnitude-based criterion. While existing pruning has dead neuron (red circle) and its dead connections (red line), AAP safely eliminates dead connections and revive other connections (green line).
Figure 2:
further compress networks; this is equivalent to improve the prediction accuracy under fixed compression ratios, particularly effective for pruning the networks at extremely high compression ratios. Secondly, since AAP eliminates all the weights associated with dead neurons, AAP results in the memory layout similar to those of structured pruning methods. As a positive byproduct of AAP, it is more compatible with modern memory architectures.
compact pruning,
4 EXPERIMENTAL RESULTS
We evaluate AAP with a variety of network architectures on MNIST, CIFAR-10, 10. and Tiny-ImageNe datasets. Concretely, we adopt LeNet-300-1 100 (LeCun et al. 1998), ResNet (He et al.) 2016). MobileNetV2 (Sandler et al., 2018). and EfficientNet (Tan & Le 2019). We employ iterative magnitude pruming (IMP) with learning rate rewinding (Renda et al. 2020). state-of-the-ar pruming method, base scheme of AAP. Also, we incorporate AAP into other saliency-based pruning methods such as SNIP (Lee et al. 2019), lookahead pruning (LAP) (Park et al.) 2020), and dynamic pruning inspired by Rigl. (Evci et al. 2019).
Implementation details. We use the same environmental setting for all experiments as proposed by [lalock et al.) (2020), including initialization values, super-parameters, and data augmentation settings. We use Adam optimizer, the batch size of 60, and the constant learning rate of 0.0003 with 50 training epochs for MNIST dataset. For CIFAR- 10 dataset, we use the batch size of 128, and the learning rate is initially set to 0.1 and is decayed by 0.1 at every 30 epochs. For Tiny- ImageNe dataset, we use the batch size 1024, and the learning rate linearly warmed up to its maximum value 0.4 at epoch 5 and decayed by 0.1 at epochs 30, 60, and 80. We use SGD with momentum of 0.9 and a weight decay rate of 0.0001 for both datasets and train them for 120 epochs and 90 epochs, respectively. For all experiments, we use 10% of the training set as the validation set.
For IMP, we rewind the learning rate schedule to the beginning the training and retrain from the final values of the weights. For iterative pruning, we prune 50% of the remaining parameters in each step. Exceptionally. since we use the constant learning rate for MNIST dataset, we use weight rewinding proposed by Frankle & Carbin (2019). For MNIST dataset, re-initialize the pruned model's 's weights to the value of the original model's initial weights.
Compression ratio. To measure the degree of network pruning, we adopt compression ratio, which calculated by original model size / compressed size. For instance, when only 1% of the parameters remaining in the compressed network, the compression ratio is 100x. Note that consider the number of all parameters, including the weights for batch normalization and biases.
4.1 EFFECT OF ALL-ALIVE PRUNING
We employ IMP with learning rate rewinding (Renda et al., (2200) and weight rewinding (Frankle & Carbin 2019) as the baseline pruning method due to their state-of- ofate-of-the-are the-art performances. At each pruning iteration, apply AAP after pruning the connections with the lowest magnitude scores. To
5

---

Under review as a conference paper at ICLR 2021
(a) LeNet-300-100 MNIST
(b) ResNet-32 CIFAR-10
100
100
Test Accuracy (88
80
Text Accuracy
60
40
40
16x
32x
64k
126x
250x
512x
10245
16x
256x
5124
Compression ratio
Compression ratio
(c) ResNet-18 CIFAR-I0
(d) ResNet-50 Tiny- ImageNett
100
60
Test Accuracy (%)
Test Accuracy
80
60
40
15
64x
128x
296x
$12x
1024
2048x
1096
16x
64x
1xx
250x
si2s
10248
Compression raiio
Compression raio
(e) Vo 2 Tiny- ImageNel
(1) ElliciemtNet-Bo Tiny- ImageNel
60
60
Text Accuracy (%)
Test Accuracy
50
50
40
30
30
32%
64%
16x
12x
64%
128K
Compression ratio
Compression ratio
IMP-AAP
K- IMP
Regions of AAP
Figure 3: Accuracy comparison between IMP and IMP-AAP across various model architectures and compression ratios by iterative pruning. We use weight rewinding for MNIST dataset and learning rate rewinding for the rest.
validate AAP can alleviate performance degradation high compression ratios, start to apply AAP to the point (gray colors in Figure B). where the performance of pruned networks begins to be worse than the original model. On the basis of IMP, pruning method, namely IMP-AAP, is compared with IMP in terms of the prediction accuracy under the same compression ratio to evaluate the effectiveness of AAP.
LeNet-300-100 on MNIST. LeNet-300- 100 consists of the two-layer fully connected (FC) networks with 300 and 100 nodes. We compute the test accuracies of IMP and IMP-AAP various compression ratios, starting from 16x to 1024x. Figure 3(a clearly showcases the performance gains by adopting AAP, particularly noticeable the compression ratio exceeds 64x. Specifically, IMP-AAP achieves gains of 10.92% at 512x and 32.25% at 1024x. From this result, we find two positive effects of AAP: 1) The impressive performance gains at the high compression ratios show the effectiveness of AAP. 2) The success of LeNet-300-10 represents that AAP is effective in fully connected layers.
ResNet-32 and ResNet-18 on CIFAR-10. We evaluate AAP on ResNet-32, and ResNet-18 18 for CIFAR-10 10 dataset. Figure 3bb) compares the test accuracies of IMP and IMP-AAP over the com- pression ratios from 8x to 512x on ResNet-32. Despite the rapid drop IMP performance after 128x, IMP-AAP slows down the degradation of IMP and always achieves higher accuracies than IMP under the same compression ratios. Figure (ic) depicts the experimental results from ResNet- 18, introducing AAP IMP-AAP improves IMP by 6% at an extremely high compression ratio 2048 x. The performance gaps are consistently in- creasing at higher compression ratios (e.g, greater than 512x). From both ResNet-32 and ResNet-
Interestingly, observe the significant benefits

---

Under review as a conference paper at ICLR 2021
One-shot MP
SNIP
LAP
LeNet-300- 100
Accuracy (8)
90
Accuracy (%)
90
Accuracy (96)
90
75
75
75
60
60
60
45
16x
256x
16x
2564
16x
256
Compression ratio
Compressinn ratio
Compression ratio
One-shot MP
SNIP
Dynamic pruning
Test Accumey (%)
80
Text Accuracy (9)
90
Text Accuracy (9)
90
ResNet-18
80
70
70
70
8
64x
128x
246
512x
1024x
16x
256x
16x
256
Compression maio
Compression miio
Compression ratio
AAP
Original
Figure 4: Accuracy of applying AAP with various saliency-basee pruning (i.e., one-shot magnitude pruning (One-shot MP) (Frankle Carbin 2019; Renda et al.j 2020), SNIP (Lee et al. 2019). lookahead pruning (LAP) (Park et 2020),, and dynamic pruning by Rigl. (Evci et al. 2019).
experiments, it is consistently observed that the pruned network by IMP-AAP often discards the entire ResBlock except for the skip connection. We conjecture that this is possible because the skip connection allows using the outputs from the previous layers while achieving high compression ratio. Overall, we confirm that AAP is also effective in compressing convolution layers and the complex structure even with skip connections.
ResNet-50, MobileNet and Efficient(Net-BO on Tiny-ImageNet. For Tiny-ImmgeNet dataset, we employ ResNet-50, MobileNetV2, and EfficientNet-BO as backbone networks. For ResNet-50, we observe that IMP-AAP improves the trade-off between sparsity and accuracy. When the compression ratio is 512x, IMP experiences severe performance drop, while IMP-AAP improves IMP by 7%. In Figures 3 (e) and we evaluate AAP on MobileNetV2 and EfficientNet-Bll to validate whether or not AAP is still effective in the compact backbone models (i.e. having fewer parameters). Due to the highly efficient nature of MobileNetV2 and EfficientNet-Bi both IMP and IMP-AAP suffer from performance degradation when pruning at high compression ratios (e.g., greater than 16x), However, IMP-AAP still consistently achieves a meaningful improvement on top of IMP. Concretely, IMP-AAP consistently enjoys the performance improvements about 1-2% the compression ratios greater than 8x for MobileNetV2. For EfficientNet-BO, the gain by IMP-AAP is also noticeable IMP-AAP shows 2% improvement on accuracy at 64x and 6% at 128x.
4.2 APPLICABILITY ON VARIOUS SALIENCY-BASED PRUNING
To evaluate the versatility of AAP, we apply AAP on top of existing pruning methods and see whether or not AAP can enjoy the performance benefits of others. We re-implement other saliency- based pruning, referring to the Py Torch implementations, i.e., SNIF" and LAF We apply the same criterion as the weights to the biases for SNIP and use global pruning for LAP. We also validate AAP in dynamic pruning setting, where the prune-and-grow strategy was introduced by Rigl. (Evci) al. 2019). Specifically, connections with the highest magnitude gradients for every 1500 steps until /4 of the full training epochs. We decay & using cosine update schedule, where the initial & is 30% of all connections in the network. Then, we apply AAP after each prune-and-grow cycle. Originally, RigL introduced the sparsity strategy for practical training, which assigns the target sparsity level at each layer for preserving low FLOPS. In this experiment, we prune and grow the weights globally, including biases
prune connections with the lowest magnitude weights then grow k
httraa://qitubbb.commmmiaaaa/a//
7

---

Under review as a conference paper at ICLR 2021
(a) One-shot MP without AAP
(b) One-shot MP with AAP
Figure Visualization of remaining weights after applying AAP to one-shot MP at 128x compression ratio using the same unpruned network. The second fully-connected layer (300 x 100) of LeNet-300- 100 on MNIST is chosen for the illustration. Each dot denotes the unpruned weight. Whereas 138 out of 300 neurons and 93 out of 100 neurons have unpruned connections in one-shot MP, only 85 out of 300 neurons and 44 out of 100 neurons have unpruned connections after applying AAP. Each achieves 73.23% and 93. 12% accuracy, respectively.
and weights for batch normalization unlike RigL. We employ LeNet-300-100 for MNIST dataset and ResNet- 18 for CIFAR-10 dataset as backhone networks.
For LeNet-300- 100, we conduct comparisons on the basis of one-shot magnitude pruning (One-shot MP), SNIP, and lookahead pruning (LAP). For ResNet-18, the experiment of LAP is conducted since our method considers all the parameters in the network, including those in skip connections and batch normalization; LAP does not fully consider skip connections and parameters associated with batch normalization. Instead, for ResNet-18. 18, we choose dynamic pruning as a base pruning scheme and apply AAP each grune-and-grow cycle. For one-shot MP. we apply weight rewinding on LeNet-300-100 and learning rate rewinding on ResNet-18.
Figure 4 depicts the efficiency of AAP over various saliency- based pruning. As expected, since one-shot pruning involves single pruning stage, any one-shot pruning methods and those with AAP are less effective than IMP. Nevertheless, applying AAP clearly improves various saliency-based, one-shot pruning methods. For ono-shot MP on LeNet-300- 100 and ResNet-18 18, we achieve 10% of the accuracy improvement at 128x (from 82.47% to 93.1 11%) and 6% of the accuracy improvement at 512x (from 76.6% to 82.3%), respectively. Like one-shot MP, AAP incorporates with SNIP well for both LeNet-300- 100 and ResNet- 18. At 256x, AAP outperforms the original SNIP by 33% on LeNet-300-100 (from 55.48% 88.47%) and 3% ResNet- 18 (from 75.85% 82.03%). On 100, LAP seems to preserve the good performance even in the high compression ratio; nonetheless, AAP can improve the performance by 12% (81.73% to 93.58% at 256x. Unlike SNIP or LAP, prune-and-grow policy in dynamic pruning leads to architectural changes in the network model. AAP is also successful in improving dynamic pruning: AAP achieves 15% improvements at 256x for dynamic pruning on ResNet-18 18. From these results, we confirm the efficacy of AAP independently of saliency-based pruning even with dynamic pruning.
In addition to the accuracy gain, ( between one-shot MP and the same one with AAP. While all active neurons from one-shot MP are spread arbitrarily, those from AAP tend to form clusters for more columns and rows. This notable because AAP does not require explicit or implicit constraints to cluster the weights. We observe that AAP produces small-dense subnetwork similar to structured pruning (Li et al. 2017] Molchanov et al., 2017b; Liu et al. 2200)), which is more favorable for modern memory and computing architectures even with unstructured pruning.
the weight distribution of the pruned network
CONCLUSION
While existing studies mainly focus on developing new saliency-based criteria or optimizing mag- nitude pruning methods, existing studies do not properly address the dead connections during the pruning process. In this paper, we propose simple-yet-effecttvv and versatile unstructured pruning method, namely all-alive pruning (AAP), to eliminate dead connections and make all weights in the subnetwork trainable. In experimental results, AAP consistently improves various saliency-basee based based pruning methods with different model architectures at 128x-4098 compression ratios, achieving Mate-ooftthe-ar performance on several benchmark datasets.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pp. 254-263, 2018.
Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag. What is the state of neural network pruning? CoRR, abs/2003.03033, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HeerbertVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 1406/2005.14165, 2020.
Miguel Á. Carreira-Perpitánn and Yerlan Idelbayev. Teaaning-compressinn algorithms for neural pruning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2532-8544 2018.
Yves Chauvin. A back-propagation algorithm with optimal use of hidden units. In Advances in Neural Information Processing Systems, pp. 519-526, 1988.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. CoRR, abs/1911.11111 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, abs/1902.09574, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dans. In Advances in Neural Information Processing Systems, pp. 1379-1387, 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. Neural Information Processing Systems 28, pp. 1135 143,, 2015.
Babak Hassibi, David G. Stork, and Gregory Wolff. Optimal brain surgeon and general network pruning. In International Conference on Neural Networks, pp.
1993.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Masumi Ishikawa. Structural learning with forgetting. Neural Networks, 9()): 521, 1996.
Ehud D. Karnin. simple procedure for pruning back-propagatio trained neural networks. IEEE
Trans. Neural Networks, 1(2):239-22) 242, 1990.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-6598 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, Yann LeCun, and Patrick Haffner. Gradient-bar based learning applied document recognition. Proceedings of the IEEE, 86(11) (1:1111:277222) 2324, 1998.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip S. Tom. SNIP: single-shot network pruning based connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Ning Liu, Xiaolong Ma, Zhiyuan Xu, Yanzhi Wang, Jian Tang, and Jieping Ye. Autocompress: An automatic DNN structured pruning framework for ultra-high compression rates. In AAAI Conference Artificial Intelligence, pp. 4876-4883, 2020.

---

Under review as a conference paper at ICLR 2021
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darell.. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 1_ regularization. In International Conference on Learning Representations, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, (1):2338, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In International Conference Machine Learning, pp. 2498-2507, 2017a.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017b.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, volume 97, pp. 4646-4655 2019.
Michael Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in Neural Information Processing Systems, pp. 107-115, 1988.
Sharan Narang, Greg Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. International Conference on Learning Representations, 2017.
Sejun Park, Jacho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude-based pruning. In International Conference Learning Representations, 2020.
Russell Reed. Pruning algorithms-a survey. IEEE Trans. Neural Networks, 4(5):740-745 1993.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 45 10-45200, 2018.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, volume 97, pp. 6105 61 6114, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy: Fixefficientnet. CoRR, abs/2003.08237, 2020.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Andreas S. Weigend, David E. Rumelhart, and Bernardo A. Huberman. Generalization by weight- climination with application to forecasting. In Advances in Neural Information Processing Systems, pp. 875-882, 1990.
Michael Zhu and Suyog Gupta. To prune, or prune: Exploring the efficacy of pruning for model compression. In International Conference on Learning Representations ( (Workshop), 2018.
10

---

Under review as a conference paper at ICLR 2021
A DETAILED ALGORITHMS OF THE PROPOSED METHOD
Finding dead connections. Given a binary mask vector с E {0, 1)m and the corresponding pa- rameter IR" find dead connection vector d 10, where d is if w/) is dead connection; de O otherwise. When the weight matrix higher than two-dimensional, e.g. filters in CNN, the first two dimensions are regarded as input and output dimensions. To simplify the notation, gi+ same as ga+.>* 0.
Algorithm is the pseudo-code of finding dead connections. First of all, we preprocess a network to check the gradient of weights. In this preprocessing stage, the activation function is regarded as an identity function. Besides, we set all weights to be positive and ignore all bias terms to avoid false positive neurons (i.e., detecting the false dead neurons, although not dead neurons). After that, we update the weights by considering the given mask vector and compute the gradient g(.) of w by taking the all-one vector as input. After the dead connection vector is initialized with a zero vector, we update the gradient for dead connections.
Pseudo-code of all-alive pruning. Algorithmiiis is the pseudo-code of the all-alive pruning algorithm. Given network, we compute the saliency score using arbitrary saliency criterion, e.g. magnitude pruning, SNIP (Lee et al., 2019), and lookahead pruning (LAP) (Park et al. 2020). Then we update using the saliency score, tee weights with the lowest scores are set to I. For the updated nub-network, we find the dead connections and recalculate the saliency scores accordingly. That is, the weight identified connections and making all-alive subnetwork by given saliency score until all connections are kept alive. Then prune the network with the finally obtained mask vector.
dead connection, its saliency score is set O. We repeat eliminating dead
Algorithm 1 Finding the dead connections
Input: Binary mask с E {0, 1)mh, Binary dead connections d E 1)m model parameters w E R Output: Updated binary dead connections d {0, 13"h corresponding to R.
Preprocess: Make all weights positive, remove all biases.
2: W < w ②
Update the pruned network with Update the gradient g for all-one input data.
3: Compute the gradient g(w; 1).
4: for I 1 to N do
5:
d
ij + if
gia = or B.) =0.
(1)
6: end for
Algorithm 2 All-alive pruning
Input: Model parameters E Thm sparsity threshold Output: All-alive subnetwork w
1: Compute saliency for w.
b Use an arbitrary saliency criterion.
2: Initialize binary dead connections d {0, 13 as zero.
3: repeat
4:
Stoopp connections by s.
5: Initialize
and update e for S. 6: Update dead connections d for w and c.
Set top-r connections in $ Cj
Refer to Algorithm
7: for 1 to N do
8:
0 if 1.
Update the score of dead connections as $ 0.
9: end for
10:
until all connections in is alive.
11: w f- w G
B MODEL ARCHITECTURE DETAILS AND HYPERPARAMETER SETTINGS
We modify the input and output layers of model architectures to handle smaller size of input data. For CIFAR-10, 10, we use 3 x 3 filter with stride of 1 at the first convolutional layer of ResNet. Because the image resolution of the Tmay-em dataset is larger than that of the CIFAR-10
11

---

Under review as a conference paper at ICLR 2021
dataset, we double the stride of the first convolutional layer. For MobileNetV2 and EfficientNet-Ba we use stride instead of 2 at the first convolutional layer. Also, we adjust the number of output nodes for all networks according to the number of labels for each dataset.
We re- re-implement SNIP and LAP, referring to the PyTorch implementations. We apply the same global pruning (i.e.prune the entire
criterion the weights the biases for SNIP. For LAP, network at once) instead of layer-wise pruning for fair comparisons with other pruning methods; the original LAP chooses layer-wise pruning. Whereas SNIP and LAP do not fully consider the biases and the weights for batch normalization, our AAP equally treats and removes all parameters in the network.
aParams
Training settings
Learning rate
Accuracy
Adam optimizer
MNIST
LeNet-300-100
267K
60
0.0012
97.76=0.19
SGD optimizer
0.1
0 <. S 30
ResNer-32
0.46M
Batch size: 128
0.01
30 60
90.14:0.69
CIFAR-10
ResNet-18
11.2M
Momentum: 0,9
0.001
60
90
91.98
Weight decay: 0.0001
0.001
ResNet-50
23.9M
0.4 $
0 t
53.55
SGD optimizer
0,1
5 30
Batch size: 1024
Tmy--mageNNet
MobileNerV2
2.5M
Momentum:
br
0,04
30 60
54.60=0.0
Weight decay; 0.0001
0.004
80
Ffficiem/Net-R0
4.3M
0.0004
90
56.2320047
Table 1: Hyperparameters and training schedules for each dataset. t indicates the training epochs. All accuracy for the original model and is the average of the three trials.
C DETAILED EXPERIMENTAL RESULTS
We report numerical results depicted in Figure 3 Note that all the results are computed by averaging three trials with different random seeds. In all tables, subscripts denote standard deviations, and bracketed numbers indicate the accuracy gains obtained by the proposed model over the original IMP.
32x
64
128x
256
512x
1024x
params
3.13%
1.56%
0.78%
0.39%
0.20%
0.10%
IMP
97.72=00.8
97.06:20.9
96.1820.1
91. a.99
79.80+2.6
47.35
97.11:021
96.44-1025
93.91 :0.44
90.72:10.9
79.60
IMP-AAP
(+0.05 ))
(+0.26 %)
(+1.97% %)
(+10.92%
(+32.255%) %)
Table 2: Accuracy of LeNet-300-100 on MNIST. The initial unpruned network has 267K parameters and 97.76% average accuracy.
16x
32x
64x
128x
256x
512x
% params
6.25%
3.13%
1.56%
0.78%
0.39%
0.20%
IMP
89.40=a.58
87. 37.0622449
82.33
73.574177
57.30=070
44,96+2.16
87.16++054
82.76
75.56
64.99 :144
53,99
IMP-AAP
(+0.10%)
(+0.43%)
(+1.99%)
(+7.69%)
(+9.03 %
Table Accuracy of ResNet-32 on CIFAR-10. The initial unpruned network has 0.46M parameters and 90.14% average accuracy.
12

---

Under review as a conference paper at ICLR 2021
128x
256x
512x
1024x
2048x
4096x
params
0.78%
0.39%
0.20%
0.10%
0.05%
0.02%
IMP
92.07
90.57 +1.38
87, 7,.77=0.3
80.96+
69.32.
41.92=11198
90.5611022 28
87.96=0.88
82.9810024
1.15
60.40
IMP-AAP
(-0.01%) %6)
(+0.49 %
(+2.02
(+6.18%)
(+18.48%)
Table 4: Accuracy of ResNet-18 on CIFAR-10. The initial unpruned network has 11.2 parameters and 91.98% average accuracy.
32x
64x
128x
256x
512x
1024x
% params
3.13%
1.56%
0.78%
0.39%
0.20%
0.10%
IMP
53.76=1.31
52.79+008
50.81
43.1540.8
20
16.31+229
52.70++109
51.11
44.79
36.22
25.70+
IMP-AAP
(-0.09%)
(+0.30%)
(+1.64%)
(+6.54%)
(+9.39 %
Table 5: Top-1 accuracy of ResNet-50 on -may-lmageNet. The initial unpruned network has 23.9M parameters and 53.55% average top-1 accuracy.
32x
64x
128x
256x
512x
1024x
params
3.13%
1.56%
0.78%
0.39%
0.20%
0.10%
IMP
77.152098
27.44
76.262073
70.61 1:047
27.01
38.90
77.10
76.48
71.53
63.88 (+6.87 %)
02.34
IMP-AAP
(-0.34%)
(+0.22%)
(+0.92
(+13.44%)
Table 6: Top-5 accuracy of ResNet-50 on Tiny-ImageNet. The initial unpruned network has 23.9M
parameters and 76.63% average top-5 accuracy.
2x
4x
8x
16x
32x
64x
params
50.00%
25.00%
12.50%
6.25%
3.13%
1.56%
IMP
55.60=0.z
55.22=a61
19
49.96
42, 22.13
32.47
53.12-0012
50.90
35.15 43.90=a21 (+1.77%)
IMP-AAP
(+0.93%)
(+0.94%)
(+2.68 %
Table 7: Top-1 1 accuracy of MobileNetV2 on Tiny- Imageee The initial unpruned network has 2.5M parameters and 54.60% average top-1 accuracy.
2x
4x
8x
16x
32
64x
params
50.00%
25.00%
12.50%
6.25%
3.13%
1.56%
IMP
79.49
79.58
77.44.0021
75.70
69.22:0.41
60.42
77.75=9.32
76.10
70.97
63.00
IMP-AAP
(+0.31
(+0.40%) %)
(+1.75%)
(+2.58 %))
Table 8: Top-5 accuracy of MobileNetV2 on Tiny-ImageNe The initial unpruned network has 2.5M parameters and 78.83% average top-5 accuracy.
D SPARSITY PATTERNS IN LENET-300-100
We visualize the change of the pruned weights in LeNet-300-100 over compression ratios (from 64x to 256x). ). In all figures, each dot denotes the unpruned weight the 300 100 matrix. To observe the differences between pruning methods, we perform one-shot MP in the same network. As we expected, all-alive pruning behaves similar structured pruning by removing dead neurons. Note
13

---

Under review as a conference paper at ICLR 2021
16x
32x
64
128x
% params
6.25%
3.13%
1.56%
0.78%
IMP
56.85=0.36
52.31
41.49+0.14
28,4644074
43.93
34.41:1026
IMP-AAP
(+2.44%)
(+5.95%
Table 9: Top-1 accuracy of EfficientNet-BO on Imny-ImageNet. The initial unpruned network has 4.3M parameters and 56.23% average top-1 accuracy.
16x
32x
64
128x
% params
6.25%
3.13%
1.56%
0.78%
IMP
80.37
17.69
69.61 1:043
55.93
70.62 1.035
62.06.10.04
IMP-AAP
(+1.01 %%)
(+6.13%)
Table Top-5 accuracy of EfficientNet-BO on Tmny-ImageNet. The initial unpruned network has 4.3M parameters and 80.24% average top-5 accuracy.
that the structured pruning makes the pruned sub-network be small-dense. Similarly, AAP also produces the sub-network to have small-dense structure because the dots are clustered along with either columns or rows if the black bars along with the vertical horizontal axis are scarce, the sub-network becomes small-dense. Interestingly, this tendency becomes prominent with an extremely high compression ratio (e.g (e.g.266x) as shown Figure 7 In other words, AAP achieves 89.65% accuracy even with weights connected with 36 out of 300 neurons and 29 out of 100 neurons.
(a) One-shot MP without AAP
(b) One-shot MP with AAP
Figure 6: Visualization of remaining weights in the second fully-connected layer of LeNet-300-11 100 after applying AAP one-shot MP at 64x compression ratio. 250 out 300 neurons and 99 out of 100 neurons have unpruned connections in one-shot MP, and 220 out of 300 neurons and 77 out of 100 neurons have unpruned connections after applying AAP, Each achieves 96.01 % and 96.48% accuracy after pruning, respectively.
in
ii Ii
(a) One-shot MP without AAP
(b) One-shot MP with AAP
Figure 7: Visualization of remaining weights in the second fully-connectee layer LeNet-300-10 after applying AAP one-shot MP at 256x compression ratio. 88 out of 300 neurons and 64 out of 100 neurons have unpruned connections in one-shot MP, and 36 out of 300 neurons and 29 out of 100 neurons have unpruned connections after applying AAP. Each achieves 38.06% and 89.65% accuracy after pruning, respectively.
14

---

Under review as a conference paper at ICLR 2021
E THEORETICAL STUDY OF ALL-ALIVE PRUNING
E.1 CONVERGENCE OF ALL-ALIVE PRUNING
In this section, we study the convergence the proposed all-alive pruning. In all iterations of AAP (Algorithm{) 2 mask all dead connections found in the previous iteration avoid reviving the dead connections. That is, the sub-network derived from one iteration of AAP should have smaller number of connections than the previous iteration, i.e., the number of connections in the sub-network excluding dead connections for t iteration is followed as:
0 < (2t)) n(6:-1) - - < n2(0-1),
(3)
where O, and D are the parameters in the sub-network and the found dead connections respectively at t iteration of AAP. Since the number of connections considered for each iteration decreases, AAP can always converge. It is observed that the 2-3 iterations are repeated in most cases.
E.2
ELIMINATING DEAD CONNECTIONS FROM LOWER COMPRESSION RATIOS
(a) LeNet-300--00 100 MNIST
(h) ResNet-18 CIFAR- 10
100
100
80
Text Accuracy (%)
90
Text Accuricy
60
80
40
70
20
16
12
256x
512%
60
16*
324
64x
256x
5128
I024X
Compression ratio
Compression ratio
MP-AAP
MP
MP-AAP (10%)
...... MP-AAP (5%)
MP-AAP (1%)
Figure 8: Accuracy of applying AAP incrementally in one-shot magnitude pruning. AAP(1%) denotes that the process of pruning a(0)-r connections and eliminating corresponding dead connections was 100
repeated in order to reach the final sparsity threshold r. Note that we only eliminate dead connections without any training process differs from the iterative pruning.
Without knowing the dependency of all connections, finding the optimal sub-network with the highest saliency scores without any dead connections is hard. All-alive pruning is proposed to obtain more efficient sub-network with the existing saliency criteria; nevertheless, it also cannot guarantee the sub-network with the highest scores. In this section, we deal with the more efficient sub-network by applying AAP incrementally from lower compression ratios (higher sparsity thresholds). With the given sparsity threshold r, the total saliency scores of the sub-networf obtained by AAP is followed as:
ST =
===
- TTp-((D- DT,,7),
(4)
where TopK (v, k) indicates the top-k elements of set v. Note that the saliency scores of the dead connections could be considered as zero they cannot properly join the regular training. we eliminate the dead connections for 7' > sparsity threshold first, then pruning up to again, the dead connections found for the final sub-network changed as:
D'' D D
(5)
since the dead connections found by the previous iteration decrease the removal of the connections of the sub-network needed for satisfying the sparsity threshold, in the same manner with the motivation
of AAP. It is obvious that the dead connections D D=+n(D"') caused by fewer pruned connections than
15

---

Under review as a conference paper at ICLR 2021
is subset of D The total saliency scores of the sub-network ST with the corresponding dead connections D are changed to:
ST' Σ TopK - 1,,))
Topk - DTtn(Dr)) ,,,7)
(6)
2 5" = Iop :
DD+a(D") CD..
According Eq. (6).
see that the dead connections eliminated in smaller units, the higher the total saliency scores can obtained; eliminating dead connections whenever prune each connection can obtain the sub-network, with much higher saliency scores. However, this does not mean that we can get an optimal network with the highest saliency scores, e.g., the connection with the lowest saliency score still be needed alive, and the existing saliency-based pruning methods do not allow to make it alive.
To validate the effectiveness of saving the saliency scores by applying AAP incrementally from the lower compression ratios (higher sparsity thresholds), we conducted the experiments in one-shot magnitude pruning. We satisfied the sparsity threshold over several pruning steps without any training process, but only with eliminating newly appearing dead connections. Figure 8 depicts the experiments of the AAP with smaller units. For example, AAP(1%) denotes that we repeat the
process of 1) prune 100 connections, then 2) eliminate the corresponding dead connections 100 times for satisfying the sparsity threshold Interestingly, we observe that the sub-network with the higher total saliency scores can achieve better accuracy at high compression ratios. Note that the existing pruning always outputs the same sub-network even with incremental pruning since they eliminate the connections with the lowest saliency scores without considering the effect of pruning on other connections.
n(8)-?
16


Reviews:
[{"review_id": "hywksm4uyD0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper talks about a novel network pruning method, called all-alive pruning (AAP), which seeks to effectively remove dead connections in a network. The proposed approach aims at enhancing the saliency-based pruning, and technically the approach searches for the dead neurons by inspecting their gradient flows.\n\nReason for score:\nOverall, I recommend accepting this manuscript. Although the proposed solution (AAP) is simple, the experimental results on several different saliency-based pruning scenarios consistently demonstrate its effectiveness and versatility. \n\nPros:\n- The motivation was effectively described and the problem was well-defined which made it easy to read. \n- versatile and applicable, state-of-the-art\n\nCons/Questions:\n- In Section 2, the authors seem to be differentiating between the terms \"useless connections\" and \"dead connections\". However, In 3.1, they make use of those two terms in a similar manner.\n- Regarding the pruning steps introduced in Sec. 3.2 (specifically the \"Repeat\" step), I would be interested in getting more information about the convergence during iteration. What would be the most significant factor for the convergence?\n- Authors repeatedly mention that one of the positive byproducts of AAP is that it is more compatible with modern memory architecture. Additional descriptions would be appreciated.\n- In Section 4.1, the authors \"conjecture that having the skip connections is advantageous when training after deleting the ResBlock\". A bit difficult to grasp the purpose of this sentence, along with the previous sentences. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P3f42vGHdGe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a novel pruning method, called all-alive pruning (AAP), which is a general technique to remove dead connections from pruned neural networks. AAP is broadly applicable to various saliency-based pruning methods and model architectures. AAP equipped with existing pruning methods consistently improves the accuracy of original methods on three benchmark datasets.\n\nStrengths\n1.\tThe motivation is very clear, and AAP is expected to improve existing pruning methods by removing dead connections especially with high compression ratios.\n2.\tThe authors perform various experiments on three benchmark datasets changing experimental settings such as classifiers, base pruning methods, and compression ratios.\n3.\tThe paper is written well. It is easy to understand, and the ideas and experiments are presented well.\n\nWeaknesses\n1.\tThere is no theoretical study of AAP. Algorithm 2 removes dead connections in a greedy manner by making the scores of dead weights as zero, even though they can be alive by revived connections at future iterations. Repeating Steps 1 & 2 of Section 3.2 until the unguaranteed convergence can make it remove important weights having high saliency scores at the worst scenario. Theoretical guarantees on the convergence of the algorithm and the scores of removed weights are needed.\n2.\tThe experiments are not practical. The proposed approach works well only with high compression ratios which degrade the performance of original models. Considering that the objective of model compression is to maintain the original accuracy requiring less resources, such high compression ratios that decrease the accuracy more than 5 points do not seem practical in real-world scenarios.\n3. The novelty is limited. Reviving dead connection is not a new idea.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NWRMC0DyVz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Summary ==\n\nThe submission deals with eliminating neurons in a network where either a) all the input connections xor b) all the output connections have been pruned. When this is the case, the unpruned a) output or b) input connections are unused and can also be pruned: and the freed parameter budget used for other more useful connections. This is shown to improve the accuracy of pruned networks at a given sparsity ratio, especially for very high levels of sparsity.\n\n== Strengths ==\n\n**Significance:** Nearly all neural network pruning approaches should consider this kind of improvement. Zombie (dead but un-pruned) connections in a pruned model will incur a cost with no benefit. As observed in the related work, some previous literature does explicitly ensure there are no dead neurons, but this is not universal. The authors propose a fairly general kind of approach, so it has potential for adoption fairly broadly across sparsity methods.\n\n**Experimental results** Overall, the experiments are well-executed and show a good effect. The effect of the proposed method, varying with the model size, on the accuracy is clearly shown. The results should be very reproducible, with explicit random seeding and multiple trials.\n\n**Clarity** The core method is described very straightforwardly and completely. Provided code is also very clear, and runnable out-of-the-box, and helped provide some answers.\n\n== Weaknesses ==\n\nWhile this is a well-executed investigation of the idea, I'm not convinced that using the gradient to detect dead neurons is the right approach.\n\n**Complexity** It's not a great deal of additional complexity, but it seems unjustified. Fig 2 itself illustrates that the \"dead\" connections could be found by directly inspecting whether the weights are non-zero. The most thorough explanation, that I can find, as to why the authors did not use the \"basic\" approach is that \"it is not applicable for complex networks with shortcut connections.\" But (Liu et. al. 2020), *do* consider a ResNet network. That paper also refers also to:\n\nHe et. al. \"Channel Pruning for Accelerating Very Deep Neural Networks\" ICCV 2017\n\nthat goes into how shortcuts affect pruning filters in more details.\n\n**Possible drawbacks** It seems possible that an activation or gradient may be zero in one iteration simply due to a filter not being active on examples in a *that minibatch*. So measured, a dead connection on this minibatch does not imply that it will be dead for all possible examples. Especially given that a filter may be preferentially responding to a particular kind of image structure, see for example:\n\nZieler & Fergus \"Visualizing and Understanding Convolutional Networks\" ECCV 2014\n\nIt has also already been observed that some pruning can essentially discard accuracy on example with rarer properties, to preserve performance on data more similar to the mode, see:\n\nHooker et. al. \"What Do Compressed Deep Neural Networks Forget?\"\n\nand it seems possible that the submission's method will tend to work this way to an even greater extent.\n\nThis also possibly isn't measured well by accuracy on small-scale experiments such as MNIST and Tiny-ImageNet: with a larger model and larger datasets I'd guess that probability is greater that a dead connection will be spuriously identified somewhere in the model.\n\nI'd especially appreciate any rebuttal that elaborates on why this method was chosen, rather than directly analyzing the elements of the weights $\\mathbf{w}$ or mask $\\mathbf{c}$ to determine if any node has all-zero input or output connections.\n\n== Misc Comments and Questions ===\n(not relevant to score)\n\n  * Grammar in \"we experiment with dynamic pruning instead of LAP is not compared since our method considers all the parameters in the network\" on page 8?\n  * Typo \"respecitvely\" on p8\n  * Why use np.abs in pruning/magnitude.py instead of torch.abs?\n  * What is the meaning of plus/minus notation in tables in Appendix C? Standard deviation? Difference between highest and lowest?\n  * Is it certain that the all-alive pruning will converge? I.e. is there an obvious reason that the outer loop in Algorithm 2 is known to always terminate in finite time?\n\n== Reason for Rating ==\n\nEliminating \"dead\" neurons, and not wasting the unused connections on them, is obviously a good choice when pruning a network. The central novel contribution of the paper, though, is in specifically how this is achieved. The method proposed in not completely justified by the submission.\n\n== After Rebuttal ==\n\nQ1 provides a helpful clarification. Some of the more concrete possible drawbacks listed above, especially those about pruning \"false positives,\" no longer seem as much of a concern.\n\nSome of the earlier investigation that the authors report, on more direct methods for finding dead connections, may need to be given a more complete treatment in the paper itself. I am not convinced that the direct methods cannot be done on arbitrary architectures, given that previous literature has managed to in a wide variety of examples. Without considering the simpler techniques, the leap to a more complex method for what could be a relatively simple task doesn't have the necessary support.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DeJ19Nouwwz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to remove dead neurons and their connected parameters through a very simple check while reviving pruned (salient) parameters up to the prespecified sparsity level, such that the sparse network obtained could perform better. The main (and perhaps the single major) contribution of this work is in its demonstration that such a simple method is indeed effective for different pruning methods on various network architectures and datasets. The proposed method (AAP) can perhaps be considered as a generic post-processing step that could be equipped to any pruning method leaving dead neurons.\n\nWhile the result is not unimportant, a few concerns remain. AAP is literally just checking for zero gradients (Line 6 in Algorithm 1) which by itself is very trivial and thus it remains questionable if this is really enough contribution as a sole idea for a conference paper (rather than for a workshop or so to speak); in other words, the idea is too generic that it could be done without having a read  of this paper. Moreover, except for visualization of remaining parameters, the idea is mainly demonstrated only for the performance, and it lacks in-depth analysis, ablations, and insights including for instance, why existing methods produce dead neurons, what’s the cost of AAP (or N value) in practice and theory, etc. So I’m leaning slightly towards rejecting it for limited novelty.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces All-Alive Pruning, an approach which checks for and removes the connections to and from units with zero gradient ("dead" units). The method is shown to improve performance of IMP at extreme (>128x) compression ratios on MNIST, CIFAR-10, and Tiny ImageNet. All reviewers felt that the problem the authors study -- how to identify and remove dead units -- is an interesting one. 

However, there were concerns about the practical utility of the method, given that AAP only improves performance for extreme compression ratios in which performance is already substantially degraded relative to unpruned models. I share these concerns, which mute the practical impact of this work. There were also concerns about a lack of proper baseline comparisons to more simple approaches to removing dead units. As mentioned by R1, given that the problem the study is an interesting one, the paper could make up for the lack of practical utility by providing detailed analyses of the settings in which dead units emerge, differences among pruning approaches, etc., but analyses provided here are limited. 

I would encourage the authors to explore these areas in a future revision of the paper, but recommend that the paper be rejected in its current form. 
Decision: Reject