AP: Selective Activation for De-sparsifying Pruned Networks
Authors: Shiyu Liu
Rohan Ghosh
Mehul Motani
Venue: Accepted by TMLR ()

Abstract: The rectified linear unit (ReLU) is a highly successful activation function in neural networks as it allows networks to easily obtain sparse representations, which reduces overfitting in overparameterized networks. However, in the context of network pruning, we find that the sparsity introduced by ReLU, which we quantify by a term called dynamic dead neuron rate (DNR), is not beneficial for the pruned network. Interestingly, the more the network is pruned, the smaller the dynamic DNR becomes during and after optimization. This motivates us to propose a method to explicitly reduce the dynamic DNR for the pruned network, i.e., de-sparsify the network. We refer to our method as Activate-while-Pruning (AP). We note that AP does not function as a stand-alone method, as it does not evaluate the importance of weights. Instead, it works in tandem with existing pruning methods and aims to improve their performance by selective activation of nodes to reduce the dynamic DNR. We conduct extensive experiments using various popular networks (e.g., ResNet, VGG, DenseNet, MobileNet) via two classical and three state-of-the-art pruning methods. The experimental results on public datasets (e.g., CIFAR-10, CIFAR-100) suggest that AP works well with existing pruning methods and improves the performance by 3% - 4%. For larger scale datasets (e.g., ImageNet) and state-of-the-art networks (e.g., vision transformer), we observe an improvement of 2% - 3% with AP as opposed to without. Lastly, we conduct an ablation study to examine the effectiveness of the components comprising AP.

Published in Transactions on Machine Learning Research (09/2023)
AP: Selective Activation for De-sparsifying Pruned Networks
Shiyu Liu
shiyp Bu Bus.u.us. edu
Department of Electrical and Computer Engineering College of Design and Engineering
National University of Singapore
Rohan Ghosh
rghosh92 com
Department of Electrical and Computer Engineering College of Design and Engineering
National University of Singapore
Mehul Motani
molani@nus.edue edu.sg
Department of Electrical and Computer Engineering College of Design and Engineering
N.1 Institute for Health
Institute of Data Science
Institute for Digital Medicine (WisDM) National Universily of Singapore
Reviewed on OpenReview: https: // openreview. metteeetuune forum? id= EGQSpkUDaD
Abstract
The rectified linear unit (ReLU) is a highly successful activation function in neural networks it allows networks to easily obtain sparse representations, which reduces overfitting in overparameterized networks. However, in the context of network pruming, find that the sparsity introduced by ReLU, which we quantify by term called dynamic dead neuron rate (DNR), is not beneficial for the pruned network. Interestingly, the more the network is pruned, the smaller the dynamic DNR becomes after optimization. This motivates us to propose method to explicitly reduce the dynamic DNR for the pruned network, ie., de-sparsify the network. We refer to our method Activate-whit while- Pruning (AP). We note that AP does function as a stand-alone method, as does not evaluate the importance of weights. Instcad, it works in tandem with existing pruning methods and aims to improve their performance by selective activation of nodes to reduce the dynamic DNR. We conduct extensive experiments using various popular networks (e.g., ResNet, VGG, DenseNet, Mo- bileNet) via two classical and three competitive pruning methods. The experimental results on public datasets (e.g., CIFAR-10, CIFAR-100) suggest that AP works well with existing pruning methods and improves the performance by 3% 4%. For larger scale datasets (e.g., ImageNet) and competitive networks (e.g., vision transformer), we observe an improvement of 2% 3% with AP opposed to without. Lastly, conduct ablation study and substitution study to examine the effectiveness of the components comprising AP.
1 Introduction
The rectified lincar unit (RcLU) (Glorot et al., 2011), (xx) max(x, 0}. is the most widely used activation function in neural networks (e.g., ResNet (He et al., 2016), Transformer (Vaswani el al., 2017)). The success of ReLU is mainly due to fact that existing networks tend to be overparameterized and ReLU can easily regularize overparamtterize networks by introducing sparsity (i.c., post-activation output is zero) (Li al., 2023: Denil et al., 2014; Wang et al., 2023), leading to promising results in many computer vision tasks (c.g.. image classification (He et al., 2016), object detection (Dai et al., 2021; Joseph et al., 2021))

---

Published in Transactions on Machine Learning Research (09/2023)
In this paper, we study the ReLU's sparsity constraint in the context of network pruning (i.., method of compression that removes weights from the network). Specifically, we question the utility of ReLU's sparsity constraint, when the network is no longer overparameterized during iterative pruning, In the following, we summarize the workflow of our study together with our contributions.
1. Motivation and Theoretical Study. In Section 3.1, we introduce a term called dynamic Dead Neuron Rate (DNR), which quantifies the sparsity introduced by ReLU neurons that are not completely pruned during iterative pruming. Through rigorous experiments on existing networks (e.g., ResNet (He et al., 2016)), we find that the more the network is pruned, the smaller the dynamic DNR becomes during and after optimization. This suggests that the sparsity introduced by ReLU is not beneficial for pruned networks. Further theoretical investigations also reveal the importance of reducing dynamic DNR for pruned networks from an information bottleneck (IB) (Tiihby& & Zaslavsky, 2015) perspective (see Section 3.2).
2. A Method for De-sparsifying Pruned Networks. In Section 3.3, we propose a method called while-tewhill-PPunnin (AP) which aims to explicitly reduce dynamic DNR. We note that AP does not function as a stand-alone method, as it does not evaluate the importance of weights. Instead, it works in tandem with existing pruning methods and aims to improve their performance by reducing dynamic DNR. AP has two variants: (i) AP-Lite which slightly improves the performance of existing methods, but. without increasing the algorithm complexity, and (ii) AP-Pro which introduces additional retraining step to the existing methods in every pruning cycle, but significantly improves the performance of cxisting methods.
3. Experiments. In Section 4, we conduct experiments on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) with various popular networks (e.g., ResNet, VGG (Simonyan & Zisserman, 2014), MobileNet (Sandler et al., 2018), DenseNet (Huang ct al., 2017)) using two classical and three competitive pruning methods. The results demonstrrte that AP works well with existing pruming methods and improve their performance by 3% 4%. For the larger scale dataset (e.g., ImageNet (Deng et al., 2009)) and competitive networks (e.g-: vision transformer (Dosovitskiy et al., 2020)), we observe improvement of 2% 3% with AP as opposed to without.
4. Ablation Study. In Section 4.3, we carry out an ablation study to further investigate and demonstrate the effectiveness of several key components that make up the proposed AP.
5. Substitution Study. In Section 4.4, we conduct substitution study replace certain components in the proposed AP and examine the impact pruning performance.
2 Background
Network pruning is a method used to reduce the size of the neural network, with its first work (LeCun et al., 1998) dating back to 1990. In terms of the pruning style, all existing methods can be divided into two classes: (i) One-Shot Pruning and (ii) Iterative Pruning. Assuming that we plan to prune Q% of the parameters of a trained network, a typical pruning cycle consists of three basic steps: (i) Prune 17% of existing parameters based on given metrics (ii) Freeze pruned weights as zero (ifi) Retrain the pruned network to recover the performance. In One-Shot Pruning, 17 is set to Q and the parameters are pruned in one pruning cycle. While for Tterative Pruning, a much smaller portion of parameters (i.e., " << Q) are pruned per pruning cycle. The pruning process is repeated multiple times until Q% of parameters are pruned. As for performance, Iterative Pruning often results in better performance compared to One-Shot Pruning (Li et al., 2017; Vysogorets & Kempe, 2023; Zhang & Freris, 2023). So far, existing works aim to improve the pruning performance by exploring cither new pruning metrics or new retraining methods.
Pruning Metrics. Weight magnitude is the most popular approximation metric used to determine less useful connections; the intuition being that smaller magnitude weights have smaller effect in the output, and hence are less likely to have an impact on the model outcome if pruned (He et al., 2020; Li et al., 2020a;b). Many works have investigated the use of weight magnitude as the pruning metric (Han et al., 2015; Frankle & Carbin, 2019). More recently, Lee et al. (2020) introduced layer-adaptive magnitude- based pruning (LAMP) and attempted to prune weights based on a scaled version of the magnitude. Park et al. (2020) proposed a method called Lookahead Pruning (LAP), which evaluates the importance of weights based on the impact of pruning neighbor layers. Another popular metric used for pruning is via the gradient, the intuition
2

---

Published in Transactions on Machine Learning Research (09/2023)
being that weights with smaller gradients are less impactful in optimizing the loss function. Examples are (LeCun et al., 1998; Theis et al., 2018), where LeCun et al. (1998) proposed using the second derivative of the loss function with respect to the parameters (i.e., the Hessian Matrix) as a pruning metric and Theis et al. (2018) used Fisher information to approximate the Hessian Matrix. A recent work (Blalock et al., 2020) reviewed numerous pruming methods and suggested two classical pruning methods for performance ovaluation: (i) Global Magnitude: Pruning the weights with the lowest absolute value anywhere in the network and (ii) Global Taylor (Molchanov et al., 2019): Pruning the weights with the lowest absolute value of (weight gradient) anywhere in the network.
1. Global Magnitude: Pruning weights with the lowest absolute value globally (anywhere in the network).
2. Global Taylor: Pruning weights with the lowest absolute value of (weightxgradient globally.
Retraining Methods. Another factor that significantly affects the pruning performance is the retraining method. For example, Han et al. (2015) trained the unpruned network with a learning rate (LR) schedule and retrained the pruned network using a constant learning rate (i.c., often the final LR of the LR schodule). A recent. work (Renda el. al., 2019) proposed learning rale rewinding which used the same learning rate schedule to retrain the pruned network, leading to a better pruning performance. More recently, Liu et al. (2021a) attempted to optimize the choice of LR during retraining and proposed & LR schedule called S-Cye, They showed that S-Cyc could work well with various pruning methods, further improving the existing performance. Most notably, Frankle & Carbin (2019) found that resetting the unpruned weights to their original values (known as weight rewinding) after each pruning cycle could lead to even higher performance than the original model. Some follow-on works (Zhou et al., 2019; Renda et al., 2019; Malach et al., 2020; Evci et al., 2021) investigated this phenomenon more precisely and applied this method in other fields (e.g., transfer learning (Mchta, 2019), reinforcement learning and natural language processing (Yu et al., 2020)) while other works (Evei et al., 2022; Paul et al., 2022) study its limitation and attempt to improve on it. One interesting work to mention is (Chen et al., 2022), which further examined the lottery ticket hypothesis from other perspectives, such as interpretability and geometry of loss landscapes.
Other Works. In addition to works mentioned above, several other works also share some deeper insights on network pruning (Liu et al., 2019; Zhu & Gupta, 2018; Li et al., 2022; Wang et al., 2022; Peste et al., 2021; Lee & et al, 2023; Gale et al., 2019). For example, Wang et al. (2020) suggested that the fully-trained network could reduce the search space for the pruned structure. More recently, Luo & Wu (2020) addressod the issue of pruning residual connections with limited data and Ye et al. (2020) theoretically proved the existence of small subnetworks with lower loss than the unpruned network. You et al. (2022) motivated the use of the affine spline formulation to analyze recent pruning techniques. Liu et al. (2022a) applied the network pruning technique in graph networks and approximated the subgraph edit distance.
3 Activate-while-Pruime
In Section 3.1, we first conduct experiments to evaluate the DNR during iterative pruning. Next, in Section 3.2, we link the experimental results to theoretical studies and motivate Aetivale-whill--rrnnngg (AP). Tn Section 3.3, we introduce the idea of AP and present its algorithm. Lastly, in Section 3.4, we illustrate how AP can improve on the performance of existing pruning methods.
3.1 Experiments on DNR
We study the state of the ReLU function during iterative pruning and introduce a term called Dead Neuron Rate (DNR), which is the percentage of dead ReLU neurons (i.e., neuron with post-ReLU output of zero) in the network averaged over all training samples when the network converges. Mathematically, the DNR can be written as
n
# of dead ReLU neurons
DNR-
(1)
n
all ReLU neurons in the unpruned network
# of dynamically + statically dead ReLU neurons all ReLU neurons in the unprumed network
(2)
3

---

Published in Transactions on Machine Learning Research (09/2023)
Dead Neuran Rate (DNR)
20%
Dynamic
35%
Pruned Network (). 80%)
Static DNR
6.1%
30%
Pruned Network 40.9%)
18%
6.3%
Dynamic DNR
25%
Pruned Network (). 26.2%)
6.7%
Before Training
10%
7.0%
20%
Convergence
15%
8%
10%
80%
51.2%
20.9%
10
20
00
90
100
110
Percent of Weight Remaining,
Training Epochs
Figure 1: Left: Dynamic and static Dead Neuron Rate (DNR) when iteratively pruning ResNet-22 using Global Magnitude; Right: The corresponding dynamic DNR during optimization.
where n is the number of training samples. We further classify dead neuron as either dynamically dead statically dead. The dynamically dead neuron dead neuron in which not all of the weights have been pruned. Hence, it is not likely to be permanently dead and its state depends neuron can be dead for sample X, but it could be active (i.c., post-ReLU output > 0) for a sample Y. The DNR contributed by dynamically dead neurons referred to as dynamic DNR. The statically dead neuron is a dead neuron in which all associated weights have been pruned. The DNR contributed by statically dead neurons referred to as static DNR.
its input. As example,
Related Works. The phenomenon of dead ReLU neurons a widely studied topic and inspire many interesting works Sokar et al. (2023); Dohare et al. (2021). DNR is term that introduce to quantify the sparsity introduced by ReLU. We note that DNR is closely related to activation sparsity (Kurtz & et al, 2020; Georgiadis, 2019) in the literature and in many scenarios, they can be interchangeable. The only difference could be that DNR a more explicit term which quantifies the percent of dead ReLU neurons in the network. In the literature, many similar sparsity metrics have also been proposed (Hurlcy & Rickard, 2009). As an example, the Gini Index (Goswarii al., 2016) computed from Lorenz curve (i.e., plot the cumulative percentages of total quantities) can be used evaluate the sparsity of network graphs. Another popular metric will be Hoyer measure (Hoyer, 2004) which is the ratio between L1 and L2 norms, can also be used to evaluate the sparsily of networks. Another interesting metric to mention will be parameter sparsity (Goodfellow et al., 2016) which computes the percentage of zero-magnitude parameters among all parameters. Both parameter sparsity and DNR will contribute to sparse representations, and in this paper, we use DNR to quantify the sparsity introduced by ReLU.
Experiment Sctup and Observations. Given the definition of DNR, static and dynamic DNR, we conduct pruning experiments using ResNet-20 on the CIFAR-10 dataset with the aim of examining the benefit (or lack thereof) of ReLU's sparsity for pruned networks. We iteratively prune ResNet-20 with pruming rate of 20 (i.e., 20% of existing weights are pruned) using the Global Magnitude (i.e., prune weights with the smallest magnitude anywhere in the network). We refer to the standard implementation reported in (Renda et al., 2019) (i.e., SGD optimizer (Ruder, 2016), 100 training epochs and batch size of 128, learning rate warmup to 0.03 and drop by a factor of 10 at 55 and 70 epochs, with learning rate rewinding (Renda et al., 2019), but without weight rewinding (Frankle & Carbin, 2019)) and compute the static DNR and dynamic DNR. while the network is iteratively pruned. The experimental results are shown in Fig. where make observations as follows.
1. As shown Fig. 1 (left), the value of DNR (i.e., sum of static and dynamic DNR) increases as the network is iteratively pruned. As expected, static DNR grows as more weights are pruned.
2. Surprisingly, dynamic DNR tends to decrease as the network is iteratively pruned (see Fig. (left)), suggesting that pruned networks do not favor the sparsity of ReLU. In Fig. (right), for pruned networks with different X (i.e., percent of remaining weights), they have similar dynamic DNR beginning, but the pruned work with smaller tends have smaller dynamic DNR during and after optimization.
4

---

Published in Transactions on Machine Learning Research (09/2023)
Result Analysis. One possible reason for the decrease in dynamic DNR could be due to the fact that once the neuron is dead, its gradient becomes zero, meaning that it stops learning and degrades the learning ability of the network (Lu et al., 2019; Arnekvist et al., 2020). This could be beneficial as existing networks tend to be overparamcterized and dynamic DNR may help to reduce the occurrence of overfitting. Howover, for pruned networks whose learning ability are heavily degraded, the dynamic DNR could be harmful as a dead ReLU always outputs the same value (zero as it happens) for any given non-positive input, meaning that it takes no role in discriminating between inputs. Therefore, during retraining, the pruned network attempts to restore its performance by reducing its dynamic DNR that the extracted information can be passed to the subsequent layers. Similar performance trends can be observed using VGG-19 with Global Taylor (see Fig. 4 in the Appendix). Next, we present a theoretical study of DNR and show its relevance to the network's s ability to discriminale.
3.2 Theoretical Insights: Relevance to Information Bottleneck and Complexity
Here, we present some theoretical results and subsequent insights that highlight the relevance of the dynamic DNR of a cortain layer of the pruned network to the Information Bottleneck (TB) method proposed in (Tishby & Zaslavsky, 2015). In the IB setting, the computational flow is denoted as X - - Y, where X represents the imput, represents the extracted representation, and Y represents the network's output. In (Tishby & Zaslavsky, 2015), the authors observed that the training of neural networks is essentially a process of minimizing the mutual information (Cover & Thomas, 2006) between X and T (denoted as ((;T)) while keeping T(Y; TT) large (precisely what IB suggests). A consequence of this is that over-compressedd features (very low [();T)) will not retain enough information to predict the labels, whereas under- coder-coepressed features (high T();T)) imply that more label-irrelevant information is retained in T which can adversoly affect generalization performance. Next, we provide few definitions.
Definition 1. SpeerfSp dynamic DNR (Drvn(T)): We are given dataset S = (X1, Xm): where X; ~ P vi (ii.d) and P is the data generating distribution. We denote the dynamic DNR of the neurons at a certain layer within the network represented by the vector T, by D DTnn(T). Dexa(T) is computed over the entire distribution of input in P.
Definition 2. Layer-Specific static DNR (Soxa(T)); In the same manner as Dnwa(T); we define the layer-specific static DNR of a network layer T.,
With this, we now outline our first theoretical result which highlights the relevance of D Dove(T) and Sosa (T) to [(; as follows.
Theorem 1. We are given the computational flow X + T - Y, where T represents the features at some arbitrary layer within a network, which are represented with finite precision (e.g., float32 or float64). We consider the subset of network configurations for which (a) the activations in T are less than a threshold 7 and (b) the zero-activation probability of each neuron in T upper bounded by some ps 1. Let dim(T) represent the dimensionality of T, i.e., the number of neurons at that depth. We then have,
I(X:T) S C x dim(T) x (Tx) - D Down (T)
1 -SSan(TT) Down (T)
C log
(3)
for finite, independent constant C' that only depends on the network architecture, T and ps.
The following corollary addresses the dependencies of Theorem 1, The proof of Theorem 1 and Corollary 1 are provided in the Appendix.
Corollary 1. In the setting of Theorem 1, the right hand side of equation decreases as D Down (T) or Soxa (T) increases.
Remark 1. (Relevance to Complexity) We see that (Shamir et al., 2010) notes how the metric I(X;T) represents the effective complerity of the network. As Theorem 3 in (Shamir et al., 2010) shows, I(X;T) captures the dependencies between X and T and directly correlates with the network's ability to fit the data. Coupled with the observations from Theorem and Corollary 1, for fixed pruned network configuration
5

---

Published in Transactions on Machine Learning Research (09/2023)
Pruning Method X
p %
p %
p %
Percent of Weights Pruned by Pruning Method
2
3
Pruning Cycles
Pruning Method X with Activate-whiie Prani--rrn-n-gnnnnnggg------- (AP)
p-q 9%
p-q%
9%
p-q %6
9%
p-q %6 9%
Percent of Weights Pruned by the Proposed
2
3
Pruning Cycles
Figure 2: Illustration of how AP works in tandem with existing pruning methods (e.g., method X).
(i.e., fixed (T)) greater Dave (T) will likely reduce the effective complerity of the network, undermining the function iunction-fitting ability of the neural network.
Remark 2. (Motivation for AP) Theorem 1 also shows that pruned network, which possesses large (Tve(T), leads to a higher risk of meer-compression of information (low [)) To address this issue, we can reduce the dynamic DNR (from Corollary 1) so that the upper bound of T(X;)) can be increased, mitigating the issue of over-coppression for a pruned network. This agrecs with our initial motivation that the spansily introduced by ReLU is not beneficial for the pruned net and reducing dynamic DNR helps in avoiding comp-seeeee features while simultaneously increasing the effective complexity of the network.
3.3 Algorithm of Activate-while- Paaiilee-runnng
The experimental and theoretical results above suggest that, in order to better preserve the learning ability of pruned networks, a smaller dynamic DNR preferred. This motivates us propose Sctivate-hhil--rmmine (AP) which aims to explicitly reduce dynamic DNR.
We note that the proposed AP does not work alone, it does not evaluate the importance of weights, Instead, it serves as booster to existing pruning methods and help to improve their pruning performance by reducing dynamic DNR (see Fig. 2). Assume that the pruning method X removes p% of weights in every pruming cycle (see the upper part in Fig. 2). After using AP, the overall pruning rate remains unchanged as p%, but 9 weights are pruned according to the pruning method X with the aim of pruning less important weights, while 9% of weights are pruned according to AP (sce the lower part in Fig. 2) with the aim of reducing dynamic DNR. Consider network f() with ReLU activation function. Two key steps reducing dynamic DNR summarized as follows.
(1) Locate Dead ReLU Neurons. Consider neuron in the hidden layer with ReLU activation function, taking n imputs (X,W,,... W, E R is the input and W2 E R is the associated weight}. Let j be the pro-activated output of the neuron (i.c., j Et X,W,) and J be the post-activated output of the neuron (I (f)) Let L be the loss function and assume the neuron is dead (J = =),, then the gradient of its associated weights (e.g., Wi) with respected the loss function will be dc ac 27 8j 0
aW. aJ 8, SW,
az 0. If neuron is often dead during training, the weight movement of its associated weights is likely to oj
be smaller than other ncurons. Therefore, we compute the difference between weights at initialization (fv) and the weights when the network convergences (0.), i.e., 10. 0o| and use it heuristic to locate dead ReLU neurons.
(2) Activate Dead ReLU Neurons. Assume we have located dead neuron in the hidden layer with n inputs [X,W,, X..,,,,WW, W,W, Xx, E IR is the input and W. E I is the associated weight). We note that X; is negatiiee as X, usually the post-activated output from the previous layer (i.e., the output of ReLU is non-nogative). Therefore, straightforward way to activate the dead neuron is prune the weights with the negative value. By pruming such negative weights, we can increase the value of the pre-activation output, which may turn the pre-activation output into positive to reduce dynamic DNR.

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 1 The Pruning Metric of the Proposed AP
Require: (i) Network f with unpruned weights 8g at initialization, f(0o); (ii) Network f with unpruned weights 0, at convergence, f(0.); (iii) Pruning Rate of AP, 9;
Locate Dead Neurons: Sort 10, Ool in an ascending order.
Activate Dead Neurons: In the ascending order of 8. - dol, prune first 9% negative weights.
Algorithm 2 The Pruning Method X with and without AP
Require: (i) Network, f(0); (ii) Pruning Rate of Method X, P; (iii) Pruning Rate of AP, gi (iv) Pruning
Cycles, n; (v) Pro Flag- = {0: AP-Lite, I: AP-Pro};
The Conventional Pruning Method X
for to do Randomly initialize unpruned weights, 9 Ou- Train the network to convergence, arriving parameters 0.. Prune p % of 0, according to the pruning method X.
end for
Retrain: Retrain the network to recover its performance.
The Conventional Pruning Method X with Proposed AP
for i= 1 to n do Randomly initialize unpruned weights, 0 + 6u-
Train the network to convergence, arriving parameters 0.. Prune (p q) % of 0. according to the pruning method X.
if Pro Flag then
# Execution of AP-Pro (i) Pruning: Prune % of parameter 0₂ according to the metric of AP (see details in Algo. 1).
(ii) Weight Rewinding: Reset the remaining porameters to their values in On-
(iii Retrain: Retrain the pruned network to recover its performance.
end if
end for
if NOT Pro Flag then
Execution of AP-Lite (i) Pruning: Prune % of the parumelers 0, according to the metric of AP (see details in Algo 1).
(ii) Weight Rewinding: Reset the remaining parameters to their values in Oo.
(iii) Retroin: Retrain the pruned network to recover its performance.
end if
3.4
How AP Improves Existing Methods
We now summarize how AP can improve existing pruming methods in Algorithm 2, where the upper part is the algorithm of standard iterative pruning method called pruning method X and the lower part is the algorithm of method X with AP. The proposed AP has two variants: AP-Pro and AP-Lite. We note that both AP-Pro and AP-Lite contain the same three steps, sunmarized as follows.
Step 1: Pruning. Given network al convergence with set of dynamically dead ReLU neurons, Ni (nI, 12, ..... The pruning step of AP aims to activate these dynamically dead ReLU neurons by pruning negative weights (i.e., see Algorithm 1), so as to preserve the learning ability of the pruned network.
Step 2: Weight Rewinding. Resetting unpruned weights to their values at the initialization. We note that different weight initializations could lead to different sets of M. In step 1, AP aims to reduce dynamic DNR for the target M1 and weight rewinding attempts to prevent the target M₁ from changing too much. Since the weights of ReLU neurons in Mr have been pruned by AP, these neurons could become active during retraining. The effect of weight rewinding is evaluated via an ablation study.
Step 3: Retraining. Retrain the pruned network recover performance.
AP-Lite and AP-Pro. The key difference between AP-Lite and AP-Pro that AP-Lite applics these three steps only once at the end of pruming. It aims to slightly improve the performance, but does not substantially increase the algorithm complexity. For AP-Pro, it applies the three steps above in every pruning cycle, which
7

---

Published in Transactions on Machine Learning Research (09/2023)
increases the algorithm complexity (mainly due to the retraining step), but aims to significantly improve the performance, which could be preferred in performance oriented tasks.
Difference with Existing Works. The AP is a pruming method which does not work alone as AP's pruming metric cannot evaluate the importance of weights (verified in Section 4.3). AP works in tandem with existing pruning methods and help to further pruned the network by reducing the occurrence of dynamic dead neurons (i.e., decrease activation sparsity). This different from existing works (Raihan & Aamodt, 2020; Liu et al., 2022b; activation sparsity for computation acceleration, the proposed AP investigates the interaction of weight and activation sparsity from a new perspective, i.c., how to tradcoff activation sparsity for more weight sparsity.
et al., 2022; Gupta et al., 2019) which jointly optimize weight and
4 Performance Evaluation
In Section 4.1, we first summarize the experiment setup. Next, in Section 4.2, we compare and analyze the results obtained. In Section 4.3, we conduct an ablation study to evaluate the effectiveness of several components in AP. Lastly, in Section 4.4, we conduct a substitution study on the proposed AP.
4.1 Experiment Setup
(1) Experiment Details. To demonstrate that AP can work well with different pruning methods, we shortlist two classical and competitive pruning methods. The details are summarized as follows.
1. Pruning ResNet-20 on the CIFAR-10 dataset using Global Magnitude with and without AP.
2. Pruning VGG-19 on the CIFAR-10 dataset using Global Taylor with and without AP.
3. Pruning DenseNet-40 (Huang et al., 2017) on CIFAR-100 using Layer-Adaptive Magnitude- Pruning (LAMP) (Lee et al., 2020) with and without AP.
4. Pruning MobileNet (Sandler et al., 2018) on the CIFAR-100 dataset using Lookahead Pruning (LAP) (Park et al., 2020) with and without AP.
5. Pruning ResNet-50 (He et al., 2016) on the ImageNet (i.e., ImageNet-1000) using Iterative Magnitude Pruning (IMP) (Frankle & Carbin, 2019) with and without AP.
6. Pruning Vision Transformer (ViT-B-16) on CIFAR-10 using IMP with and without AP.
We train the network using SGD with He initialization (He et al., 2015), momentum 0.9 and weight decay of 1e-4 (same as (Renda et al., 2019; Frankle & Carbin, 2019)). For the benchmark pruning method, we prune the network with pruning rate p = 20 (i.e., 20% of existing weights are pruned) in 1 pruning cycle. After using AP, the overall pruning rate remains unchanged as 20%, but 2% of existing weights are pruned based on AP, while the other 18% of existing weights are pruned based on the benchmark pruning met to be compared with (see Algorithm 2). We repeat 25 pruning cycles in run and use the early-stop top-1 test accuracy (i.c., the corresponding test accuracy when carly stopping critcria for validation error is met) to evaluate the performance. The experimental results averaged over 5 runs and the corresponding standard deviation are summarized in Tables 1 - 6, where À is the percentage of weights remaining. The bolded results indicate that AP is significantly better than benchmarks results after accounting for the standard deviation.
(2) Hyper-parameter Selection and Tuning. To ensure fair comparison against prior results, we utilize standard implementations (i.e., network hyper-parameters and learning rate schedules) reported the literature. Specifically, the implementations for Tables 1 are from (Frankle & Carbin, 2019), (Zhao et al., 2019), (Chin al., 2020), (Renda et al., 2019) and (Dosovitskly et al., 2020). The implementation details can be found in Section B.2 of the Appendix. In addition,
also tune syper-parameten for each experiment setup mentioned above using the validation datasct via grid scarch. Some hyper-parametors are luned as follows. (i) The training batch size is luned from (64, 128, ...., 1024]. (ii) The learning rate is tuned from 1e-3 to le-1 via a stepsize of 2e-3. (iii) The number training epochs tuned from 80 to 500 with a stepsize of 20. The validation performance using our tuned parameters are close to that of using standard implementations. Therefore, we use standard implementations reported in the literature reproduce benchmark results.
8

---

Published in Transactions on Machine Learning Research (09/2023)
Original Top-1 Test Accuracy: 91.7% (A = 100%)
32.8%
26.2%
13.4%
5.72%
Global Magnitude
90.3 0.4
89.8 + 0.6
88.2 + 0.7
81.2 = 1.1
Global Magnitude with AP-Lite
90.4 0.7
90.2 + 0.8
88.7 + 0.7
82.4 = 1.4
Global Magnitude with AP-Pro
90.7 0.6
90.4 + 0.4
89.3 0.8
84.1 1.1
Table 1: Performance (top-1 test accuracy + standard deviation) of pruning ResNet-20 CIFAR-10 using Global Magnitude with and without the proposed AP.
Original Top-1 Test Accuracy: 92.2% (A 100%)
32.8%
26.2%
13.4%
5.72%
Global Taylor
90.2 + 0.5
89.8 + 0.8
89.2 + 0.8
76.9 + 1.1
Global Taylor with AP-Lite
90.5 + 0.8
90.3 + 0.7
89.7 + 0.9
78.4 + 1.4
Global Taylor with AP-Pro
90.8 + 0.6
90.7 + 0.9
90.4 + 0.8
79.2 + 1.3
Table 2: Performance (top-1 test accuracy + standard deviation) of pruning VGG-19 on CIFAR-10 using Global Taylor with and without the proposed AP.
(3) Reproducing Benchmark Results. By using the implementations reported in the literature, we have correctly reproduced the benchmark results. For example, the benchmark results in Tables 1 6 are comparable to Fig.11 and Fig.9 of (Blalock et al., 2020), Table.4 in (Liu et al., 2019), Fig.3 in (Chin et al., 2020), Fig. 10 in (Frankle et al., 2020), Table 5 in (Dosovitskiy et al., 2020), respectively.
(4) Source Code & Devices: We use Tesla V100 devices to conduct our experiments. The datasets are preprocessed using the conventional method. The source code available at https://githut: com/ watpa//////ctttattt-l--eeeeeeee
4.2 Performance Comparison
(1) Performance using Classical Pruning Methods. In Tables 1 & 2, we show the performance of AP using classical pruning methods (e.g., Global Magnitude, Global Taylor) via ResNet-20 and VGG-19 on CIFAR-10. We observe that as the percent of weights remaining, À docreases, the improvement of AP becomes larger. For example, in Table 1, the performance of AP-Lite at 26.2% is 1.3% higher than the benchmark result. The improvement. increases to 2.6% at À 5.7%. Note that AP-Lite does not increase the algorithm complexity of existing methods. As expected, in Table 1, AP-Pro leads more significant improvement of 2.0% and 4.1% at 26.2% and X 5.7%, respectively. Similar performance trends be observed in Table 2 as well. The results for more values of X can be found in the Appendix.
(2) Performance using Competitive and Classical Pruning Methods. In Tables 3 and 4, we show that AP can work well with Competitive pruning methods (e.g., LAMP, LAP). In Table 3, we show the performance of AP using LAMP via DenseNeet-40 on CIFAR-100. We observe that AP-Lite improves the performance of LAMP by 1.2% at X 13.4% and the improvement increases to 1.6% at X 5.7%. Note that AP-Lite does not increase the algorithm complexity of existing methods. For AP-Pro, causes larger improvement of 4.6% and 3.8% at X 13.4% and X 5.7%, respectively. Similar performance trends be observed in Table 4, where we show the performance of AP using LAP via. MobileNet V2 on 100.
(3) Performance on ImageNet. In Table 5, we show the performance of AP using Iterative Magnitude Pruning (IMP, i.e. the lottery ticket hypothesis pruning method) via ResNet-50 on ImageNet (i.e., the ILSVRC version) which contains over 1.2 million images from 1000 different classes. We observe that AP- Lite improves the performance of IMP by 1.5% 5.7%. For AP-Pro, it improves the performance of TMP by 2.8% at = 5.7%.
9

---

Published in Transactions on Machine Learning Research (09/2023)
Original Top-1 Test Accuracy: 74.6% (A 100%)
32.8%
26.2%
13.4%
5.72%
LAMP LAMP with AP-Lite
71.5 = 0.7
69.6 + 0.8
65.8 + 0.9
61.2 + 1.4
71.9 0.8
70.3 + 0.7
66.6 + 0.7
62.2 + 1.2
LAMP with AP-Pro
72.2 = 0.7
71.1 0.7
68.8 0.9
63.5 + 1.5
Performance (top-1 test accuracy + standard deviation) of pruning DenseNet-40 on CIFAR-100 using Layer-Adaptive Magnitude Pruning (LAMP) with and without the proposed AP.
Table 3:
Original Top-1 Test Accuracy: 73.7% (A 100%)
32.8%
26.2%
13.4%
5.72%
LAP LAP with AP-Lite
72.1 + 0.8
70.5 + 0.9
67.3 0.8
64.8 + 1.5
72.5 = 0.9
70.9 + 0.8
68.2 1.2
66.2 + 1.5
LAP with AP-Pro
72.8 = 0.7
71.4 +
69.1 0.8
67.4 + 1.1
Performance (top-1 test accuracy + standard deviation) of pruning MobileNetV2 on CIFAR-100 using Lookahead Pruning (LAP) with and without the proposed AP.
Table 4:
Original Top-1 Test Accuracy: 77.0% (A = 100%)
A IMP
32.8%
26.2%
13.4%
5.72%
76.8 4: 0.2
76.4 + 0.3
75.2 + 0.4
71.5 d: 0.4
IMP with AP-Lite
77.2 + 0.3
76.9 + 0.1
76.1 + 0.3
72.6 + 0.5
IMP with AP-Pro
77.5 + 0.4
77.2 + 0.3
76.8 + 0.6
73.5 0.4
Table 5:
Performance (top-1 validation accuracy + standard deviation) of pruning ResNet-30 on ImageNet using Tterative Magnitude Pruning (TMP) with and without. AP.
Original Top-1 Test Accuracy: 98.0% (A 100%)
32.8%
26.2%
13.4%
5.72%
IMP
97.3 + 0.6
96.8 + 0.7
88.1 + 0.9
82.1 + 0.9
IMP with AP-Lite
98.0 + 0.4
97.3 +
89.9 + 0.6
83.6 +
IMP with AP-Pro
98.2 + 0.6
97.6 + 0.5
91.1 + 0.8
84.8 + 1.0
Table 6: Performance (top-1 (est accuracy + standard deviation) of pruning Vision Transformer (ViT-B-16) on CIFAR-10 using IMP with and without AP.
(3) Performance on Competitive Networks (Vision Transformer). Several recent works (Liu et al., 2021b; Yuan et al., 2021; Chen al., 2021) demonstrated that transformer based networks tend to provide excellent performance in computer vision tasks. We now examine the performance of AP using Vision Trausformer (i.e., VIT-B16 with a resolution of 384, pretrained on ImageNet dataset). We note that the ViT-B16 uses Gaussian Error Linear Units (GELU, GELU(x) x&(z), where ((x) is the standard Gaussian cunulative distribution function) as the activation function. Similar to ReLU which blocks the negative pre- activation output, GELU heavily regularizes the negative pre-activation output by multiplying an extremely small value of (xx,, suggesting that AP could be helpful with pruning GELU based models as well.
We repeat the same experiment sctup as above and evaluate the performance of AP using ViT-B16 in Table 6. We observe that AP-Lite helps improve the performance IMP by 1.8% at À 5.7%. For AP-Pro, it improves the performance of IMP by 3.3% at X = 5.7%.
10

---

Published in Transactions on Machine Learning Research (09/2023)
À
32.8%
26.2%
13.4%
5.72%
AP-Lite
90.4 + 0.7
90.2 + 0.8
88.7 + 0.7
82.4 + 1.1
AP-Lite-SOLO
86.0 + 1.0
84.3 + 1.5
81.5 + 2.0
74.5 +: 3.1
AP-Lito-NO-WR
87.5 0.9
87.1 1.2
84.7 1.5
78.8 + 2.3
Table 7: Ablation Study: Performance Comparison (top-1 test accuracy standard deviation) between Li-Lite and AP-SOLO, AP-Lite-NO--R on pruning ResNet-20 on CIFAR-10 via Global Magnitude.
À
32.8%
26.2%
13.4%
5.72%
AP-Pro
90.8 + 0.6
90.7 + 0.9
90.4 + 0.8
79.2 + 1.3
AP-Pro-SOLO
85.8 1.5
83.2 1.7
81.5 = 1.9
70.3 + 2.7
AP-Pro-NO-WR
88.1 1.2
86.3 + 1.5
85.6 + 1.5
74.8 + 2.1
Table &
Ablation Study: Performance Comparison (top-1 test accuracy standard deviation) between AP-Pro and AP-Pro-SOLO, AP-Pro-NO-WR on pruning VGG-19 using CIFAR-10 via Global Taylor.
A
32.8%
26.2%
13.4%
5.72%
AP-Lite
77.2 + 0.3
76.9 + 0.4
76.1
0.3
72.6 + 0.5
AP-Lite-SOLO
75.8 + 0.5
74.3
0.7
71.1
0.6
68.5 + 0.9
AP-Litc-NO-WR
76.3 + 0.6
74.9 + 0.8
73.2 + 0.8
70.3 + 1.1
Table 9:
Ablation Study: Performance Comparison (top-1 test accuracy standard deviation) between
AP-Lite and AP-Lito-SOLO, AP-Lito-NO-WR pruning ResNet-50 on ImageNet via IMP.
4.3
Ablation Study
We now conduct an ablation study to evaluate the effectiveness of components in AP. Specifically, we remove one component at a time AP and observe the impact on the pruning performance.
1. (P-iicc/Pro))NOO-V WR: Using AP without the weight rewinding step (i.c., remove step (fi) from Algo. 2). This aims to evaluate the effect of weight rewinding on the pruning performance.
AP-((ita/Pro)-SOOO: Using only AP-(Lite/Pry) without the benchmark pruning method (i.e., in every pruning cycle, pruning weights only based on AP). This aims evaluate if the pruning metric of AP alone be used to evaluate the importance of weights.
In Tables 7 and 8, we conduct experiments of pruning ResNet-20 on the CIFAR-10 dataset using Global Magnitude (AP-Tite) and pruning VGG-19 on CIFAR-10 using Global Taylor (AP-Pro) respectively. Tn each case, we compare the performance of AP-(Lite AP-([LttePro))NOO--M) Pro)-NO-WR, APP((Lite/ro))SSOO.O to AP-(Litc/Pro)) to demonstrate the effectiveness of components in AP. We note that, same as before, utilize the implementation reported in the literature. Specifically, the arper-parameters and the learning rate schedule are from (Frankle & Carbin, 2019).
Efffec of Weight Rewinding. In Tables and 8, we compare the performance of PP-(Lite/Pro--NO--A to AP- (AiteLie/) The key difference that AP-(Lite/Pret uses weight rewinding (see Algorithm 2) whereas the NO-WR approaches do not. We find that the performance AP-(Lite/Pro) is always higher across all X. For instance, at X 5.72%, AP-Pro-NO WR yields an accuracy of 74.8%, which 4.4% lower than AP-Pro itself. This suggests the crucial role of weight rewinding in improving the performance.
11

---

Published in Transactions on Machine Learning Research (09/2023)
32.8%
26.2%
13.4%
5.72%
Global Magnitude (GM)
90.3 (6.7%)
89.8 (6.3%)
88.2 (5.8%)
81.2 (5.4%)
GM + AP-Pro
90.7 (6.3%)
90.4 (5.9%)
89.3 (5.1%)
84.1 (3.9%)
GM + AP-output
90.6 (6.2%)
90.5 (5.8%)
89.4 (5.1%)
84.4 (3.8%)
GM + AP-bias-0.2
90.1 (6.6%)
89.4 (6.3%)
88.1 (5.8%)
82.3 (5.4%)
GM + AP-bias-0.5
89.9 (6.5%)
89.6 (6.1%)
86.9 (5.5%)
82.8 (4.8%)
GM + AP-bías-1.0
89.5 (6.1%)
88.7 (5.7%)
87.0 (5.0%)
81.5 (3.6%)
Table 10: Performance comparison (i.e., top-1 test accuracy (dynamic DNR)) between AP-Pro and alterna-
tive methods in terms of dead neuron location and activation (rows 4 - 7) on ResNet-20 using CIFAR-10.
32.8%
26.2%
13.4%
5.72%
LAP
72.1 (7.8%)
70.5 (6.9%
67.3 (6.5%)
64.8 (6.1%)
LAP + AP-Pro
72.8 (7.4%)
71.4 (6.4%)
69.1 (5.6%)
67.4 (5.1%)
LAP + AP-output
72.5 (7.6%)
71.7 (6.2%)
69.3 (5.8%)
67.2 (5.3%)
LAP + AP-bias-0.2
71.8 (7.8%)
70.2 (6.8%)
66.8 (6.4%)
63.3 (6.1%)
LAP + AP-bias-0.5
71.4 (7.7%)
70.4 (6.6%)
67.1 (6.2%)
65.1 (5.8%)
LAP + AP-bias-1.0
71.2 (7.5%)
70.1 (6.5%)
67.7 (6.1%)
65.9 (5.4%)
Table 11: Performance comparison (i.e., top-1 test accuracy (dynamic DNR)) between AP-Pro and alterna- tive methods in terms of dead neuron location and activation (rows 4 4- 7) on MobileNet using CTFAR-100.
When AP Works Solely. The pruning metric of AP (see Algorithm 1) aims to reduce dynamic DNR by pruning. We compare the performance AP-(Litt/Pro))-SO00 to (P-(Lite/PP)))) (Pro) to evaluate if the pruming metric of AP can be used solely, without working with other pruning methods. In Tables 7 and 8, we observe that the SOLO methods perform much worse. For example, SOLO is 74.5, which is 7.9% lower than AP-Lite. It suggests that the pruning metric of AP alone is not suitable to evaluate the importance of weights. The effect of AP's metric on reducing dynamic DNR and its pruming rate q on pruning performance are discussed in Scction 5.
= 5.72%, the performance of AP-Lite-
More Results using ResNet-50 on ImageNet. To further validate the results, we also conduct the ablation study using AP-Lite on ImageNet (ResNet-50). We summarize the results in Table 9, and we find that the results largely mirror those in Tables 7 and 8. Thus, both SOLO and NO-WR approaches perform significantly worse than the baseline.
4.4 Substitution Study
In this subsection, we conduct a substitation study on the proposed AP. Specifically, AP consists of two key components: dead ReLU neuron location and activation (see Algorithm 1). We replace one component at a time and observe the effect on dynamic DNR reduction and pruming performance (i.c., accuracy). we construct two variants of AP as follows and compare to the original AP-Pro.
We replace the existing dead neuron location mechanism (i.c., weight movement) with directly observing the post-activated output of each ReLU neuron.
AP-output:
AP-bias-k: We replace the existing dead neuron activation mechanism (i.e., prune negative weights) by adding constant value k to the bias of dead ReLU neurons.
The results of ResNet-20 & MobileNet are summarized in Tables 10 11. The value in parentheses is dynamic DNR. Note that static DNR values remain roughly the same for all methods cach column and are therefore not shown. This is mainly because that majority of pruned weights are determined by the pruning method that AP works with (more details in Section 5). The findings from Tables 10 11 follows:
12

---

Published in Transactions on Machine Learning Research (09/2023)
AP Pruning Rate,
1%
2%
3%
5%
AP-Lite (A 64.0%)
89.8 + 0.1
90.0 + 0.2
89.5 + 0.4
89.2 = 0.6
AP-Lite (A = 40.9%)
88.2 + 0.4
88.9
0.6
88.5 + 0.7
87.3 + 0.5
AP-Lite (A = 26.2%)
87.1 + 0.5
87.9 + 0.8
86.7 + 0.8
86.3 + 0.9
Table 12: Performance (top-1 test accuracy + standard deviation) of AP-Lite when iterative pruning RosNet- 20 on CIFAR-10 with different pruning rate.
AP Pruning Rate, !
1%
2%
3%
5%
AP-Lite (A = 64.0%)
91.1 + 0.2
91.7 0.3
90.2 + 0.5
89.8 = 0.6
AP-Lite (A = 40.9%)
88.7 + 0.5
89.8 + 0.9
89.3 + 1.1
88.0 = 0.9
AP-Lite (A = 26.2%)
87.9 + 0.8
88.5 + 0.7
88.1 + 1.3
87.1 = 1.3
Table 13: Performance (Lop-1 (est accuracy + standard deviation) of AP-Lite when iteralive pruning VGG- 19 on CIFAR-10 10 with different pruning rate.
1. Effect of AP on reducing dynamic DNR. When conventional pruning methods work with AP, the dynamic DNR is significantly reduced (compare GM to GM + AP-Pro in Table 10).
2. Ceiling analysis on dead neuron location. The performance of AP-Pro and AP-output compa- rable in terms of reducing dynamic DNR and pruning performance. This suggests that AP-Pro works as expected and is able to locate dynamic dead neurons as dircctly observes the post-activated output.
3. Implementation complexity. We note that implementing AP-output is more complex than AP-Pro. AP-output requires practitioners to record the state of each neuron and then average over every training batch. For AP-Pro, we only need to compute the difference between two weight matrices. Given that AP-Pro can provide comparable performance to AP-output, and is relatively simpler to implement. As such, we still recommend using weight movements to locate dead ReLU neurons, as AP-Pro does now.
4. Comparison to AP-bias-k. When comparing AP-bias to AP-Pro, we find that a small value of k fails to reduce dynamic DNR as AP-Pro does. While for a large value of k, it can directly reduce dynamic DNR, but the performance performance is still not comparable to AP-Pro. We suspect. that adding a large bias may hinder the optimization of the network during retraining, leading to uncompetitive results.
5 Reflections
In this section, we discuss several important points and present some experimental results.
(1) Pruning Rate of AP, q. Active Pruning removes 9% of remaining parameters in every pruning cycle, as to reduce dynamic DNR. The value of is usually much smaller than the pruning rate of the pruming method it works with. As an example, in Section 4, the overall pruming rate is fixed as 20% and 2% of weights are pruned based on Active Pruning, which is much smaller than the pruming rate of the benchmark method compared with (i..., 18%). Adjusting the value of g is a trade-off between pruning less important weights and reducing dynamic DNR. A large q value indicates preferential reduction of dynamic DNR, while a small q value mcans preferential removal of less important weights.
We repeat the experiments of pruning ResNet-20 CIFAR-10 using Global Magnitude and AP-Lite. We note that the overall pruning rate is fixed as 20% and the pruning rate of AP increases from 1% to 5%. Correspondingly. the pruning rate of Global Magnitude decreases from 19% to 15%. The experimental results are summarized in Table 12. We observe that as we increase the pruning rate of AP from 2%, the performance tends to decrease. Similar performance trends can be observed using VGG-19 on CIFAR- 10 as well (see Table 13). The thcoretical determination of the optimal value of q clearly worth deeper thought. Alternatively, q can be thought of as parameere and tuned via the validation dataset and let q = 2 could be good choice it provides promising results in various experiments.
13

---

Published in Transactions on Machine Learning Research (09/2023)
q-best
3.0%
q-best
regeiin
q-feasible-region
Value of AP's pruning rate,
2.5%
2.0%
2.0%
AP's
1.5%
1.5%
1.0%
1.0%
0.5%
10%
12%
15%
18%
20%
22%
25%
28% 30%
10%
12%
15%
18%
20%
22%
25%
28%
30%
overall pruning rate,
Value
pruning rate,
Figure 3: The value of g-best and gefeasible-region as gradually increases when iteratively pruning ResNet- 20 (Left) and VGG-19 (Right) on CIFAR-10 using Global Magnitude and Global Taylor, respectively.
(2) Relationship between p and q. In the above experiments, examine feasible values of AP's pruming rate q when the overall pruming rate p is fixed at 20%. We now conduct experiments to find feasible values of when is changing, so to suggest the relationship between them. We conduct experiments with p values from 110%, 15%, ... 30%] on ResNet-20 and VGG-19. We gradually increase the value of with a step size of 0.25% and define two terms about as follows.
q-best: The value of q that provides the best pruning performance in terms of accuracy.
2. q- feasible-region: The value region of that provides comparable performance to q-best (i.e., 0.5% accuracy difference).
The results are depicted in Fig 3, where we make two observations: (i) As increases, & should increase as well. (ii) A heuristic of q = 0.1p seems to a promising method to determine the value of g. Alternatively, q can also be considered as hyperparameter and tuned via validation dataset.
(3) Comparison to Activation Sparsity Baselines. We note that AP works in tandem with existing pruning methods and decrcases the activation sparsity of pruned networks by pruning its negative weights. There could be other approaches which can decrease the activation sparsity well, such as L1 regularization (Georgiadis, 2019) and boosted Hoyer Regularization (Kurtz & et al, 2020). Specifically, L1 regularization and boosted Hoyer rogularization can be applicd in the opposite direction to decrease the activation sparsity.
We now replace AP with activation sparsity baselines (e-g., boosted Hoyer regularization) and examine the performance when they work in tandem with existing pruning methods. The results are summarized in Tables 14 15. The takeaway message is two-fold: (i) When conventional pruning methods work with AP, this leads a better performance than working with other activation sparsity baselines. The reason could be that AP explicitly targets dynamic dead neurons by pruming negative weights. This decreases the activation sparsity in more precise manner. While other activation sparsity baselines use an augmented loss function and the decrease of activation sparsity becomes implicit and hard to control during optimization (i.e., the selection of which neuron is activated not clear and there no precise control). (ii) Interestingly, when we gradually prune the network, using conventional pruning mothods with activation sparsity basclines outperforms the original counterpart (i.e., using conventional pruning method solely). For example, compare Global Magnitude + Booster Hoyer to Global Magnitude when À = 5.72%. This suggests that activation sparsity approaches can help to improve the performance of pruning methods. New methods to reduce activation sparsity to obtain more weighted sparsity are definitely worth exploring.
(4) Static DNR. During iterative pruning, the static DNR tends to increase as expected (see Fig. 1). It is interesting to mention that, after incorporating with AP, the static DNR of pruned networks remains almost the same opposed without. For example, the static DNR Global Magnitude in Table 10 (second row) increases from 7.1% to 8.4% and finally reaches 15.9% when À decreases from 32.8% to 26.2% and finally to
14

---

Published in Transactions on Machine Learning Research (09/2023)
A
32.8%
26.2%
13.4%
5.72%
Global Magnitude (GM)
90.3 = 0.4
89.8 + 0.6
88.2 0.7
81.2 +
GM + AP-Pro
90.7 =
90.4 + 0.4
89.3
84.1 + 1.1
GM + Booster Hoyer
89.5 + 0.6
88.7 + 0.8
86.1 1.0
82.5 0.7
GM + L1 Rogularization
89.8 = 0.5
88.3 + 0.7
85.6 = 0.9
81.9 + 0.8
Table 14: Performance (top-1 test accuracy + standard deviation) of pruning ResNet-22 on CIFAR-10 using Global Magnitude with the proposed AP and other activation sparsity baselines.
32.8%
26.2%
13.4%
5.72%
Global Taylor (GT)
90.2 = 0.5
89.8 + 0.8
89.2 + 0.8
76.9 = 1.1
GT + AP-Pro
90.8 + 0.6
90.7 + 0.9
90.4 + 0.8
79.2 + 1.3
GT + Booster Hoyer
89.8 0.7
88.6 + 0.6
87.1 1.2
77.5 0.9
GT + L1 Regularization
89.2 + 0.4
88.1 + 0.7
86.3 + 1.0
77.1 + 1.2
Table 15: Performance (top-1 test accuracy + standard deviation) of pruning VGG-19 on CIFAR-10 using
Global Taylor with the proposed AP and other activation sparsity baselines.
13.4%. After incorporating with AP-Pro (third row in Table 10), the static DNR almost remains the same. This is mainly because that the majority of pruned weights are still determined by the Global Magnitude (i.e., 18%) while only 2% of pruned weights are determined by AP.
The Theorem 1 shows that, in addition to dynamic DNR, reducing static DNR also can improve the upper bound of TT. In fact, reducing static DNR has been incorporated directly or indirectly into the existing pruning methods. As an example, LAMP (i.e., one Competitive pruning method used in performance evaluation, Table 3) takes the number of unpruned weights of neurons/layers into account and avoids pruning weights from neurons/filters with less number of unpruned weights. This prevents neurons from being statically dead. Differing from existing methods, AP is the first method targeting the dynamic DNR. Hence, as a method that works in tandem with existing pruning methods, AP improves existing methods by filling in the gap reducing dynamic DNR, leading to much better pruning performance.
(5) Working with Non-ReLU based Networks. We would like to highlight that AP also works well with non-ReLU based networks. For example, in Table 5, show the performance of AP using Vision Transformer which uses GELU as the activation function. In this setup, AP also leads to improvement of 2% 3%. We posit that this could be due to the fact. that, similar to ReLU which blocks the negative pre- activation output, GELU heavily regularizes the negative pre-activation output by multiplying an extremely small value of (x)), suggesting that AP could be helpful with pruning GELU based models as well.
(6) Future Research. (i) We only examine the effect of AP on network pruning using image datasets. In fact, AP may not be limited to this, but can also be applied to dynamic sparse training algorithms or for NLP tasks. (ii) The activation sparsity methods which enforce activation sparsity could also be used in the opposite way to decrease the activation sparsity. Such methods could be an alternative for AP and new methods to reduce activation sparsity to obtain more weight sparsity are definitely worth exploring. (iti) The pruning rate of AP q is a important hyperparameter to tune and may significantly affect the performance. In above, we suggest the heuristic of q = 0.1p to determine 9 from the overall pruning rate p. A theoretical way to determine the value of g also worth exploring and may change in non-linear way with p.
6 Conclusion
In this paper, we propose a new pruning method called Activate-while-Prunnne (AP). Unlike existing pruming methods which remove less important parameters, the proposed AP works in tandem with existing pruming methods and aims improve their pruning performance by de-sparsifying pruned networks. It is also
15

---

Published in Transactions on Machine Learning Research (09/2023)
interesting to mention that the proposed AP studies the interaction of weight and activation sparsity from new perspective, i.e., how to tradeoff activation sparsity for more weight sparsity.
Theoretically, we show the benefits of de-sparsifying pruned networks from the perspective of information bottleneck. Empirically, we use six different sets of experiments to demonstrate that AP can work well with a diverse range of networks (e.g., ResNet, VGG, DenseNet, MobileNet) and pruning methods (e.g. IMP, LAP, LAMP, etc) on both CIFAR-10/100 and ImageNet. It should be noted that AP is a generic approach, and by using the proposed AP, the pruning performance of existing pruning methods can be improved by 3% conduct an ablation study to further investigate and demonstrate the effectiveness of
8%. Furthermore, several key components that make the proposed AP. Lastly, we conduct substitution study to replace certain components in AP with alternative methods, further verifying the design of the proposed AP.
Acknowledgements
This research is supported by S*STRR, CISCO Systems (USA) Pte. Ltd and the National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award T21001R0002). Additionally, we would like to thank the members of the Kent-Ridge Al research group at the National University of Singapore for helpful feedback and interesting discussions on this work.
References
Ruth Akiva-Hochman, Shahaf E Finder, Javier S. Turek, and Eran Treister. Searching for n: M fine-grained sparsity of weights and activations in neural networks. In European Conference on Computer Vision (ECCV), pp. 130 143. Springer, 2022.
Isac Arnekvist, J. Frederico Carvalho, Danica Kragic, and Johannes A. Stork. The effect of target normal- ization and momentum on dying relu. CoRR, abs/2005.06195, 2020.
Davis Blalock et al. What is the state of neural network pruning? In Proceedings of the Machine Learning and Systems (MLSys), 2020.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVE International Conference Com- puter Vision (ICCV), pp. 357-366 2021.
Tianlong Chen, Zhenyu Zhang, Jun Wu, Randy Huang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Can you win everything with lottery ticket? Transactions of Machine Learning Research (TMLR), 2022. sest/forrumdd===LNNNe
URL https: htttt::///pnnreveer
Ting-Wis Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVP Conference Compuler Vision and Pallern Recognition (CVPR), pp.
2020.
Thomas M Cover and Joy Thomas. Elements of Information Theory, 2nd edition. John Wiley & Sons, 2006.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang, Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 7373-7382, 2021.
Jia Deng et al. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE/CVE Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 248 255, 2009.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurclio Ranzato, and Nando de Freitas. Predicting pa- rameters in deep learning. Xiv preprint: 1306.05. 0543, 2014.
Shibhansh Dohare, A. Rupam Mahmood, and Richard S. Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. CoRR, hts/2108.06322 2021.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint:2010.1111 2020.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. arXiv 1911.11194 cs.LG, 2021.
Utku Evci, Yani A. Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. arXie
Is..G, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.011 2019.
Jonathan Frankle et al. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 2020.
International Conference on Machine Learning (ICML), pp.
Trevor Gale, Erich Flsen, and Sara Hooker. The state of sparsity deep neural networks. ar preprinl arXiv:1902.09574, 2019.
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceed- ings of the IEEE/CV Conference on Computer Vision and Pallern Recognition (CVPR), pp. 7085 7095, 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Swati Goswami,, C. A. Murthy, and Asit K. Das. Sparsity measure of a network graph: Gini index, 2016.
Udit Gupla, Brandon Reagen, Lillian Pentecost, Marco Donato, Thierry Tambe, Alexander M Rush, Gu- Yeon Wei, and David Brooks. Masr: A modular accelerator for sparse rmns. In 2019 28th International Conference Parallel Architectures and Compilation Techniques (PACT), pp. 1-14. IEEE, 2019.
Song Han et al. Learning both weights and connections for efficient neural network. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1135 1143, 2015.
Kaiming He et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classifica- tion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He et al. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 2009 2018, 2020.
Patrik O. Hoyer. Neg-tige matrix factorization with sparseness constraints. Journal of Machine Learning Research (JMLR), 5(9), 2004.
Gao Huang et al. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 1700-4788, 2017.
Niall P. Hurley and Scott. T. Rickard. Comparing measures of sparsity. ar Xío preprint: 0811.4 2009.
17

---

Published in Transactions on Machine Learning Research (09/2023)
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world ob- ject detection. In Proceedings of the IERE/CVE Conference Compuler Vision and Poltern Recognation (CVPR), pp. 5830 5840, 2021.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Mark Kurtz and et al. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (ICML), pp. 5533 5543. PMLR, 2020.
Yann LeCun et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 (11):2278-222) 1998.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-sdaptive sparsity for the magnitudc-based pruning. In International Conference Learning Representations (ICLR), 2020.
Joo Hyung Lee and et al. Jaxpruner: A concise library for sparsity research. arXiv preprint: 2304.14088 2023.
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference on Computer Vision (ECCV), pp. Springer, 2020a.
Hao Li et al. Pruning filters for elficient convnets. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc Van Gool. Revisiting ran- dom channel pruning for neural network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 191 201, 2022.
Yun Li, Weiqun Wu, Zechun Liu, Chi Zhang, Xiangyu Zhang, Haotian Yao, and Baoqun Yin. Woight- dependent gates for differentiable neural network pruning. In European Conference on Computer Vision (ECCV), pp. 23-37. Springer, 2020b.
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint: 2210.06313, 2023.
Linfeng Liu, Xu Han, Dawei Zhou, and Liping Liu. Towards accurate subgraph similarity computation via neural graph pruming. Thansactions on Machine Learning Research (TMLR), 2022a. URL https: //openreview
Shiyu Liu, Chong Min John Tan, and Mehul Motani. S-cye: A learning rate schedule for iterative pruning of relu-based networks. CoRR, abs/21100 211.0110,,,764,, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVP International Conference on Compuler Vision (ICCV) pp. 10012 10022, 2021b.
Zhi-Gang Liu, Paul N Whatmough, Yuhao Zhu, and Matthew Mattina. S2ta: Exploiting structured sparsity for energy-effciien mobile enn acceleration. In 2022 IEEE International Symposium on Bigh--Prrfomanee Computer Architecture (HPCA), pp. 573-586. TEEE, 2022b.
Zhuang Liu et al. Rethinking the value of network pruming. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
La Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:9933.06733 2019.
Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-comnecionss and limitod-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1458 1467, 2020.
18

---

Published in Transactions on Machine Learning Research (09/2023)
Eran Malach et al. Proving the lottery ticket hypothesis: Pruning is all you need. In Proceedings of the International Conference on Machine Learning (ICML), pp. 6682 (882-6691, 2020.
Rahul Mehta. Sparse transfer learning via winning lottery tickets. In Proceedings of the Advances in Neural Information Processing Systems Workshop on Learning Transferable Skills, 2019.
Pavlo Molchanov et al. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11264-11272, 2019.
Sejun Park, Jacho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude- based pruning. In International Conference Learning Representations (ICLR), 2020.
Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded a winning licket's mask? arXiv 2210.03044 Cs.LG, 2022.
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. Ac/dc: Alternating com- prcopp/dccomppresse training of deep neural networks. arXiv preprint: 2106. 12979, 2021.
Md Aamir Raihan and Tor Aamodt. Sparse weight activation training. Advances in Newral Information Processing Systems (NewrIPS), 33:15625 15638, 2020.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations (ICLR), 2019.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4510 4520, 2018.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411 (29) 2711, 2010.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint (rXtc::409.1556, 2014.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. arXin, 2302. 12902, Ls. 2023.
Lucas Theis et al. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.057877 2018
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. anXiv 1503. 02400 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advonces in Neurol Information Processing Systems (NeurIPS), pp- 5998-6008, 2017.
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. Journal of Machine Learning Research, 24(99):1-22 23, 2023.
Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. Recent advances neural network pruning initialization. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAT), pp.
23-29, 2022.
Yifei Wang, Yixuan Hua, Emmanuel Candés, and Mert Pilanci. Overparameterized relu neural networks learn the simplest models: Neural isometry and exact recovery. ar Xiv preprint: 192951 2023.
19

---

Published in Transactions on Machine Learning Research (09/2023)
Yulong Wang et al. Pruning from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12273-12280, 2020.
Mao Ye al. Good subnetworks provably exist: Pruning via greedy forward selection. In Proceedings of the International Conference on Machine Learning (TCML.) pp. 10820-10830. PMT.R, 2020.
Haoran You, Randall Balestriero, Zhihan Lu, Yutong Kou, Huihong Shi, Shunyao Zhang, Shang Wu, Yingyan Lin, and Richard Baraniuk. Max- adfine spline insights into deep network pruning. Transactions on Machine Learning Reseorch (TMLR), 2022. ISSN 2835-8856.
Hao nan Yu et al. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprinl or Niv:2106. 13112, 2021.
Yuyao Zhang and Nikolaos M Freris. Adaptive filter pruning via sensitivity leedback. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2023.
Chenglong Zhao et al. Variational convolutional neural network pruning- In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2780 2789, 2019.
Hattie Zhou et al. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Proceedings of the Advances in Newral Information Processing Systems (NeurIPS), 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
20


Reviews:
[{"review_id": "SZmxqiYKzj", "reviewer": "Reviewer_jwS7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. As far as I am aware, the proposed method is novel in its approach and provides interesting insights into the relationship between parameter sparsity and ephemeral sparsity induced by commonly used nonlinear activation functions. \n1. Extensive experiments are conducted across a wide variety of datasets and network architectures. \n1. This paper studies a very timely and important topic as sparse neural networks are an important ingredient in improving the overall efficiency of serving ML models at scale. \n1. The proposed method complements existing pruning algorithms and appears to demonstrate moderate improvements across each dataset/architecture pair studied. Of particular note are improvements in generalization performance at high sparsities. \n1. The Information Bottleneck analysis provides a compelling theoretical justification for the proposed method and generalization improvements observed. \n\n**Weaknesses**\n1. The heuristic used to locate dead ReLU neurons could also be used to locate neurons that were initialized at values close to their final converged value. I am curious to know if the authors conducted any empirical studies to determine how accurate this heuristic is in practice at locating truly dead neurons. \n1. A more formal analysis of the additional FLOPs and/or wall-clock timings required to perform the AP-Pro algorithm would help the reader understand what kind of overhead should be expected when adopting this method. \n1. The paper could use another proofread as there were some minor typos and formatting issues that must be addressed. See the Requested Changes section below for more details. \n1. Presentation of results could be improved by combining tables and using more plots, where appropriate. Further, maintaining the same sparsities across Tables 1-8 would make the ablation studies more compelling. \n1. Additional investigations into applying AP in the context of dynamic sparse training and NLP tasks would be of additional interest to the broader sparse neural network research community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method, AP,  for improving the performance of pruned neural networks which works in parallel with existing pruning methods. AP allocates a portion of the desired overall pruning ratio to negative weights which remain close to their initialized values at convergence. These weights are pruned in addition to those determined by the base method saliency criterion as required to meet the desired sparsity. AP reduces the DNR by increasing the number of active ReLU neurons, on average, across a given training dataset distribution. The author’s provide theoretical justification for their claims by considering how DNR relates to the Information Bottleneck framework. Empirical evidence of AP’s performance is presented across a number of datasets and network architectures for image classification. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Typos** (Must change for acceptance)\n1. Formatting of in-text citations seems to be missing parentheses throughout the paper. \n1. (“ii) AP-Pro which introduces an **addition** retraining step to the existing methods in every pruning cycle, but significantly improves the performance of existing methods.” -> Addition should be “additional” here. \n1. “According to **Han et al. (2015), Han et al.** trained the unpruned network” -> Double in-text citation for Han et al. Only cite once. \n1. A citation for VGG-19 appears to be missing. This should be added at the first mention of the network, similar to other network architectures.  \n1. “Haoran You and et al.” is listed as the author of “Max-affine spline insights into deep network pruning”. This should simply say “Haoran You et al.”. The corresponding in-text citation also shows the extra “and”. \n1. Section 5.3 -> “**Differ** from existing methods, AP is the first…”. Should read: “Differing from …”\n\n**Clarifications** (Further information will help secure acceptance, but is not specifically required)\n1. “The success of ReLU is mainly **due to fact _(sic)_** that existing networks tend to be overparameterized and ReLU can easily regularize overparameterized networks” -> This claim does not appear to be supported by Glorot et al. (2011). The cited paper refers to several advantages related to vanishing / exploding gradient, sparse representations, and computational simplicity, but regularization is not listed as a motivation. Consider rephrasing this sentence or clarifying with additional works cited. \n1. “For example, Liu et al. (2019a) demonstrated that training from scratch on the right sparse architecture yields better results than pruning from pre-trained models.” -> Please check this citation. The paper cited is for DARTS which is an AutoML technique that does not discuss sparsity as far as I am aware. It appears that this citation should be for Liu et al. (2019b). In any case, this is a strong claim to draw from Liu et al. (2019b) and is at odds with the majority of sparsity literature to my knowledge. Especially since Liu et al. explicitly noted that their claims only hold true for structured sparsity and they were unable to maintain accuracy with unstructured sparse networks on large scale datasets (imagenet). Given that the context of the paper under review is unstructured pruning, I encourage the authors to remove this claim unless better supporting justification can be provided. \n1. In Fig 1 (Left), the figure is missing the 64% weight remaining column. What was the reason for this exclusion? All other weights remaining appear to follow the 20% pruning ratio. Consider including 64% or clarifying why this result is excluded. \n1. While the pruning results augmented with AP do demonstrate improved generalization performance, in some cases this improvement is not clearly significant across the full range of sparsities considered. For the sake of clarity, I would encourage the authors to only bold tabulated results that are significant after accounting for the standard deviation. For instance, in Table 1 where lambda=32.8%, AP-Pro is bolded but the vanilla global magnitude (GM) results are not significantly worse considering the standard deviations of each result. I.e., 90.3+0.4 > 90.7-0.6.\n1. Were any data augmentation schemes used in training? If so, those should be included in the experimental details section or in the appendix. \n1. What initialization method was used to train the networks? This information should be included for reproducibility. \n1. Table 7: Why do lambda=32.8% results for AP-Lite not match Table 1 for the same lambda? It appears that these should be identical runs? \n1. For tables 7 and 8, why are the weights remaining depicted >> those shown in tables 1-4? As a reader, I am most interested in seeing how the ablations differ from the benchmark and results presented previously. Especially since AP is most beneficial in the highly sparse regime. \n\n**Recommendations** (Changes not required, mere suggestions to consider)\n1. The numerator and denominator in equation 1 could be expanded to clarify the contributions of static and dynamic DNR.  I.e., # of dead ReLU neurons + # of neurons with pruned incoming weights for the numerator. For denominator, consider clarifying that these are all \"relu\" neurons in the unpruned network (i.e., no conv, batch norm, etc. neurons included in this sum)\n1. Figure 2 -> I suggest revising the AP % labels to depict (p-q)+q. As currently depicted, it appears to be p-q*q. The color labels help here but it could be more clear. Another recommendation here is to use the length of the color bars to show the overall model sparsity decreasing with each pruning cycle. I.e., Pruning Method X could show blue shading for 20% of the total bar at cycle 1, 20% + 16% for cycle 2, etc. This may help clarify to the reader how each pruning cycle removes a smaller and smaller portion of the total dense model weights. \n1. In Section 4.1.2, the authors discuss a grid search conducted on a variety of hyperparameters. However, the results of this search do not appear to have been included as the “validation performance of using our tuned parameters are close to that of using standard implementations”. While I can appreciate the effort, I do not recommend including this description if the results are not included in the manuscript or appendices. Further, given that the purpose of this work is to establish the benefit of AP over existing methodologies, I believe the standard implementations already provide the most useful information to the reader when determining the overall impact of AP.\n1. In Section 5.1, the authors discuss the effect of the pruning rate of AP, q. This is a very important inclusion as the introduction of any new hyperparameter deserves scrutiny. It would be interesting to see how this optimal q value changes with various p values. Perhaps q = 0.1 *p is a good heuristic across a multitude of p values?\n1. “In this section, we first conduct experiments to evaluate the DNR during iterative pruning in Section 3.1.” -> I suggest referring to the current section only once. I.e., “In section 3.1, we first …”\n1. The results section is needlessly verbose in some paragraphs with several sentences simply repeating the results listed in tabular form elsewhere. Where these sentences are used to emphasize a finding they work well; however, each paragraph in section 4.2 contains a similarly formatted sentence which diminishes the emphasizing effect. I encourage the authors to simply remove sentences that repeat the tabulated results without expanding or providing further relevant information. \n1. Further investigation of AP by applying it to dynamic sparse training algorithms (SET, RigL, etc.) would be of additional interest to the broader sparse neural network research community. \n1. Only image classification tasks were considered. It would be interesting to see how AP performs on NLP tasks as well. \n\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C4tcDHASIQ", "reviewer": "Reviewer_YnV4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n- Paper is mostly well written and easy to follow\n- Authors use different network architectures and datasets in image classification, which includes compact architectures like MobileNet and more recent Vision transformers with GELU activations.\n- Authors investigate an interesting and under-studied area of neural network pruning\n\n# Weaknesses\n(1) I found few things about the experimental evaluation concerning. Happy to update my review if these concerns are addressed. \n\n(a) It is not clear whether proposed algorithm AP uses more training steps than the baselines. Training the networks longer would give better pruning results, therefore baseline pruning methods should be trained longer using either Cosine cyclic learning rate (i.e. learning rate restarts) or regular linear scaling as used in [1]. \n\n(b) It is not clear whether proposed technic works as concluded (i.e. by reducing DNR) due to lack of relevant ablations (see requested changes below). It's not obvious why authors doesn't use \n\n[1] https://arxiv.org/abs/1902.09574", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies dead neurons in pruned networks and classifies them as (1) neurons that have connections but give zero activations  neurons and (2) neurons without incoming connections. They observe that during pruning \"static dead neuron\" rate increases, however \"dynamic dead neuron rate\" (DNR) decreases. Authors argue that it would be better for learnability to have smaller DNR, thus they propose removing some of the negative valued incoming connections to push activations to the positive side of the ReLU. Authors combine this strategy with existing competitive pruning methods and prune a fraction of the weights (2%) using proposed approach while pruning the rest using the original pruning metric. Authors show improved generalization across different datasets and networks. Despite its strengths (see below), I found few things about the experimental evaluation concerning. Happy to update my review if these concerns are addressed. \n\nAlso, I had limited time to review this paper, please let me know if you have any questions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Major\n- *AP-Lite and AP-Pro* The difference between the two approach seems whether Iterative magnitude pruning is used. As argued by the authors AP can be applied to any pruning method and defining two version for 2 different pruning method seems unnecessary. As stated in (1a) and in the paper #training_steps is a hyper parameter increasing which often leads to better results [2].\n- It is not clear why authors look at the movement of the neurons instead of their activations directly? I would guess with large enough batch size looking at the activations one can directly predict dead neurons. Without this it is not clear whether proposed pruning metric targets dynamic dead neurons directly. Similarly, effect of removing negative weights can be achieved by adding a constant value to the bias of the dead neurons. Would that work as well? \n- What happens to Static Dead neurons in Table 11/12? It would be nice to add those numbers. To me it seems like proposed method increased static dead neurons by pruning weights from about to die neurons. If so it would be nice to say that.\n- Is the hyper parameter search as explained in Section 4.1 done for both baseline pruning methods and AP separately? It is not clear hyper-parameters for which experiments are optimized.\n\n# Minor\n- I think we shouldn't be training VGG-19 on Cifar-10 and more importantly use it in pruning baselines. It is extremely over-parameterized and has much worse scaling than more recent architectures. \n- I wouldn't call algorithms given SOTA. \"Competitive\" would be a better word. There are more recent works which report improved performance [3]. Global magnitude pruning often doesn't perform as well [4] gradual magnitude pruning (Zhu and Gupta).\n- Better to call\"Global Gradient\"  as \"Global Taylor\" [5].\n- There are quite a few references for lottery ticket hypothesis and IMP, which is not a competitive pruning method (see Renda et. al.) If authors like to keep these references, I would recommend adding more recent work discussing the limitations of lottery tickets [6].\n- I don't think ViT paper as Cifar-10 pretraining. I assume authors prune the imagenet pretrained networks. Is that the case? If so, would be nice to mention this. Also in general when standard deviations overlap, one would conclude none of the methods are better, thus bolding both. It would be nice to do that in all results.\n- \"the the\" -> \"the\"\n- I would recommend authors to cite relevant work which looks into dead neurons in neural networks [7]\n\n[2] https://arxiv.org/abs/1911.11134\n[3] https://arxiv.org/abs/2106.12379\n[4] https://arxiv.org/abs/2304.14082\n[5] https://arxiv.org/abs/1906.10771\n[6] https://arxiv.org/abs/2010.03533, https://arxiv.org/abs/2210.03044\n[7] https://arxiv.org/abs/2302.12902, https://arxiv.org/abs/2108.06325", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uykl3qMNQK", "reviewer": "Reviewer_4xQt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strength.__ The proposed algorithm is quite well-motivated. The observations in Figure 1(right) is quite interesting (and makes a lot of sense), and could potentially inspire many future research efforts. I have not seen many results that explicitly study the intersection of the weight and activation sparsity (with a delightful exception of https://arxiv.org/abs/2112.13896 )---there could be some fundamental tradeoff between two notions of sparsity.\n\nAnother strength is the originality of the idea that authors propose to enhance the activation sparsity of the pruned models. As far as I know, typical approach to control the activation sparsity is by an explicit regularization of the intermediate activations. In contrast, this paper pioneers an alternative way, which is to perturb the weights along a specific direction (which makes most sense in the pruning context) so that the pre-activations shift toward more frequent activation. Perhaps a similar idea could also be used whenever a less frequent activation sparsity is needed.\n\n__Weakness.__ The most significant weakness of this manuscript is that it almost completely disregards any related work on the activation sparsity. This paper revolves around a core concept, which the authors call \"dead neuron rate,\" and this is actually a well-studied notion under the name of \"activation sparsity\" or sometimes ephemeral sparsity (see https://proceedings.mlr.press/v119/kurtz20a.html for instance). Hopefully to the authors, there is a small difference that usually the activation sparsity literature aims to maximize the activation sparsity, while this work aims to minimize. But still, the concept itself is not new, and there are some well-established techniques to control the activation sparsity which can generalize to the case of minimization. Also, the relationship between two notions have already been studied in some prior works (e.g., https://arxiv.org/abs/2112.13896 ).\n\nAlso, it seems like the manuscript is still under preparation. In the bottom of page 10, authors mention that \"We intend to show results on larger datasets (e.g., ImageNet-1000) in the camera ready version.\" TMLR is a journal, where there is no fixed submission deadline. In this respect, deferring the ImageNet-1k result for the camera ready is not really understandable. If authors need more time, perhaps consider submitting the paper after all results are ready.\n\nThe writing in section 3.1. is somewhat unclear and can be improved. Could the readers clarify which pruning algorithm the authors are using? Are we doing as Han et al. (2015) does, or are we doing the full weight rewind as Frankle & Carbin (2019) does, or do a partial rewinding with the learning rate rewinding as Renda et al. does?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a mechanism (which can presumably be combined with other pruning algorithms) to improve the performance of the pruned ReLU network. The idea is to maximize the utilization rate of the neurons (of the pruned model) by pruning the model in a way that prevents outputting many zeros in the intermediate layer (i.e., minimizing the activation sparsity). To achieve this goal, authors propose a simple method: prune the negative weights, so that the pre-activation becomes larger, so that ReLU gets activated more often. The proposed method boost the performance of conventional pruning algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please re-write the main text with more details about the existing works on activation sparsity.\n\n- Compare with the activation sparsity baselines. For example, the one that uses Hoyer regularization can be used the other direction to decrease the activation sparsity, which could be a nice baseline method that this paper currently misses. The work also uses a specially designed activation function, and perhaps comparing with a similarly designed activation could help us better understand the benefit of the approach that modifies the weight pruning procedure.\n\n- Please clarify the setups on Section 3.1.\n\n- Please include the ImageNet1k experiment.", "broader_impact_concerns": "This work may not really need a broader impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission clearly meets the bar in terms of claims and evidence and interest to TMLR's audience. I encourage the authors to incorporate the following reviewer feedback in the camera-ready version:
* As per Reviewer YnV4's suggestion, using the `<X>+AP` notation, where X is either "One-Shot" or "IMP".
* Incorporating a discussion on the effect (or lack thereof) of training length in the main text.
* Combining tables and using more plots, where appropriate, and maintaining the same sparsities across Tables 1-8.
* Removing sentences that repeat the tabulated results without expanding or providing further relevant information.


Decision: Accept (poster)