Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection
Authors: Jianwei Li
Weizhi Gao
Qi Lei
Dongkuan Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Pruning
Random
Language Models
TL;DR: The paper introduces a new model pruning method using randomly generated masks and an early evaluation pipeline, achieving top performance on GLUE datasets, particularly at high sparsity levels

Abstract: It is widely acknowledged that large and sparse models have higher accuracy than small and dense models under the same model size constraints. This motivates us to train a large model and then remove its redundant neurons or weights by pruning. Most existing works pruned the networks in a deterministic way, the performance of which solely depends on a single pruning criterion and thus lacks variety. Instead, in this paper, we propose a model pruning strategy that first generates several pruning masks in a designed random way. Subsequently, along with an effective mask-selection rule, the optimal mask is chosen from the pool of mask candidates. To further enhance efficiency, we introduce an early mask evaluation strategy, mitigating the overhead associated with training multiple masks. Our extensive experiments demonstrate that this approach achieves state-of-the-art performance across eight datasets from GLUE, particularly excelling at high levels of sparsity.

Breaking through Deterministic Barriers:
Randomized Pruning Mask Generation and Selection
Jianwei Li
Weizhi Gao!
Qi Lei
Dongkuan Xu
North Carolina State University, (jli265, wgao23, dxu27| @ncsu.edu
New York University, q1518 8 nyu.edu
Abstract
Min
28.74%
Max
15.0%
34.8%
It is widely acknowledged that large and sparse models have higher accuracy than small and dense models under the same model size straints. This motivates us to train a large model and then remove redundant neurons weights by pruning- Most existing works pruned the networks in a deterministic way, the performance of which solely depends on a single pruning criterion and thus lacks va- riety. Instead, this paper, we propose model pruning strategy that first generates sev- eral pruning masks in a designed random way. Subsequently, along with an effective mask- selection rule, the optimal mask is chosen from the pool of mask candidates. To further en- hance efficiency, we introduce an early mask evaluation strategy, mitigating the overhead as- sociated with training multiple masks. Our ex- tensive experiments demonstrate that this ap- proach achieves wate-of-He of-the-art performance across eight datasets from GLUE, particularly excelling high levels of sparsity.
Magnitude
1.15e-8 5e-8
2/3r
0.027
4/3r
0.8645
(1)
Min
28.16
28.16%
Max
64%
18.35
9.8%
7.18%
Magnitude
1.15e-8
2/3r
0.055
4/3r
0.8645
(2)
Figure 1: Weight Distribution Feedforward Layer of
BERT Base at Various Sparsity Levels (0.52 and 0.83), Corresponding to Pruning Thresholds T = 0.027 and == 0.055. Notably, around 29% of the weights lic within the range [ 5. This observation puts into question the efficacy of magnitude- pruning, as these weights, despite their proximity the threshold. might play a crucial role in maintaining the model's is ac- curacy. This suggests that directly eliminating weights with smaller magnitudes could potentially lead a sub- optimal pruning strategy.
Base
1 Introduction
One of the main challenges in deploying large neu- ral networks (such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020)) in production is the huge memory footprint and computational costs, Meanwhile, studies show that large and sparse mod- els often yield higher accuracy than small but dense models (Gomez al., 2019). As result, pruning has been popularized to dramatically reduce mem- ory size and computational power consumption with little to no performance degradation. (Hoe- fler et al., 2021; Glorot et al., 2011; Kaplan et al., 2020; Li et al., 2020; Mhaskar and Poggio, 2016; Brutzkus et al., 2017; Du al., 2018).
2020), namely to remove the elements with the smallest magnitude. Here the magnitude refers to not only the weights but also the output sensitiv- ity, gradients, Hessian matrices of the training loss (Luo et al., 2017: Yu al., 2018; He et al., 2019; Lis al., 2019; Molchanov al., 2019; Singh and Alistarh, 2020; Dong et al., 2017). While magnitude- -hase pruning can generate state-of-the- art results in a wide range of tasks, its pruning strategy is deterministic and solely depends on a single criterion, which lacks variety (we demon- strate this more thoroughly in the next paragraph). Furthermore, magnitude-based pruning is proven not optimal high-level sparsity (Sanh al., 2020). To further improve the pruning perfor- mance, Zhuang et al. (2020); Ge et al. (2011); Savarese et al. (2020); Verdenius et al. (2020);
Pruning aims eliminate redundant weights, neurons, and even layers in models. Many works focus on magnitude-based pruning (Hagiwara, 1993; Gale et al., 2019; Thimm and Fiesler; Han et al., 2015; 5; Zhu and Gupta, 2017; Cuadros et al.,

---

Azarian et al. (2020) try to enlarge the search space of sparse architecture with regularization-bane methods, which are non-deterministic. They de- sign fine-designed Lo or L1 penalty terms added to the loss function. In this way, the model proac- tively shrinks some of the weights until they don't t contribute to the final loss. Regularization-ben based methods can achieve noticeably better results than magnitude--asee methods, especially at high-level sparsity (Sanh al., 2020). However, this line of work often suffers from non-convex landscape and is challenging to optimize with extra hyper- parameters. In parallel, Su et al. (2020); Liu et al. (2022) adopt a more aggressive strategy to prune el- ements in completely random way. Their methods demonstrate the competitive effect in small datasets (such CIFAR-10 10, CIFAR-100 (Krizhevsky et al., 2009)) but fail in large datasets (such Ima- geNet (Deng et al., 2009)). Different from these works, this paper introduces a mildly random prun- ing method that brings a controllable degree of randomness into the pruning mask generation pro- cedure.
these results have not been universally extended language models. We conjecture that their methods introduce unbridled randomness but do not pro- vide any effective negative feedback. Moreover, exploring the extent of introduced randomness in principled way has also not been explored in the previous literature. In this paper, we study and extend the above question systematically.
Can we design consistently effec- tive randomized pruning method?
Question 2.
This paper answers the above question with the following contribution. First, we propose a ran- domized pruning mask generation strategy that can introduce controllable randomness in a principled way. Second, we design Mask Candidate Selec- tion Strategy (MCSS) to choose the optimal mask from the pool of mask candidates, ensuring the introduced randomness always guides pruning in a beneficial direction. Third, to further enhance efficiency, we introduce Early Mask Evaluation Pipeline (EMEP) to mitigate the overhead associ- ated with training under multiple pruning masks. Last, offer empirical guidance for random- ized pruning on Bertbase and Bertiarge- Our results show a consistent accuracy boost (0.1 2266% on the GLUE benchmark, outperforming other state- of-the-art pruning techniques at a 16x compression rate. Notably, our approach showcases even more significant enhancements (2 (2%~44% %) at extreme sparsity levels like the 100x compression rate.
We demonstrate the weakness of magnitude- based pruning in Figure 1. It presents the weight distribution of a feedforward layer of BERT (De- vlin et al., 2019). Define T as the pruning bound- ary. We consider two scenarios: T 0.027 and I = 0.055, leading to sparsity levels of 0.52 and 0.83, respectively. As shown in Figure 1, a large portion of weights (rs 29%) falls into the range of T 1T, which cannot be overlooked because that it is unclear if the pruned weights close to the threshold contribute less than the kept weights the final accuracy. The weights with smaller magnitudes can still be crucial, especially when dealing with edge cases or infrequent situations. Proximity between weights can intensify decision- making challenges. This is why the direct removal of weights with smaller magnitudes is sub-optimal, also demonstrated in Gomez et al. (2019). Based on the above observations, we investigate the fol- lowing questions in this paper:
2 Preliminaries
2.1 Pruning
Iterative Magnitude Pruning Iterative Magni- tude Pruning (IMP) is the most well-known strat- egy because it yields state-of-art results than oth- (Frankle and Carbin, 2019; Frankle al., 2020), such as Single-shot Network Pruning (SNIP) (Lee et al., 2018). Specifically, we divide the pruning process into multiple stages by gradually increas- ing the sparsity. In each stage, pruning is to find and eliminate redundant parameters or neurons at that time. The most intuitive approach is to assign an importance score to each element and keep only the top-k elements. The score used to rank ele- ments can be the absolute value of weights, output sensitivity, gradients, or other fine-designed met- rics (Hagiwara, 1993; Gale al., 2019; Thimm and Fiesler; Han et al., 2015; Zhu and Gupta, 2017; Cuadros et al., 2020; Luo et al., 2017). In this work,
Which is better for pruning? a de- terministic way or a randomized way?
Question 1.
Previous literature has not reached consistent conclusion. While Su et al. (2020) and Liu al. (2022) have provided evidence that random pruning can yield competitive or better results compared to deterministic methods, this finding does not con- sistently hold true for larger datasets. Moreover,

---

different from the traditional deterministic way, we extend the IMP random way.
we aim to introduce a certain degree of random- ness into the process of pruning mask generation, thereby expanding the search space for locally op- timal pruming masks at each stage. Specifically, we propose a strategy for generating and selecting randomized pruning masks at cach pruning stage.
2.2 Knowledge Distillation
Knowledge Distillation (KD) (Hinton et al., 2015) is another compressing technique trying to transfer the knowledge from well-trained large model T to a small model S. Many previous works have proved that pruning with KD can significantly reduce accu- racy loss for Transformer-basel models (Xu et al.. 2021; Xia et al., 2022). Our experiments evalu- ate the pruning methods based on BERT (Devlin et al., 2019), and we apply the KD method to both the baseline and our strategy. Specifically, we dis- till the knowledge from the hidden state of each transformer block and the attention score of each self-attention layer. Figure 6 demonstrates the dis- tillation strategy in our settings.
3.2 Randomized Pruning Mask Generation
3.2.1 Mask Sampling
Different from the deterministic mask generation, we seek to infuse controllable randomness into this process. In essence, our approach is to sample the retained clements from a multinomial distribution without replacement. Specifically, the first step is to derive probability distribution by normalizing the magnitude of the elements. Within our frame- work, magnitude defined as the absolute value of the weight. Subsequently, we sample & indices from this distribution, where & represents the num- ber of elements retained. Finally, we generate a binary mask, with locations corresponding to these indices set to one, effectively outlining the sparse architecture of the model. The utilization of this approach offers refreshing departure from the de- terministic way and creates a larger optimization space for model pruning.
2.3 Multinomial Distribution
In probability theory, a multinomial distribution describes the probability distribution of n (n > 2) sampling trials from elements with & (k: > 2) cat- egories, which is generalization of the binomial distribution (Ross, 2010). In our setting, the num- ber of categories equals the number of elements, and the target sparsity and the total number of el- ements determine the number of trials. Note that the sampling process in this paper is done without replacement. This kind of sampling is also referred to as sampling from a multivariate hypergeometric distribution (Berkopec, 2007).
3.2.2 Controllable Randomness
We have proposed random method to generate pruning masks. However, for current models with several million parameters per layer, a single itera- tion of sampling leads to considerable randomness due to the minute probability post-normalization. To quantify this randomness, we propose ir (intro- duced randomness) Equation 1:
3 Methodology
In this section, we first rethink the traditional deter- ministic pruning method and introduce our basic idea and method. Following that, we elaborate on the details of our randomized pruning mask genera- tion and selection strategy. The architecture of our strategy is depicted in Figure 2, and the detailed procedure is outlined step-by-step in Algorithm 1.
(C + sparsity C.)/C.
Here, C' and C, represent the total count of weights and the count of weights pruned by both determin- istic and random approaches, respectively. A small value of ir indicates that the sampled mask resem- bles the deterministic one. Conversely, a larger value suggests a noticeable departure from the de- terministic method.
3.1 Rethink Iterative Magnitude Pruning
Traditional IMP divides pruning into multiple stages and generates a deterministic pruning mask by retaining the top-k elements at each stage. This process is based on the assumption that the top-k elements contribute more than the removed part. However, given the complex topology of model architecture and observations from Figure 1, it is difficult to draw such a conclusion. this paper,
We assess the introduced randomness with ir and simultaneously strive to regulate its quantity. Drawing inspiration from the concept of model soup (Wortsman al., 2022), we manage the ran- domness by sampling M masks and adding them element-wist wise to craft an ensemble mask. This mask has its top-k values set to 1, with the remainder set to 0, thus yielding a mask with controllable

---

lterative Magnitude Based Pruning
Dense
Prine
Spaniity 0.31
Prime
Sparily 0.52
Pune
Deterministic Praning
Randomize Proning
Randomized Pruning Mask Selection
Abs (Weigtt Marnx)
Mark Early Exalnation Pipeline
Train mark candidate with linge veliddeion highest
Mose
Randocniaed Pruning Mask Generation
Top
Samplis
Top
Pruning Madk
Probubility Distriation
Spanse Weight
Figure 2: Main Architecture of Our Strategy. We replace the deterministic mask generation way in IMP with our randomized method. Specifically, we first introduce degree of randomness into the process of mask generation principled way, @ then we employ specific mask selection rule, paired with an efficient evaluation pipe, to distinguish the optimal mask from pool of candidates.
randomness (k is the number of kept elements). Importantly, the degree of introduced randomness shows a positive correlation with the value of M.
racy because we introduce randomness without pro- viding any effective negative feedback to the model optimization. To address this, we propose Mask Candidate Selection Strategy (MCSS) to ensure the introduced randomness always guides the model optimization in beneficial direction. Specifically, at cach pruning stage, generate N candidate masks and select the best for the next prun- ing stage. To ensure robustness in our approach. we adopt a deterministic mask as one our mask candidates. By doing so, we are not solcly relying on random or heuristic methods but also have a reliable fallback.
3.2.3 Accelerated Mask Sampling
Controlling randomness solely by increasing the number of sampled masks can be time-intensive. To address this, we suggest deriving the sampling probability distribution from the w", where w is the weight of the corresponding layer. In this sce- nario, the power value T in the exponential term is used control the variance of the sampling proba- bility. As T increases, the sampling probabilities for larger and smaller magnitudes diverge more, allowing Mask Sampling to curtail the introduced randomness swiftly. Moreover, aligning with our motivation to introduce randomness in mask gener- ation, we only sample weights whose magnitudes are close to the pruning boundary We introduce more details Appendix.
3.3.2 Early Mask Evaluation Pipeline
To accelerate the mask selection, we design Early Mask Evaluation Pipeline (EMEP) to reduce com- putational costs. Specifically, we only fine-tune the model one epoch with large learning rate for each candidate mask. The candidate that achieves a superior early-stop evaluation metric on the valida- tion dataset is then deemed the winner. We crafted this strategy based on findings by Li et al. (2019) and You al. (2019), which suggest that using high learning rate during the earlier optimiza- tion iterations can yield a good approximation the sparse network structure. Once the winner has
3.3 Randomized Pruning Mask Selection
3.3.1
Mask Candidate Selection Strategy
Our sampling approach expands the search space for locally optimal masks compared to the deter- ministic way. However, this inadvertently intro- duces undesired noise, leading to poor model accu-

---

Algorithm I: Randomized Pruning Mask
et al., 2019). Stru-PruningRobrra (Wang et al.,
Generation and Selection
2020). SNIP (Lin et al., 2020), EBERT (Liu et 2021), BERT-of-Theseus (Xu et al., 2020), EfficientBERT (Dong et al., 2021), Sparse- BERT (Xu et al., ), RPP (Guo et al., 2019), Pretrained Ticket (Chen et al., 2020), Lottery Ticket (Prasanna et al., 2020), Prune at Pre- training (Gordon al., 2020), Movement Prun- ing (Sanh et al., 2020), DistillBerte (Sanh et al., 2019), and TinyBerte (Jiao et al., 2020).
Input: w;
/* one layer weight */
Result: w. M:
/* mask */
8 s [81,82,..]]
/* pruning schedule */
sr <- 0.00005;
1* sampling ratio */
< 8: train wi:
/* # of candidates x/
/* dense model */
foreach 8, C do
for i 4- 0 to do
< /l/
p
1* sampling prob x/
<- &t numn : 1 is of zeros
Wj after pruning /* of + nume - x non-zeros in wy after pruning */
4.2 Datasets and Data Augmentation
Following previous works, we select eight tasks from the GLUE dataset (excluding WNLI) to evaluate the effectiveness of our pruning strat- egy (Wang et al., 2018). We also follow the data augmentation method from TinyBert (Jiao et al., 2020). More details can be found in Appendix.
M; zeros like w; <- zeros__oo__ like
y + int(ar x)): for
0 to y do
pos sampling positions from
m [os]
<
4.3 Setup
M, + m
end
We follow the strategy from SparseBert (Xu al., 2021) to do pruning and knowledge distillation simultaneously at downstream tasks. We also im- itate the setting from (Frankle and Carbin, 2019) to adopt a simple pruning schedule in our experi- ments. Specifically, we only use 4-9 pruning stages to increase the sparsity gradually (such as 0.54, 0.83. 0.91, 0.9375). Furthermore, after choosing decision mask at each pruning stage, the number of epochs in the finetuning phase no longer lim- ited until the model converges. We apply exactly the same setting for the IMP baseline and our ap- proach. For more details about hyperparameters, please refer to Appendix.
M.ttopa + I; otherwise 0
M x Mj; finetune one epoch with large fy; metric, t evaluate validation dataset;
end
select M best metric;
rewind w and
x M; finetune w
/R sparse model */
end
been chosen, we revert the weights and learning rate to their state before the last pruning step. Sub- sequently, the winning candidate mask is employed for continuous regular training.
4.4 Main Results and Analysis
4 Experiments
We report the results on dev set of 8 datasets from GLUE and summarize them in Table 1-2. We also compare results with iivillaationbbased methods and describe the results in Appendix.
We evaluate the effectiveness of our pruning strat- egy in wide range of natural language understand- ing tasks. Following previous work, we BERT our backbone and then apply different pruning methods compare their performance.
From the above results, we can easily observe that our strategy consistently generates better per- formances than the main baseline IMP in a totally same setting. Moreover, the result of our method is also optimal in similar settings compared to other pruning techniques. These findings demonstrate the superiority of our random way in the mask gen- eration process over the deterministic approach and confirm that our mask selection rule can effectively navigate the optimization after introducing random- ness. In other words, our methods successfully increase the probability of finding better pruning
4.1 Baselines
BERT and BERT large are first chosen as baseline models. Based them, we ap- ply IMP generate 16x sparse models. These sparse models are used as our main baselines. In addition, we compare our strategy with pre- vious works that have reported results on the same datasets, which including: BERT-PKD (Sun

---

Methods
#params
MNLI-m Acc
QNLI Acc
QQP F1/Acc
MRPC
SST-2
COLA Mrr
RTE
STS-B B
F1
Acc
Acc
Spear
BERT Roze
110M
84.5
91.4
89.59/91.0
90.1
92.5
56.3
69.3
89.0
left #params 2 50%
BERT-PKD
50%
81.3
88.4
-/88,4
85.7
91.3
45.5
66.5
86.2
Stru Pruning
73%
89.05
88.61
92.09
88.18
SNIP
50%
82.4
89.5
88.1
91.8%
EBERT
60%
83.1
90.2
87.5/90.8
92.2
HERT-of-Theseer
50%
82.3
89.5
-/89.6
91.5
51.1
68.2
88.7
Pretrained Ticket
50%-90%
82.6
88.9
-/90.0
84.9
91.9
53.8
66.0
88.2
Lottery Ticket
38%-51 %
84.0
91.0
-/91.0
84.0
92.0
54.0
61.0
88.0
IMP
50%
84.6
91.3
88.0/91.0
90.8
92.8
53.1
72.0
89.4
Ours
50%
84.7
91.5
88.1/91.1
91.5
93.0
54.3
72.3
89.5
left #params < 10%
RPP
10%
78
87
88.0/-
80.0
89
Movement Pruning
10%
80.7
87.1/90.5
EfficienBERT
9%
81.7
89.3
86.7/-
90.1
90.1
39.1
63.2
79.9
SparseBERT
5%
90.6
88.5
52.1
69.1
IMP
6%
83.3
90.5
87.6/90.8
90.2
92.2
53.1
66.7
87.0
Ours
83.4
90.9
87.9/90.9
91.5
92.7
53.4
69.3
87.5
Table Main Comparison Results between Our Strategy and Other Pruning Baselines with BertBase on dev Sets of 8 Datasets from GLUE Benchmark. Note that the pruning results of IMP and Ours are achieved by ourselves in a totally same setting, while others are from the corresponding literature.
masks by introducing randomness in principled way.
4.5.1 Impact of Randomness and Schedule
Prior studies have demonstrated that there is no loss in accuracy at lower levels of sparsity, particularly when sparsity is less than 50%. This suggests that the model maintains robustness with the architec- tures identified in the early stages of pruning. We conjecture there high level of redundancy the weights at the early pruning stage. As such, introducing more randomness could potentially ex- pand the search space for sparse architecture in the early stages without hurt to the accuracy. How- ever, previous works also argue the concept of early model adaptions and emphasize the importance of early searched architecture to the final pruming tar- gets. Thus, introducing too much randomness at the beginning stage may not be good idea.
We also notice that the potential improvement of performance may be limited by the magnitude used to derive sampling probability. In our set- ting, we use the absolute value of weights to de- cide the importance of each neuron connection. Thus our pruning results can not surpass the theo- retical optimal result (upper bound) when pruning with the absolute value of weights. This reminds that method can be easily transplanted to other magnitude-based pruning methods, such as the gradients- based methods, and may produce the same effect, helping us find better pruning masks. Furthermore, we realize that the effect of our strategy in different datasets is not uniform. We obtain a more noticeable improvement in accuracy small datasets. We argue that small datasets have more local minimums of loss surface, and therefore our strategy can more easily help find better pruning masks.
In the last pruning stage, the magnitudes of the remaining weights significantly impact accuracy. Careful selection is required for the elements to be pruned. Too much randomness could corrupt the model, hindering recovery from the last pruning step, while too little might restrict our search for potentially optimal masks. Hence, deciding the best schedule of randomness at each pruning stage crucial.
4.5 Ablation Study
We try different ablation settings to figure out the functionality of each part of our strategy and ana- lyze why they are effective for model pruning.

---

Methods
#params
MNLI-m Acc
QNLI Aec
QQP
MRPC
SST-2
COLA
RTE
STS-B B
F1
F1
Acc
Mrr
Acc
Spear
BERT Targe
330M
86.6
92.3
91.3
89.1
93.2
60.6
74.8
90.0
IMP
20%
85.2
91.6
90.8
90.9
92.8
59.0
73.2
89.1
Ours
20%
86.2
91.8
91.1
91.9
93.7
60.9
75.5
89.9
LERTLarge Table 2: Comparison between Our Strategy and IMP with BERT Large on GLUU dev Sets.
0.88
0.925
0.87
0.66
0.920
0.86
Method
0.64
Method
Method
Randomness (decreese)
Randomness (decrease)
0.915
Randomness (decrease)
Randomness (increase)
0.62
Randomness (increese)
Randomness lincrease)
0.910
0.04
0.5
0.0
0.5
1.0
0.5
0.0
0.5
1.0
0.5
0.0
0.5
1.0
(a) MRPC 16x
(b) RTE 16x
(c) SST-2 16x
Figure 3: Comparing the Impact of Randomness in Two Different Schedules with a Deterministic Approach (IMP), which features zero randomness. The horizontal axis presents the logarithmic outputs ir, with larger ir indicating greater amount of total introduced randomness. The vertical axis signifies the model's accuracy.
To investigate the impact of randomness, we introduce hyper-parameter sr that controls the number of sampled masks M,, and thereby con- trols the total introduced randomness. We also propose two simple randomness schedules at vary- ing pruning stages: (1) Decrease, where we re- duce the introduced randomness by increasing the number of sampled masks the count of pruned weights increases (M
This proves that the model accuracy is sensi- tive to randomness in the early pruning stage and is gradually becoming sensitive as approaches target sparsity.
4.5.2 Impact of MCSS
We assessed the role of MCSS by removing it from strategy and comparing the results with primary findings. The results summarized in Figure 4. We make the following observation: 1) the setting with MCSS, there a certain threshold of randomness below which our strategy signif- icantly outperforms the deterministic way. 2) In contrast, In the setting without MCSS, the model's performance against the deterministic approach is inconsistent and lacks clear trend or pattern. This precisely demonstrates that MCSS ensure the introduced randomness consistently guides the model optimization toward a beneficial direction. In other words, MCSS effectively enhances the lower accuracy boundary in our experiments.
Cpruned). and (2) Increase, where we enhance the introduced ran- domness by decreasing the number sampled masks as the count of pruned weights increases (M = sr (C Cpruned))-
We conduct experiments comparing these two schedules against our primary baseline (IMP) under different values. The results displayed in Fig- ure leading us the following observations: 1) Excessive randomness results in our strategy per- forming even worse than the deterministic method (the region blue and orange lines below the green line). In this region, the Decrease strategy outper- forms the Increase strategy. 2) Existing threshold below which both our two randomness schedules outperform IMP, highlighting the superiority of our random approach over the deterministic way. 3) Existing another threshold above which the perfor- mances of the two randomness schedules become virtually identical. 4) The Decrease strategy consis- tently equals or outperforms the Increase strategy.
4.5.3 Impact of Sparsity
We have examined our strategy across various lev- cls of sparsity, and the findings have been encapsu- lated in Figure 5(a). We observe that random pruning strategy has consistently demonstrated perior performance compared to the baseline (IMP) across all levels of compression. The advantage is particularly pronounced at higher levels sparsity,

---

0.88
0.87
0,860
0.86
0.840
Method
0.85
Randomness (decrease)
0.820
Method
Randomness (increase)
Bandormess (decrewon)
0.800
-0.5
0.0
0.5
20%
60x
80x
100k
(a) MRPC with MCSS
(a)
0.880
0.875
0.86
0.865
0.84
Method
0.860
0.83
Randomness (decrease)
Randomness (Increase)
IMP
0.850
0.81
-0.5
0.0
0.5
1.0
(b)
(b) MRPC w/o MCSS
Figure 5: Impact Sparsities 5(a) and Impact of #Mask Candidates in MCSS 5(b), The horizontal axes represent sparsity level and the number mask candidates for Figures 5(a) and 5(b) respectively, while the vertical axes in both figures denote model accuracy.
Figure 4: Mask Sampling 4(a) v.s. Mask Sampling + MCSS 4(b). Note that the green line in 4(a) and 4(b) represents the same value of accuracy from IMP. The value on the horizontal axis represents the amount of introduced randomness. The value the vertical axis indicates model accuracy.
such as those equal to or greater than 16x sparsity.
additional computation mainly from the random- ized mask generation and selection. For the gen- eration process, it's crucial to emphasize that the creation of each mask is independent of the oth- ers. This independence allows for parallel process- ing, meaning that the time consumption doesn's 't increase linearly as the increase of mask candidates. On the other hand, we measured the GFLOPS re- quired generate single mask and compared it with the GFLOPS needed for one forward pass BERT base- It's roughly percent to the later opera- tion. However, due to implementation challenges, we couldn I concurrnttyy sample k positions mul- tiple times from the weight matrix, leading to overall increase in processing time for single ran- domized mask generation. For the selection pro- cess, we only require one epoch identify the optimal mask, where the overhead is minimal com- pared with the entire pruning process.
4.5.4 Impact of Masks Candidates
To verify the relationship between the number of mask candidates in MCSS with the final perfor- mance, we design experiments to increase the num- ber of candidate masks at each pruning stage from 2 to 10, and the results are depicted in Figure 5(b). We conclude a positive correlation between the quality of the searched mask and the number of candidate masks each stage. As the number of mask candidates close 10, the performance gain is gradually missing. Additionally, the variance of the performance gain is also gradually minimized, which proves that the MCSS can effectively nav- igate the optimization of pruning in beneficial direction.
4.6 Discussion
4.6.1 Efficiency Analysis
We analyze the efficiency of our method through the training and inference phases.
Inference Phase:
In real-world applications, al- though there might be overheads during training.
the training phase, we com- pare the required computation of our method with the traditional IMP method. It's easy to find that
Training Phase:
the benefits reaped during inference make it worth- while. Our method stands out with a 16x com- pression rate and can sustain performance even at

---

higher sparsity levels, achieving up 100x com- pression rate. This ensures that our pruned neural networks, once deployed, bring about significant improvements in performance and efficiency.
loss function and are difficult to navigate optimiza- tion. On the other hand, there lack of research on random pruning applied to Transformer- based models (such as BERT) in previous studies. There- fore, our paper complements the gap in this area.
4.6.2 Extending to Billion Parameters
Conclusion
In the current age of large language models, achiev- ing effective pruming is a formidable challenge, particularly when striving to preserve high spar- sity without sacrificing performance. While initia- tives like SparseGPT have ventured into pruning for these colossal models, they have only managed 2x compression rate (Frantar and Alistarh, 2023). The computational complexity of our method is primarily determined by the number of parame- ters involved. Consequently, our random pruning technique is not yet adaptable to models with bil- lions of parameters. Nevertheless, we are diligently working on refining methods that incorporate con- trollable randomness more efficiently.
This paper presents a non-deterministic model prun- ing method, introducing controllable randomness by generating binary masks in specific random fashion. Coupled with our specific mask candidate selection rule, our method exhibits significant effec- tiveness enhancing model accuracy in pruning, particularly at high levels of sparsity.
Limitations
Previous random pruning techniques do not scale to large datasets probably because the search space is too large to find winning pruning mask. There's considerable likelihood that our method isn't 't flaw- less either. For example, more fine-designed randomness schedule could potentially yield more substantial benefits. In addition, our method might not be appropriate for models with billions of pa- rameters due to the cost of training under multiple pruning masks. A potential solution could be to execute this process in parallel since pruning masks are independent of each other.
5 Related Work
A number of researchers have explored pruning BERT. Prasanna et al. (2020) prunes the model with Michel et al. (2019)'s first-order importance metric and proves that unstructured magnitude- based pruming always produces sparser and higher- accuracy models than structured pruning. Gordon et al. (2020) and Chen et al. (2020) both argue that pruning in the pre-trained stage is better than fine- tuning stage because there is no need to prune for each downstream task. They also conclude that knowledge in sparse training can be transferred as well as dense models. In contrast, Xu et al. (2021) find that pruning at the pre-trained stage has huge computational problems while pruning at the finetuning stage can save computational efforts and keep accuracy simultaneously. These pruning methods are based on the magniude and prune weights in a deterministic way. In parallel, linear works defeats magnitude- based methods at high- level sparsity by applying a non-deterministic way: regularization-bant based pruning. Specifically, a fine- designed Lo or L1 penalty terms are added to the loss function. Then the model proactively shrinks some of the weights until they do not contribute to the final loss. The regularization-basee method can generally achieve significantly better results than the magnitude-based methods, especially at high- level sparsity (Sanh et al., 2020). However, penalty terms can introduce additional local minima to the
Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments.
Ethics Statement
This research is conducted in compliance with the ACL Ethics Policy. In the pursuit of advancing the efficiency of model pruning strategies, our meth- ods raise several ethical considerations. First, our research proposes new model pruning strategy that significantly enhances the efficiency neural networks, which could potentially contribute to the democratization of artificial intelligence by making it accessible to systems with lower computational resources.
However, despite these potential benefits, the increased efficiency and performance of neural net- works might lead to misuse if not properly reg- ulated. For instance, such improvements could potentially contribute to the spread of deepfake technology, an area of AI that has been used to

---

spread disinformation and cause harm. It impor- tant to note that our rescarch conducted with the intent of improving efficiency and accessibility in AI, and we explicitly denounce any misuse of our work for harmful purposes. We encourage further discussions on the implementation of safeguards to prevent misuse and to guide the ethical use of such technology.
volume 33, pages 1877-1901. Curran Associates, Inc.
Alon Brutzkus, Amir Globerson. Eran Malach, and Shai Shalev-Shwartz. 2017. Sgd learns over- parameterized networks that provably general- ize on linearly separable data. arXiv preprint arXiv: 10174.
Daniel Cer, Mona Diab, Eneko Agirre, lnigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1Ith International Workshop on Semantic Evaluation (SemEval- 2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
In addition, we must consider the implications of our method on fairness and bias. While our work is focused on model efficiency, the datasets used in the process may contain inherent biases, which can result in the propagation of these biases in pruned models. We believe it is essential to ensure that data used in our processes is as unbiased and representative as possible, and to strive for transparency in this regard.
Tianlong Chen, Jonathan Frankle. Shiyu Chang. Sijia Liu. Yang Zhang. Zhangyang Wang. and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 15834 15846. Curran Associates, Inc.
Lastly, we acknowledge that the reduction of model size may result in the loss of interpretability. as smaller models often have more complex and less interpretable decision-making processes. We urge researchers and practitioners to maintain balance between model efficiency and transparency ensure fair and explainable Al practices.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018 Quora question pairs.
Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. 2020. Filter distillation for network com- pression. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 3140-3149.
We hope that our work will stimulate further discussions around these considerations, and that it will encourage future research to continuously consider the ethical implications of their work.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. 2009, Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
References
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kambiz Azarian, Yash Bhalgat, Jinwon Lee, and Tij- men Blankevoort. 2020. Learned threshold pruming. arXiv preprint 200000003..
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
Aleš Berkopec. 2007. Hyperquick algorithm for dis-
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
crete hypergeometric distribution. Journal of Dis- crete Algorithms, 5(2):3413347 2004 Symposium on String Processing and Information Retrieval.
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng, Xiaozhe Ren, and Xiaodan Liang. 2021. Efficient- BERT: Progressively searching multilayer percep- tron via warm-up up knowledge distillation. In Find- ings of the Association for Computational Linguis- ties: EMNLP 2021, pages 1424-1437, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems,
Xin Dong, Shangyu Chen, and Sinno Pan. 2017. Leam- ing to prune deep neural networks via layer-wise op- timal brain surgeon. Advances in Neural Information Processing Systems, 30.

---

Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 17929.
Song Han, Huizi Mao, and William Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman cod- ing. arXiv preprint ar0iv:1510.00149.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019. Filter pruning via geometric me- dian for deep convolutional neural networks acceler- ation. In Proceedings of the IEEE/CVF conference computer vision and patiern recognition, pages 434004349.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 2018. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.0205.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 2(7).
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OvenReview.net.
Torsten Hoefler, Dan Alistarh, Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode con- nectivity and the lottery ticket hypothesis. In Inter- national Conference on Machine Learning, pages 3259-3269, PMLR.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4163- 4174, Online. Association for Computational Lin- guistics.
Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Mas- sive language models can be accurately pruned in one-shot.
Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state sparsity in deep neural networks. ArXiv, obs/1990..9554.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. abs/2001.08361.
Dongdong Ge, Xiaoye Jiang, and Yinyu Ye. 2011. note on the complexity of minimization. Mathe- matical programming, 129.2)225--299.
CoRR,
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 201 I. Deep sparse rectifier neural networks. In Pro- ceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 315-323, Fort Lauderdale, FL, USA. PMLR.
Alex Krizhevsky, Geoffrey Hinton, al. 2009. Learn- ing multiple layers of features from tiny images.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint 1rXiv:1810.02340..
Aidan N Gomez, Ivan Zhang, Siddhartha Rao Ka- malakara, Divyam Madaan, Kevin Swersky, Yarin Gal, and Geoffrey E Hinton. 2019. Learning sparse networks using targeted dropout, arXiv preprint arXiv:1905.13678
Yuanzhi Li, Colin Wei, and Tengyu Ma. 2019. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP. pages 143-155, Online. Association for Com- putational Linguisties.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for ef- ficient training and inference of transformers. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5958 5968. PMLR
Fu-Ming Guo, Sijia Liu, F. Mungall, Xue Lin, and Yanzhi Wang. 2019. Reweighted proximal prun- ing for large-scale language representation. ArXiv. abs/1909, 12486.
Zi Lin, Jeremiah Liu, Zi Yang. Nan Hua, and Dan Roth. 2020. Pruning redundant mappings transformer models via spectral-normalized identity prior. In Findings of the Association for Computational Lin- guistics: EMNLP 2020. pages 719-730, Online. As- sociation for Computational Linguistics.
M. Hagiwara. 1993. Removal of hidden units and weights for back propagation networks. In Proceed- ings of 1993 International Conference on Neural Net- works (IJCNN-93-N) -NNgoyy, Japan), volume I. pages 351-354 v.

---

Mieszko Lis, Maximilian Golub, and Guy Lemieux. 2019. Full deep neural network training on a pruned weight budget. Proceedings of Machine Learning and Systems, 1:252-263.
Sheldon M. Ross. 2010. Chapter random variables. In Sheldon M. Ross, editor, Introduction to Proba- bility Models (Tenhh Edition), tenth edition edition, pages 21-95. Academic Press, Boston.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang. and Mykola Pechenizkiy. 2022. The unreasonable ef- fectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXiv:2202.026443
Devendra Sachan and Graham Neubig. 2018. Parame- ter sharing methods for multilingual attentatentionl translation models. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pages 261-271, Brussels, Belgium. Association for Computational Linguistics.
Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. 2021. EBERT: Efficient BERT inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814 Online. Association for Computa- tional Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108..
Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Sys- tems, volume 33, pages 90378-20389 Curran Asso- ciates, Inc.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-5066.
Pedro Savarese, Hugo Silva, and Michael Maire. 2020. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 1380-113900
Hrushikesh N Mhaskar and Tomaso Poggio. 2016. Deep vs. shallow networks: An approximation theory per- spective. Analysis and Applications, 14(06):829- 848.
Sidak Pal Singh and Dan Alistarh. 2020. Woodfisher: Efficient second-order approximation for neural net- work compression. Advances in Neural Information Processing Systems, 33:1 18098-18109.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? Advances neural information processing systems, 32.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estima- tion for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11264-112 11272.
Richard Socher, Alex Perelygin, Jean WU, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631- Seattle, Washington, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. 2020. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in Neural Information Processing Systems, 33:20390-2044
Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT Plays the Lottery, All Tickets Are Win- ning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3208 3229, Online. Association for Computational Linguistics.
Siqi Sun. Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCH)) 4323 0223,433,,11 Hong Kong, China, Association for Com- putational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lec. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67
pages
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Georg Thimm and Fimile Fiesler. Evaluating pruning methods.
Stijn Verdenius, Maarten Stol, and Patrick Forré. 2020. Pruning via iterative ranking of sensitivity statistics. arXiv preprint arXiv:2006.00890

---

Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2376-2382, Online. Association for Computational Linguistics.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. 2019. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint 19XXiv:909.11995
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1151-61622 Online. Association for Computa- tional Linguistics.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching-Yunn Lin, and Larry S Davis. 2018. Nisp: Pruning net- works using neuron importance score propagation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 9194 -2003.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-64
Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 1X100071000177
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Tao Zhuang. Zhixuan Zhang. Yuheng Huang, Xiaoyi
7eng, Kai Shuang, and Xiang Li. 2020. Neuron- level structured pruning using polarization regular- izer. Advances in neural information processing sys- tems, 33:9865-9877..
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averag- ing weights of multiple fine-tuned models improves accuracy without increasing inference time. In In- ternational Conference on Machine Learning, pages 23965-23998. PMLR.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contex- tual augmentation. In Computational Science- -CCS 2019: 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV 19, pages 95- Springer.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers). pages 1513-1528, Dublin, Ireland. Association for Computational Linguistics.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. BERT-of- -of--heseus:: Com- pressing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 7859-7869, Online. Association for Computa- tional Linguistics.
Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. 2021. Rethinking network pruning under the pre-train and tine-tune paradigm. In Proceedings of

---

A Appendices-A
the original TinyBert study (Jiao et al., 2020). However, unlike TinyBert (Jiao al., 2020), our application of data augmentation is limited to smaller datasets, namely MRPC, RTE, SST2, and CoLA, Conversely, the larger datasets MNLI, QNLI, and QQP are sizable enough for our exper- iments without the need for augmentation. Addi- tionally, enhance the STS-B dataset by merging with the MNLI-m dataset.
A.1 Model Compression
Transformer- models proven effective both in natural language processing and computer vision tasks (Dosovitskiy al., 2020; Raffel et al.. 2020). However, this series of models is restricted by its massive memory storage and computational cost at inference: for example, BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) are hardly possible to deploy in practical scenarios, with no mention for edge devices. To alleviate this problem, many approaches have been invented to compress large language models, such as Knowledge Distil- lation, Parameter Sharing (Jiao et al., 2020; Sachan and Neubig, 2018), Quantization, and Model Prun- ing. Morcover, previous works also show that train- ing a large but sparse model leads to better results than a small but dense model (Gomez et al., 2019). Therefore, pruning techniques have gained more attention than other compression techniques.
attention score
Loss
score
Aztendion output
Attention
Add Norm
Add & Norm
Feedforward 1st
Feedforward 1st
A.2 Datasets and Data Augmentation
Feodforward 2nd
Feedforward 2nd
We select eight tasks from the GLUE dataset to evaluate the effectiveness of our pruning strategy. They are: CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), 3 sentence similarity tasks: MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), QQP (Chen et al., 2018), and 3 natural language inference tasks: MNLI (Williams al., 2018), QNLI (Rajpurkar al., 2016), and RTE (Bentivogli al., 2009). The metrics for these tasks can be found in the GLUE paper (Wang et al., 2018).
Add & Norm
Teacher
Layer output
Layer output
Student
Distillation Loss
Figure 6: Details of KD in our experiments. In the prun- ing process, the sparse student model (S) continuously learns the knowledge from the dense teacher model (T).
This paper also employs the data augmentation method proposed in TinyBert (Jiao et al., 2020). We have adopted two-pronged approach for word- level replacement data augmentation by combining pre-trained language model, BERT, with Glo Ve word embeddings (Pennington et al., 2014). For single-piece words, the BERT model is used pre- dict word replacements (Wu et al., 2019), while for multi-piece words, we rely on the GloVe word embeddings to identify the most similar words as potential replacements. The process of word re- placement is guided by specific hyperparameters.. These parameters control the proportion of words replaced within sentence and the overall volume of the augmented dataset. In our methodology, maintain the same hyperparameters as utilized in BinyBert (Jiao et al., 2020). For further details re- garding these hyperparameters, we direct readers
A.3 Compare with Distillation-based Models
In addition to comparisons with other pruning- based methods, our strategy also outperforms preva- lent iisiillatiio-bbssee methods, such as DistillBerts (Sanh et al., 2019) and TinyBerts (Jiao et al., 2020), The details of these comparisons presented in Table 3.
B
Appendices-B
B.1
Acceleration of Mask Sampling
Our process involves sampling M pruning masks and summing them together to manage the intro- duced randomness. However, this method proved be time-consuming. Given our original intent incorporate randomness into the mask maak-gennrrttion process, our goal was to increase the retention prob- ability of weights near the pruning boundary during

---

models
#params
MNLI-m Ace
QNLI
QQP F1/Acc
MRPC
SST-2
COLA Mr
RTE
STS- Spear
Ace
F1
Acc
Acc
BERT RASE
110M
84.5
91.4
89.59/91.0
90.1
92.5
56.3
69.3
89.0
DistillBert
50%
82.2
89.2
-/88.5
87.5
92.7
51.3
59.9
86.9
Tiny Berts
50%
84.5
91.1
88.0/91.
90.6
93.0
54.0
73.4
89.6
ours 50%
84.7
91.5
88.1/91.1
91.5
93.0
54.3
72.3
89.5
Table 3: Comparison between Our Strategy and Distillation-based Methods with BERT Bose on GLUE dev Sets.
the pruning process. Consequently, limit sampling to target weights with magnitudes within the range of [4oopc- m], where the pruning boundary, & is the number weights retained, m is the maximum magnitude, r' is a hyperparameter used to control the sampling range. Specifically, we assigned zero sampling probability to weights with magnitudes less than (to This method provided more efficient and effective strategy for incorporating randomness into our pruning process.
Ckept C sparsity
M Ckept
B.2 More Implementation Details
We provide additional details regarding the hyper- parameters utilized in our experiments. We fol- low the settings from Frankle and Carbin (2019). implementing a straightforward pruning schedule. Specifically, for MRPC, RTE, and SST2 tasks, we employed [0.54, 0.83, 0.91, 0.9375] while for the MNLI-m, QNLI, QQP. STS-B, and CoLA tasks, the sequence [0.54, 0.83, 0.875, 0.9, 0.92, 0.9275, 0.93, 0.935, 0.9375] was adopted. Although more complex pruning schedule, such cubic ap- proach, might enhance the final performance, that not our primary objective in this research.
B.1.1 Randomness Schedule
Our method is supposed to sample M masks and then add them together to control the introduced randomness. One left concern is how much ran- domness should be introduced for each pruning stage.
We also use the hyperparameter sr to manage the number of sampling masks M, choosing from several options (3e-6, 5e-6, 8e-6, le-5, 3e-5, 5e-5). Additionally, T, used in the exponential function to accelerate the reduction of introduced random- ness, is set to 5. For each pruning stage, lected 8- 8-10 candidate masks. The learning rate remains constant during the pruning process yet linearly declines when the model achieves the tar- geted sparsity. The optimization process man- aged by Adam, with a warm-up phase constituting 10% of the steps.
As introduced Ablation 4.5.1, we propose two most straightforward sampling schedules to introduce randomness at different pruning stages: 1) Decrease, decreasing the introduced random- ness by sampling more masks with the increase of pruned weights. 2) Increase, increasing the in- troduced randomness by sampling fewer masks with the increase of pruned weights. Specifically, a sr is introduced to control these two schedules. Equation and Equation 3 demon- strate how to calculate the number of sampled masks for two strategies. Here, M is the num- ber of masks to be sampled, is the sampling ratio, Opruned and Ckept refers the number of weights pruned and kept, respectively. In the it- erative setting, with the sparsity reaching the get gradually, Cpruzed becomes larger, and Ckept becomes smaller. Therefore, given the sampling ratio, Increase gradually increases the amount of imported randomness, while Decrease gradually decreases the amount of imported randomness.
Opruned sporsity = AF + Opruned
(2)

---

deterministic mask
dynamic mask
simple
complicated implementation
easy optimizable
more
parameters
Magnitude-based
Regularization-based
lower acc
hard-decide hyperparameters
low hardware requirement
higher acc
expected sparsity
uncontroable sparsity
low hardware requirement
simple
mask with randomness
Pruning with randomness
easy optimizable
expected sparsity
higher acc
Figure 7: Pruning categories

---

Last Prune Stage
Weight
Mask sampling
Non-deterministic
Weight
Deterministic
iterative
Mask candidate selection
top-k magnitude
Best
Next Prune Stage
Figure Deterministic Pruning v.s. Our Strategy


Reviews:
[{"review_id": "Y0FIz59wqp", "reviewer": "Reviewer_cTJA", "summary": "", "questions": "- could you please elaborate a little more on the motivation? especially the problem with magnitude-based methods. please give some clarification on the lack of variety and the example in Figure 1. Why τ = 0.027 and τ = 0.055 set and why 29% cannot be simply ignored even if these 29 percent of params can be pruned based on the magnitude?\n-regarding question 2 in page 2: (Can we design a consistently effective randomized pruning method) I'm not sure how efficient your method is compared with a method like IMP? did you compare these two not just for performance but how efficient they are?\n\n- although the proposed method beats existing methods compared with IMP that you implemented yourself, the difference is not significant ~0.1-0.5%. do you know which one is better in other aspects like computation? IMP or your method?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and questions, which have offered us a clear perspective on areas for improvement. We have carefully gone through the feedback and summarized the major points raised. Below, we have listed the primary concerns and questions from the review to address them systematically in our rebuttal.\n\nHere is a summarized list of the questions extracted from the content:\n\n### Motivation Related:\n- **Question 1:** What is the limitation of deterministic magnitude-based pruning methods?\n  - Concerns about:\n    - Lack of variety in the approach.\n\n- **Question 2:** Can you explain Figure 1, specifically regarding:\n  - The reasoning behind selecting specific values of $\\( \\tau \\)$ (i.e., $\\( \\tau = 0.027 \\) and \\( \\tau = 0.055 \\)$).\n  - The significance of the 29% in the range $\\( \\left[ \\frac{2}{3} \\tau, \\frac{4}{3} \\tau \\right] \\)$.\n\n### Performance Gain and Pruning Cost Related:\n- **Question 3:** Is the performance gain from your method significant, especially in comparison to other methods like IMP?\n- **Question 4:** How efficient is your method?\n\nPlease see our responses below to clarify the main concerns. Also, if you feel that your original concerns have been resolved, we would appreciate it if you would update your evaluation to reflect this. Thank you!\n\n## Motivation Related\n\nAs highlighted in the introduction (`Line 086~091, 103~108`), traditional deterministic pruning methods are constrained by their sole reliance on magnitude information, resulting in limited adaptability. In contrast, this paper proposes an approach that infuses controlled randomness into the pruning strategy, striving to find a middle ground between fully deterministic and entirely random methods.\n\n---\n### Question 1. What is the limitation of deterministic magnitude-based pruning methods?\n\nDeterministic magnitude-based pruning methods aim to trim weights, neurons, or layers from a neural network that appear least influential. The absolute value of their weights primarily determines this. The rationale behind these methods is that eliminating what seem to be the least consequential elements can reduce the overall model size and computational requirements without a drastic loss in performance.  \n\nHowever, several challenges arise with this approach:\n\n1. **Lack of Variety:** Operating strictly under a deterministic paradigm, these methods focus solely on the magnitude of the weights for pruning decisions. Consequently, they might miss out on smaller weights that, despite their size, play pivotal roles in certain tasks, especially when handling edge cases or rarer instances.  \n  \n   **Let's consider the following small case:**\n  \n   - Neural Network Output:\n      $\\[ Y = w_i \\times F_i \\]$\n  \n    **Where:**\n    - $\\( Y \\)$ is the network output.\n    - $\\( F_i \\)$ represents a feature.\n    - $\\( w_i \\)$ is the corresponding weight.\n    \n   **Magnitude-based Pruning:**   \n    - If $\\( |w_i| < T \\)$, then $\\( w_i \\times F_i \\)$ is pruned.\n  \n   **The Highlighted Risk:**  \n    - The impact on $\\( Y \\)$ due to a feature is not solely determined by $\\( w_i \\)$. Instead, it's the combined effect of $\\( w_i \\)$ and the sensitivity of $\\( F_i \\)$.\n      - **High sensitivity (Example)**: Suppose $\\( F_i \\)$ represents the sharpness of an image, a small change in sharpness (even if $\\( w_i \\)$ is small) might drastically affect our recognition of an object in the image. Thus, even if $\\( |w_i| = 0.01 \\)$ (small weight), the influence of $\\( F_i \\)$ could make a difference of 0.5 in $\\( Y \\)$, making it critical.\n        \n      - **Low sensitivity (Example)**: On the other hand, consider $\\( F_i \\)$ representing the hue of the image background. Even if this hue has a large weight $\\( w_i = 5 \\)$, a change in the background hue might not alter our recognition much, making the product $\\( w_i \\cdot F_i \\)$ yield only a minimal difference of 0.1 in $\\( Y \\)$.\n\n   **Conclusion:**  \n    - The influence of removing weights on $\\( Y \\)$ is not solely reliant on $\\( w_i \\)$. Instead, it's the joint effect of $\\( w_i \\)$ and $\\( F_i \\)$'s sensitivity. Hence, the deterministic nature might lead to a \"lack of variety,\" potentially overlooking these nuanced relationships.\n\n2. **Sub-optimal at high-level sparsity:** \n   It's essential to underscore that deterministic magnitude-based pruning exhibits suboptimal performance when pushed to extreme sparsity levels. This limitation becomes evident at high compression rates such as 20x, 40x, 60x, 80x and 100x. Our exhaustive experiments on the BERT model across datasets like MRPC, RPC, and MNLI corroborate this observation. Notably, utilizing the Iterative Magnitude Pruning (IMP) techniques largely deteriorated performance at these higher sparsity levels. The table below details the performance degradation for various datasets and sparsity levels when using deterministic and our randomized iterative magnitude-based pruning:  \n\n   | Method (IMP) | Dataset | 20x Compression  | 40x Compression| 60x Compression | 80x Compression | 100x Compression |\n   |---------|---------|-----------------|-----------------|-----------------|-----------------|------------------|\n   | Determinstic | MRPC    |  -2.6% |  -2.7% | -5.0% |  -6.1%   | -6.7%   |\n   | **Randomized** | MRPC    | **-0.5%** | **-1.4%** | **-2.9%** | **-3.7%** | **-3.9%** |\n   | Determinstic | RPC     |  -2.8% |  -3.5% | -5.7% |  -7.4%   | -8.3%   |\n   | **Randomized** | RPC    |  **-0.2%** |  **-1.1%** | **-2.3%** | **-3.5%**  | **-4.0%**  |\n\n### Question 2. Can you explain Figure 1?\n\n  1. **Weight Distribution of BERT Layer:** \nThe figure displays the weight distribution for a particular BERT layer, spotlighting weights situated around two specific pruning thresholds: $\\( \\tau = 0.027 \\)$ and $\\( \\tau = 0.055 \\)$.\n\n  2. **Selection of $\\( \\tau = 0.027 \\) and \\( \\tau = 0.055 \\)$:**\nThe selected $\\( \\tau \\)$ values directly relate to sparsity levels of 0.52 and 0.83. Their selection is intricately connected to the Iterative Magnitude Pruning (IMP) process. Within specific phases of IMP, the task necessitates pruning 52% or 83% of the weights, guided by their magnitude. These $\\( \\tau \\)$ values act as exact pruning thresholds. It's pivotal to understand that these figures aren't arbitrary. Their determination is primarily anchored in the established IMP pruning schedule. In our illustration, while we highlighted sparsity levels of 0.52 and 0.83, it's noteworthy that analogous outcomes were consistently observed across various sparsity levels.\n\n  3. **The Significance of the 29% in the Range $\\([ \\frac{2}{3} \\tau, \\frac{4}{3} \\tau ]\\)$:**\nI apologize for any earlier ambiguities. While magnitude is undeniably significant, it doesn't singularly determine performance. Even weights with smaller magnitudes can be crucial, especially when dealing with edge cases or infrequent situations. Proximity between weights can intensify decision-making challenges. Importantly, weights within this particular range, which approach the pruning threshold, constitute 29% of all weights. This notable fraction emphasizes their potential outsized influence on the total output, marking their indispensable role.\n\n## Performance Gain and Pruning Cost Related\n---\n\nThank you for pointing out the comparison between our method and IMP. To clarify:\n\n### Question 3:  Performance Gain is not Significant\n\n1. **Performance Improvement**: In our study, we did observe a modest improvement in the range of 0.1~2.6% at a sparsity level of 16x. Nonetheless, as can be seen from Table I in our paper, our method consistently outperforms all the baselines at similar sparsity levels. The best results for each dataset are highlighted in bold. At a 16x compression rate (with 6% of the parameters retained), the performance advantage of our strategy over IMP for each dataset is as outlined below:\n       \n      | Paper  | QNLI | RTE | MRPC | QQP | SST-2 | MNLI | CoLA | STS-B |\n      | ------ | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: |\n      | Ours (acc)   | +0.4 | **+2.6** | **+1.3** | +0.3 | **+0.5** | +0.1 | +0.3 | **+0.5** |\n      | Ours (std)   |  0.1 |  0.4 | 0.1 | 0.07 | 0.1 | 0.02 | 0.1 | 0.2 |\n      | Ref    | -0.5 | +1.5 | +1.2 | +0.1 | +0.9 | +1.2 | N/A  | -1.9 |\n\n      For comparison, we also included the performance gains reported on these datasets from a previous study titled `Structured Pruning Learns Compact and Accurate Models` published at `ACL 2022`. Given these results, we believe the gains we achieved are noteworthy.\n\n2. **Gains at Higher Sparsity Levels**: Our method exhibits even more pronounced improvements, in the range of 2~4%, at higher sparsity levels such as 20x, 40x, 60x, 80x, and 100x, as demonstrated in Figure 5a.\n\n### Question 4:  How efficient is your method?\n In response to your query about the efficiency of our method, we address this concern from two perspectives: the training and inference phases.\n\n  - **Training Phase**: Concerning the computational overhead of our method\n    * **Randomized Mask Generation:**  \n         **(1)** At first glance, the computation in this phase might seem directly proportional to the number of randomized mask candidates we aim to generate.  \n         **(2)** **`Utilizing Parallel Processing to Expedite Randomized Mask Generation`**: It's crucial to emphasize that the creation of each mask is independent of the others. This independence allows for parallel processing, meaning that the time consumption doesn't increase linearly as one might expect.  \n         **(3)** **`In fact, the computation required to generate one mask candidate is far less than one forward and backward pass in BERT`**: Let's delve into the computational cost of generating a single mask candidate to understand our method's efficiency better.\n         \n        **3.1** Let's first calculate the required computation by mask candidate generation step by step.\n              \n       ```python\n       reshape_weight = weight.cpu().reshape(-1)\n       eshape_weight = reshape_weight ** 5\n       weight_probality = abs(reshape_weight) / sum(abs(reshape_weight))\n\n       masks = torch.zeros_like(reshape_weight)  \n       sample_times = max(int((len(reshape_weight) - keep_k) * self._sample_ratio), 1)\n       for i in range(sample_times):\n            sampling_mask = weight_probality.multinomial(\n                num_samples=keep_k, \n                replacement=False\n            )\n            masks[sampling_mask] += 1\n            index = torch.topk(masks, keep_k)[1]\n       masks[:] = 0\n       masks[index] = 1 \n       mask = masks.reshape(weight.shape).to(weight.device)\n        ```\n     \n      1. **Reshaping and Probability Computation**: This process is `4 x d^2` FLOPs for `d x d` weight matrices and `4 x 4d x d` FLOPs for `4d x d` weight matrices. \n\n      2. **Sampling using `multinomial`**: Given the average sparsity from the schedule and `sample_times`, the computation for the `d x d` matrices is approximately `avg_sparsity x avg_sample_times x d^2` FLOPs. For the `4d x d` matrices, it is `avg_sparsity x avg_sample_times x 4d x d` FLOPs.\n\n      3. **Top-k Operation**: For `d x d` matrices, it's about `d^2 x log2(d^2 x avg_sparsity)` FLOPs. For `4d x d` matrices, it is `4d x d x log2(4d x d x avg_sparsity)` FLOPs.\n\n      4. **Total**: Given BERT base's architecture with 12 blocks, each having 4 `d x d` layers and 2 `4d x d` layers, the total computation for pruning the entire BERT model is **1.34 GFLOPs** (with 1.34 being the computed value).\n\n      **3.2** To provide a more intuitive comparison of the computation required, we calculated both the computational cost for one forward pass and one backward pass of Bert.\n      \n         1. **Multi-Head Attention**:\n            - Q, K, V calculations: $\\(3 \\times d \\times d \\times s\\)$\n            - Scaled dot-product attention: $\\(d \\times s \\times d \\times s\\)$\n            - Output projection: $\\(d \\times d \\times s\\)$\n         2. **Feed Forward Network**:\n            - First layer: $\\(4d \\times d \\times s\\)$\n            - Second layer: $\\(d \\times 4d \\times s\\)$\n         3. **Total**: Given the architecture of BERT base with 12 blocks, the total FLOPs for one forward pass is: $\\[ \\text{FLOPs for one forward pass} = 12 \\times (3d^2s + d^2s^2 + d^2s + 4d^2s + d^2s) \\]$. For the backward pass, it's approximately three times the forward pass, equating to **496.47 GFLOPs** for both forward and backward passes combined.\n\n         **3.3** Comparison: Given the above analysis, single mask candidate generation requires **1.34 GFLOPs** for the entire BERT model, whereas a single forward and backward pass of BERT demands **496.7 GFLOPs**. Importantly, while computationally involved, pruning operations are not executed as frequently as the forward and backward passes during training. Hence, over a training session, the added computational cost due to randomized mask generation remains relatively low.\n     \n    * **Randomized Mask Selection:** \n        * Requires only **one epoch** to identify the optimal mask.\n        * Consequently, the overhead here is minimal in the grand scheme of the entire pruning process.\n\n  - **Inference Phase**: It's essential to note that when assessing sparse neural networks' deployment in real-world scenarios, the computational effort during the pruning phase is a significant factor. Thus, even if there's a computational overhead during this phase, it's often justified by the benefits obtained during inference in production environments. Moreover, our method not only demonstrates superior performance at a 16x compression rate compared to other methods, but it also maintains performance at even higher sparsity levels, achieving up to a 100x compression rate. This means that when our pruned neural networks are deployed in production settings, the gains in performance and efficiency are even more pronounced.\n\n**Finally, we thank the valuable suggestions from the reviewer again.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "achieving SOTA in 8 tasks of GLUE. \nproposed the idea of controllable randomness is interesting\nextensive experiments prove the idea works (although the margin is not significant)", "reasons_to_reject": "- motivation for proposing an alternative approach for controllable randomness and the advantage over magnitude-based method is missing\n- the results of the proposed method beat SOTA. However, the difference is not significant. additionally the analysis of how the proposed framework is efficient compared with other methods like IMP is missing", "questions_to_authors": "- could you please elaborate a little more on the motivation? especially the problem with magnitude-based methods. please give some clarification on the lack of variety and the example in Figure 1. Why τ = 0.027 and τ = 0.055 set and why 29% cannot be simply ignored even if these 29 percent of params can be pruned based on the magnitude?\n-regarding question 2 in page 2: (Can we design a consistently effective randomized pruning method) I'm not sure how efficient your method is compared with a method like IMP? did you compare these two not just for performance but how efficient they are?\n\n- although the proposed method beats existing methods compared with IMP that you implemented yourself, the difference is not significant ~0.1-0.5%. do you know which one is better in other aspects like computation? IMP or your method?", "paper_topic_and_main_contributions": "the paper proposed that the deterministic-based method for pruning is sub-optimal and they proposed a pruning method that has controllable randomness for pruning the search space which could lead to better performance", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6b0HyWKyfN", "reviewer": "Reviewer_xLoZ", "summary": "", "questions": "1. What is the time and calculation cost of the method proposed in this paper?\n2. Distillation is mentioned but not explained in Section 3\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and questions, which have offered us a clear perspective on areas for improvement. We have carefully gone through the feedback and summarized the major points raised. Below, we have listed the primary concerns and questions from the review to address them systematically in our rebuttal.\n\nHere is a summarized list of the questions extracted from the content:\n\n- **Question 1:** Is the performance gain from your method significant, especially in comparison to other methods like IMP?\n- **Question 2:** Is applying the method to the current large language model with billions of parameters difficult?\n- **Question 3:** How do you find a suitable ir to prune different models?\n- **Question 4:** How efficient is your method?\n- **Question 5:** Distillation is mentioned but not explained in Section 3\n- **Question 6:** Presentation Improvement \n\nPlease see our responses below to clarify the main concerns. Also, if you feel that your original concerns have been resolved, we would appreciate it if you would update your evaluation to reflect this. Thank you!\n\n### Question 1:  Is the performance gain from your method significant, especially in comparison to other methods like IMP?\n\n1. **Performance Improvement**: In our study, we did observe a modest improvement in the range of 0.2~2.6% at a sparsity level of 16x. Nonetheless, as can be seen from Table I in our paper, our method consistently outperforms all the baselines at similar sparsity levels. The best results for each dataset are highlighted in bold. At a 16x compression rate (with 6% of the parameters retained), the performance advantage of our strategy over IMP for each dataset is as outlined below:\n       \n      | Paper  | QNLI | RTE | MRPC | QQP | SST-2 | MNLI | CoLA | STS-B |\n      | ------ | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: |\n      | Ours (acc)   | +0.4 | **+2.6** | **+1.3** | +0.3 | **+0.5** | +0.1 | +0.3 | **+0.5** |\n      | Ours (std)   |  0.1 |  0.4 | 0.1 | 0.07 | 0.1 | 0.02 | 0.1 | 0.2 |\n      | Ref    | -0.5 | +1.5 | +1.2 | +0.1 | +0.9 | +1.2 | N/A  | -1.9 |\n\n      For comparison, we also included the performance gains reported on these datasets from a previous study titled `Structured Pruning Learns Compact and Accurate Models` published at `ACL 2022`. Given these results, we believe the gains we achieved are noteworthy.\n\n2. **Gains at Higher Sparsity Levels**: Our method exhibits even more pronounced improvements, in the range of 2~3%, at higher sparsity levels such as 20x, 40x, 60x, 80x, and 100x, as demonstrated in Figure 5a.\n\n### Question 2:  Is applying the method to the current large language model with billions of parameters difficult?\n\n1. **Core Focus:** Our work's central aim is to explore the influence of randomness on traditional deterministic pruning. Direct scalability to LLMs wasn't the primary intent but understanding this interplay is foundational for future research.\n\n2. **Comparison to Existing Methods:** While methods like SparseGPT are tailored for LLMs, they don't delve into randomness's role in pruning. It's crucial for the community to understand how randomness can influence pruning before we delve deep into scaling up these techniques. \n\n3. **Evolutionary Nature of Research:** Initial research often starts with foundational exploration, leading to more scalable methods over time. We see our work as an initial step, which, while not immediately scalable, provides insights for future refinements. Recognizing the reviewer's point, we can suggest potential approaches to scale controllable random pruning methods for large language models, ensuring the broader community understands our findings and future directions.\n\n### Question 3: How do you find a suitable ir to prune different models?\n\nWe've elaborated on this in our paper, and specific locations of these details are provided below.\n\n1. **`[Line 226-235]` Firstly, the parameter `$ir$` is introduced as a metric to quantify the amount of randomness we insert during the pruning process.** As described by Equation 1, it measures the deviation of our random approach from a deterministic one. In simple terms, `$ir$` is merely an evaluative measure of the randomness we introduce, not a direct control parameter for pruning different models. \n\n2. **`[Line 238-246]` The actual control of how much randomness to introduce is governed by the number of masks we sample, denoted as `$M$`.** This `$M$` is directly influenced by the sample ratio, `$sr$`, which serves as our hyper-parameter. As detailed in our manuscript, the more masks we sample (higher `$M$`), the greater the randomness introduced. Conversely, sampling fewer masks will lead to pruning closer to deterministic methods.\n\n3. **`[Section 4.5.1: Impact of Randomness and Schedule]` We have exhaustively explored the influence of varying levels of introduced randomness (via `$ir$`) on the pruning results in our ablation study.** Figure 3 shows how different schedules of introducing randomness impact the pruning efficiency. It’s worth noting that an optimal threshold of randomness exists below which our method consistently outperforms deterministic pruning (IMP). These findings emphasize the advantage of introducing controlled randomness over purely deterministic pruning strategies.\n\n4. **`[Line 1088-1090]` Further, in the appendix of our manuscript, we have detailed the hyper-parameters (especially the sample ratio, `$sr$`) used across different experiments**, offering guidance on their applicability to various models. Our experiments with different `$sr$` values, such as `3e-6`, `5e-6`, and so forth, can serve as a reference for practitioners.\n\n5. **To conclude, while `$ir$` serves as a measure of the introduced randomness, the actual control over the randomness is exerted through the number of sampled masks `$M$`, governed by the sample ratio `$sr$`.** We believe our thorough experiments and discussions in the main manuscript and appendix address the concerns raised.\n\n### Question 4:  How efficient is your method?\n    \nIn response to your query about the efficiency of our method, we address this concern from two perspectives: the training and inference phases.\n\n  - **Training Phase**: Concerning the computational overhead of our method\n    * **Randomized Mask Generation:**  \n         **(1)** At first glance, the computation in this phase might seem directly proportional to the number of randomized mask candidates we aim to generate.  \n         **(2)** **`Utilizing Parallel Processing to Expedite Randomized Mask Generation`**: It's crucial to emphasize that the creation of each mask is independent of the others. This independence allows for parallel processing, meaning that the time consumption doesn't increase linearly as one might expect.  \n         **(3)** **`In fact, the computation required to generate one mask candidate is far less than one forward and backward pass in BERT`**: Let's delve into the computational cost of generating a single mask candidate to understand our method's efficiency better.\n         \n        **3.1** Let's first calculate the required computation by mask candidate generation step by step.\n              \n       ```python\n       reshape_weight = weight.cpu().reshape(-1)\n       eshape_weight = reshape_weight ** 5\n       weight_probality = abs(reshape_weight) / sum(abs(reshape_weight))\n\n       masks = torch.zeros_like(reshape_weight)  \n       sample_times = max(int((len(reshape_weight) - keep_k) * self._sample_ratio), 1)\n       for i in range(sample_times):\n            sampling_mask = weight_probality.multinomial(\n                num_samples=keep_k, \n                replacement=False\n            )\n            masks[sampling_mask] += 1\n            index = torch.topk(masks, keep_k)[1]\n       masks[:] = 0\n       masks[index] = 1 \n       mask = masks.reshape(weight.shape).to(weight.device)\n        ```\n     \n      1. **Reshaping and Probability Computation**: This process is `4 x d^2` FLOPs for `d x d` weight matrices and `4 x 4d x d` FLOPs for `4d x d` weight matrices. \n\n      2. **Sampling using `multinomial`**: Given the average sparsity from the schedule and `sample_times`, the computation for the `d x d` matrices is approximately `avg_sparsity x avg_sample_times x d^2` FLOPs. For the `4d x d` matrices, it is `avg_sparsity x avg_sample_times x 4d x d` FLOPs.\n\n      3. **Top-k Operation**: For `d x d` matrices, it's about `d^2 x log2(d^2 x avg_sparsity)` FLOPs. For `4d x d` matrices, it is `4d x d x log2(4d x d x avg_sparsity)` FLOPs.\n\n      4. **Total**: Given BERT base's architecture with 12 blocks, each having 4 `d x d` layers and 2 `4d x d` layers, the total computation for pruning the entire BERT model is **1.34 GFLOPs** (with 1.34 being the computed value).\n\n      **3.2** To provide a more intuitive comparison of the computation required, we calculated both the computational cost for one forward pass and one backward pass of Bert.\n      \n         1. **Multi-Head Attention**:\n            - Q, K, V calculations: $\\(3 \\times d \\times d \\times s\\)$\n            - Scaled dot-product attention: $\\(d \\times s \\times d \\times s\\)$\n            - Output projection: $\\(d \\times d \\times s\\)$\n         2. **Feed Forward Network**:\n            - First layer: $\\(4d \\times d \\times s\\)$\n            - Second layer: $\\(d \\times 4d \\times s\\)$\n         3. **Total**: Given the architecture of BERT base with 12 blocks, the total FLOPs for one forward pass is: $\\[ \\text{FLOPs for one forward pass} = 12 \\times (3d^2s + d^2s^2 + d^2s + 4d^2s + d^2s) \\]$. For the backward pass, it's approximately three times the forward pass, equating to **496.47 GFLOPs** for both forward and backward passes combined.\n\n         **3.3** Comparison: Given the above analysis, single mask candidate generation requires **1.34 GFLOPs** for the entire BERT model, whereas a single forward and backward pass of BERT demands **496.7 GFLOPs**. Importantly, while computationally involved, pruning operations are not executed as frequently as the forward and backward passes during training. Hence, over a training session, the added computational cost due to randomized mask generation remains relatively low.\n     \n    * **Randomized Mask Selection:** \n        * Requires only **one epoch** to identify the optimal mask.\n        * Consequently, the overhead here is minimal in the grand scheme of the entire pruning process.\n\n  - **Inference Phase**: It's essential to note that when assessing sparse neural networks' deployment in real-world scenarios, the computational effort during the pruning phase is a significant factor. Thus, even if there's a computational overhead during this phase, it's often justified by the benefits obtained during inference in production environments. Moreover, our method not only demonstrates superior performance at a 16x compression rate compared to other methods, but it also maintains performance at even higher sparsity levels, achieving up to a 100x compression rate. This means that when our pruned neural networks are deployed in production settings, the gains in performance and efficiency are even more pronounced.\n\n### Question 5: Distillation is mentioned but not explained in Section 3\n\n1. **Placement in Preliminaries**:  In the Preliminaries, we indeed touched upon the concept of Knowledge Distillation (KD) as an essential background. We intended to provide a foundational understanding of this technique as it is used in the literature.\n\n2. **Relation to Our Methodology**:  While Knowledge Distillation was not detailed in the Methodology section, it plays a pivotal role in our experimental setup. Specifically, in the Setup subsection, it is clearly highlighted that we incorporate KD in our approach. By stating, \"Our experiments evaluate the pruning methods based on BERT~\\citep{Devlin:2019}, and we apply the KD method to both the baseline and our strategy,\" we imply that KD is an integral part of the experimentation.\n\n3. **Connection to the Baseline**:  It is crucial to emphasize that our primary baseline, IMP, and our proposed strategy both utilize the same distillation method. This methodological consistency ensures that any performance differences observed between our proposed strategy and the baseline can be attributed to our novel contributions rather than variations in the distillation approach.\n\n4. **Knowledge Gained from Literature**:  It is important to highlight that our decision to use KD (Knowledge Distillation) to enhance performance was influenced by the work presented in SparseBert~\\citep{Xu:2021}. Our goal was to employ a proven technique to attain the best results. Additionally, many pruning methods in NLP now consider knowledge distillation a standard practice.\n\n5. **Decision on Exclusion from Methodology**:  The reason for not explicating KD in the Methodology section was our judgment that it wasn't the focal point of our work. Since KD is a widely accepted technique, we believe the emphasis should be on our unique contributions. Nevertheless, we acknowledge the importance of ensuring clarity and will consider providing a brief note on its role within the methodology in future versions.\n\n6. **In-Depth Explanation Provided**:  For readers interested in understanding our specific application of KD, we've included a dedicated subsection on Knowledge Distillation, elucidating how we distill knowledge from the hidden states and attention scores. This ensures that those keen on the details have access to this information.\n\n7. **In summary**: while Knowledge Distillation was not explicitly discussed in the methodology, it is woven into the fabric of our experimental setup. We appreciate the feedback and will ensure that all pivotal components, even if they're not the primary focus, are adequately explained in future work to avoid ambiguity.\n\n### Question 6: Presentation Improvement\n\nThank you for your meticulous observations regarding the typos and presentation in our manuscript. We sincerely appreciate your time and effort to provide this feedback.\n\n1. We have noted the error on Line 133 \"others than others\" and will rectify it in the revised version.\n2. The double semicolons in Pseudocode 3 and 4 will be addressed and corrected.\n3. We will clarify and correct the discrepancy where \"k is equal y\" in the pseudocode comments.\n4. The typo on Line 252 \"of conresponding\" will also be rectified.\n\nYour feedback has been instrumental in enhancing the clarity and accuracy of our work. We will ensure that these points are addressed in our revised version.\n\n**Finally, we thank the valuable suggestions from the reviewer again.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The description in this article is clear, the pseudo-code given is highly readable, and has an acc increase in the experiments of all tasks.\n2. The theoretical argument provided for why this method works is reasonable.\n3. The influence of each step in the pipeline is gradually analyzed, and the ablation experiment is complete.\n", "reasons_to_reject": "1. The rate of increase is too small, most of which are less than 1%.\n2. The amount of calculation is large, and it is difficult to apply to the current large language model with billions of parameters.\n3. It is not clear how to find a suitable ir to prune different models.\n", "questions_to_authors": "1. What is the time and calculation cost of the method proposed in this paper?\n2. Distillation is mentioned but not explained in Section 3\n", "paper_topic_and_main_contributions": "This article belongs to the lightweight field of NLP model, which is improved on the IMP iterative amplitude pruning. The weight of amplitude pruning near the threshold accounts for about 30%, and simply pruning the weights smaller than the threshold may not be the optimal solution. In view of the above problems, this paper derives the probability distribution by standardizing the absolute value of weights, then randomly samples multiple times, retains the first k weights to obtain masks, and obtains multiple masks repeatedly to select the optimal mask through training with a large learning rate of 1epoch. After experiments, the method proposed in this paper performs better than the original method in all tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 133 ‘others than others’\n2. Pseudocode 3, 4 lines with two semicolons\n3. k is equal y in pseudocode comments\n4. Line 252 of conresponding\n"}, "extra_text": {}}, {"review_id": "zoyweAL0zi", "reviewer": "Reviewer_gFqB", "summary": "", "questions": "1. How to evaluate and select the best one for pruning in MCSS? What if all of them are not good enough? Moreover, as more candidates lead to more complexity, I wonder the training cost of the proposed method as it is not reported and compared with other methods in the paper.\n2. How does ir work in your methods?\n3. Do you consider appling the method on larger models in addition to BERT?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and questions, which have offered us a clear perspective on areas for improvement. We have carefully gone through the feedback and summarized the major points raised. Below, we have listed the primary concerns and questions from the review to address them systematically in our rebuttal.\n\nHere is a summarized list of the questions or concerns extracted from the content:\n\n**Question 1**: The phenomenon of how randomness outperforms determinism is not explained and analyzed in detail  \n**Question 2**: No deeper analysis and comparison with other methods.?  \n**Question 3**: How to evaluate and select the best one for pruning in MCSS? What if all of them are not good enough?  \n**Question 4**: How efficient is your method?  \n**Question 5**: How does ir work in your methods?  \n**Question 6**: Do you consider applying the method on larger models in addition to BERT?  \n\nPlease see our responses below to clarify the main concerns. Also, if you feel that your original concerns have been resolved, we would appreciate it if you would update your evaluation to reflect this. Thank you!\n\n### Question 1: The phenomenon of how randomness outperforms determinism is not explained and analyzed in detail.\n\nWe have previously discussed this topic in our paper, specifically in **`[Section 4.5.1: Impact of Randomness and Schedule]`**. In this section, we delve into the subtleties of incorporating randomness during the pruning stages. However, we appreciate the opportunity to elaborate further.\n\n- **`[Line 388-398]` Redundancy in Early Stages:** The fundamental understanding revolves around the high level of redundancy in the weights during the early pruning stages. Such redundancy suggests that there's significant flexibility in weight selection without adversely impacting accuracy. By introducing randomness, we potentially expand the architecture search space, offering a chance to discover more optimal pruned architectures.\n\n- **`[Line 399-407]` Delicate Balance in Late Stages:** On the other hand, the latter stages of pruning demand a more delicate approach due to the closely matched magnitudes of the remaining weights. Here, the balance becomes pivotal:\n  * Excessive randomness could jeopardize the model's integrity, making post-pruning recovery challenging.\n  * A meager introduction of randomness might restrict the exploration of potential optimal masks, possibly limiting the pruning efficiency.\n\nOur dedicated experiments, visualized in Figure 3, specifically delve into this balance between randomness and determinism. Key observations from these experiments include:\n\n1. **`[Line 429-432]` Thresholds of Randomness:** There's a notable threshold beneath which our randomness-infused strategies eclipse the deterministic method. In parallel, there's another threshold, beyond which the performances of our two randomness schedules seem to converge.\n2. **`[Line 435-440]` Sensitivity Towards Randomness:** The `\\textit{Decrease}` strategy corroborates that the model's accuracy isn't particularly susceptible to randomness during the early pruning phase. However, as pruning continues, this resilience wanes, making the model progressively more sensitive to randomness.\n\nTo wrap up, we've endeavored to offer insights and empirical findings to illustrate how calibrated randomness can surpass deterministic pruning in certain scenarios. Recognizing the reviewer's request for a more profound analysis, we're committed to furnishing a more meticulous examination in the ensuing version of our manuscript, ensuring lucidity.\n\n### Question 2: No deeper analysis and comparison with other methods.\n\nOur primary focus in the study was to understand the effect of randomness relative to pruning, **`especially when benchmarked against our deterministic baseline, IMP`.** We aimed to examine, at a granular level, how randomness interacts with pruning at different stages of the model's life cycle, as elucidated in **`[Section 4.5.1 Impact of Randomness and Schedule]`**.\n\n- **Robustness at Early Stages:** Existing research indicates that models maintain their accuracy even with a high degree of sparsity, especially when it's below 50%. This reinforced our hypothesis about the redundancy in early-stage weights. Therefore, introducing randomness at this juncture potentially aids in widening the architecture search space, possibly leading to more efficient pruned architectures.\n\n- **Tuning Randomness:** The tail-end of pruning is particularly delicate due to the intricate balance required between weight magnitudes. Here, the judicious introduction of randomness becomes crucial: \n  * Too much randomness might jeopardize the model's stability.\n  * Insufficient randomness may curtail our quest for optimal masks.\n\nOur experiments, particularly depicted in Figure 3, accentuate this delicate interplay **`between randomness and deterministic pruning`**. Key takeaways from our findings include:\n\n1. **Performance Dips with Excessive Randomness:** There exists a region where undue randomness makes our strategy lag behind the deterministic approach. Interestingly, in this very region, the **`Decrease`** strategy consistently surpasses the **`Increase`** one.\n2. **Superiority of Random Approach:** We identified a threshold wherein our randomness-driven strategies consistently outperform IMP. This empirically underscores the potential benefits of integrating randomness into pruning.\n3. **Convergence of Performance:** Beyond a certain threshold, the performance dynamics of our two randomness schedules become strikingly similar.\n\nWith all due respect, while we deeply delved into the effect of randomness in pruning against our main baseline **`IMP`**, we admit that an exhaustive comparison with other methods wasn't the mainstay of this study. \n\n### Question 3: How do you evaluate and select the best one for pruning in MCSS? What if all of them are not good enough?\n\nWe evaluate masks using the validation dataset, testing their performance on unseen data. The best mask is selected based on this performance. As a precaution, we also incorporate a deterministic mask among our candidates, ensuring we don't solely depend on random or heuristic approaches. More details are provided below:\n\n1. **Firstly**, as you pointed out, evaluating and selecting the best mask for pruning in MCSS is essential. To ensure robustness in our approach, we adopt a deterministic mask as one of our mask candidates. By doing so, we're not solely relying on random or heuristic methods but also have a reliable fallback.\n\n2. **Secondly**, regarding the selection of the best mask, we conduct evaluations on the validation dataset (validation dataset). Doing this ensures that our model is tested on unseen data, providing an objective measure of the mask's performance. This process of selecting the best mask from multiple candidates allows us to mitigate the risk of all the masks performing sub-optimally.\n\n3. **To directly address your concern**, \"What if all of them are not good enough?\", our deterministic mask acts as a safety net. Moreover, by evaluating multiple masks on the validation dataset, we largely ensure that our selection won't be compromised because all mask candidates perform poorly. This mechanism safeguards the quality of our selection during the pruning phase.\n\n### Question 4: How efficient is your method?  \nIn response to your query about the efficiency of our method, we address this concern from two perspectives: the training and inference phases.\n\n  - **Training Phase**: Concerning the computational overhead of our method\n    * **Randomized Mask Generation:**  \n         **(1)** At first glance, the computation in this phase might seem directly proportional to the number of randomized mask candidates we aim to generate.  \n         **(2)** **`Utilizing Parallel Processing to Expedite Randomized Mask Generation`**: It's crucial to emphasize that the creation of each mask is independent of the others. This independence allows for parallel processing, meaning that the time consumption doesn't increase linearly as one might expect.  \n         **(3)** **`In fact, the computation required to generate one mask candidate is far less than one forward and backward pass in BERT`**: Let's delve into the computational cost of generating a single mask candidate to understand our method's efficiency better.\n         \n        **3.1** Let's first calculate the required computation by mask candidate generation step by step.\n              \n       ```python\n       reshape_weight = weight.cpu().reshape(-1)\n       eshape_weight = reshape_weight ** 5\n       weight_probality = abs(reshape_weight) / sum(abs(reshape_weight))\n\n       masks = torch.zeros_like(reshape_weight)  \n       sample_times = max(int((len(reshape_weight) - keep_k) * self._sample_ratio), 1)\n       for i in range(sample_times):\n            sampling_mask = weight_probality.multinomial(\n                num_samples=keep_k, \n                replacement=False\n            )\n            masks[sampling_mask] += 1\n            index = torch.topk(masks, keep_k)[1]\n       masks[:] = 0\n       masks[index] = 1 \n       mask = masks.reshape(weight.shape).to(weight.device)\n        ```\n     \n      1. **Reshaping and Probability Computation**: This process is `4 x d^2` FLOPs for `d x d` weight matrices and `4 x 4d x d` FLOPs for `4d x d` weight matrices. \n\n      2. **Sampling using `multinomial`**: Given the average sparsity from the schedule and `sample_times`, the computation for the `d x d` matrices is approximately `avg_sparsity x avg_sample_times x d^2` FLOPs. For the `4d x d` matrices, it is `avg_sparsity x avg_sample_times x 4d x d` FLOPs.\n\n      3. **Top-k Operation**: For `d x d` matrices, it's about `d^2 x log2(d^2 x avg_sparsity)` FLOPs. For `4d x d` matrices, it is `4d x d x log2(4d x d x avg_sparsity)` FLOPs.\n\n      4. **Total**: Given BERT base's architecture with 12 blocks, each having 4 `d x d` layers and 2 `4d x d` layers, the total computation for pruning the entire BERT model is **1.34 GFLOPs** (with 1.34 being the computed value).\n\n      **3.2** To provide a more intuitive comparison of the computation required, we calculated both the computational cost for one forward pass and one backward pass of Bert.\n      \n         1. **Multi-Head Attention**:\n            - Q, K, V calculations: $\\(3 \\times d \\times d \\times s\\)$\n            - Scaled dot-product attention: $\\(d \\times s \\times d \\times s\\)$\n            - Output projection: $\\(d \\times d \\times s\\)$\n         2. **Feed Forward Network**:\n            - First layer: $\\(4d \\times d \\times s\\)$\n            - Second layer: $\\(d \\times 4d \\times s\\)$\n         3. **Total**: Given the architecture of BERT base with 12 blocks, the total FLOPs for one forward pass is: $\\[ \\text{FLOPs for one forward pass} = 12 \\times (3d^2s + d^2s^2 + d^2s + 4d^2s + d^2s) \\]$. For the backward pass, it's approximately three times the forward pass, equating to **496.47 GFLOPs** for both forward and backward passes combined.\n\n         **3.3** Comparison: Given the above analysis, single mask candidate generation requires **1.34 GFLOPs** for the entire BERT model, whereas a single forward and backward pass of BERT demands **496.7 GFLOPs**. Importantly, while computationally involved, pruning operations are not executed as frequently as the forward and backward passes during training. Hence, over a training session, the added computational cost due to randomized mask generation remains relatively low.\n     \n    * **Randomized Mask Selection:** \n        * Requires only **one epoch** to identify the optimal mask.\n        * Consequently, the overhead here is minimal in the grand scheme of the entire pruning process.\n\n  - **Inference Phase**: It's essential to note that when assessing sparse neural networks' deployment in real-world scenarios, the computational effort during the pruning phase is a significant factor. Thus, even if there's a computational overhead during this phase, it's often justified by the benefits obtained during inference in production environments. Moreover, our method not only demonstrates superior performance at a 16x compression rate compared to other methods, but it also maintains performance at even higher sparsity levels, achieving up to a 100x compression rate. This means that when our pruned neural networks are deployed in production settings, the gains in performance and efficiency are even more pronounced.\n\n### Question 5: How does ir work in your methods?\n\nWe appreciate the opportunity to clarify the intricacies of our methodology further. \n\n1. **`[Line 226-235]` Clarification on the Role of $\\( ir \\)$**: The equation (Equation 1) quantifies how much our random pruning method diverges from a deterministic approach. Essentially, the $\\( ir \\)$ value gauges the degree of randomness introduced by our random method relative to the deterministic one. A crucial aspect is that a lower $\\( ir \\)$ value signifies a closer resemblance to deterministic masking, which may be desirable in some cases, but our method aims to harness the power of certain randomness levels.\n\n2. **`[Line 238-246]` Handling the Randomness via Ensemble Masks**: While randomness is an integral part of our methodology, it is also important to ensure that it does not spiral out of control. Hence, our ensemble mask approach. By aggregating $\\( M \\)$ masks and setting top-k values to 1, we can curate a mask with controlled randomness. The $\\( M \\)$ value directly impacts the magnitude of randomness.\n\n3. **`[Line408-420]` Hyper-parameter $\\( sr \\)$ and its Function**: The hyper-parameter $\\( sr \\)$ offers control over the total introduced randomness by regulating the number of sampled masks $\\( M \\)$. It is instrumental in tailoring our pruning strategy according to specific requirements, allowing for a dynamic adaptation of the level of randomness.\n\n4. **`[Line 424-440]` Comparison of Randomness Schedules**: Our experiments aimed to highlight the implications of varying the introduced randomness. As shown in Figure 3, the findings underscore the importance of controlling randomness. Interestingly, our results show that there are indeed regions where our random approach outperforms a deterministic one, reaffirming the significance of our methodology.\n\n5. **`[Line1088-1093]` Hyperparameter Settings**: We meticulously chose the $\\( sr \\)$ range based on preliminary experiments to ensure that the randomness remains within the desired bounds. The exponential function parameter $\\( T \\)$ accelerates the reduction of randomness, providing a balance in the method. \n\n6. **In conclusion**, the concept of $\\( ir \\)$ is central to our methodology, as it measures the introduced randomness, but our framework employs multiple strategies to manage and control this randomness for improved pruning performance. We believe that by blending deterministic principles with controlled randomness, we achieve a balanced approach that capitalizes on the strengths of both strategies.\n\n### Question 6: Do you consider applying the method on larger models in addition to BERT?\n\nCertainly, the idea of scaling our method to larger language models, such as those at the billion parameter level, is intriguing. However, there are challenges to consider. **The current state of pruning for models of this magnitude is still a conundrum**. SparseGPT, for instance, is a recent method introduced for pruning at the billion-parameter level. It's crucial to note that SparseGPT is a post-training pruning method, requiring no further training. Therefore, its effectiveness is mainly evident at a 2x compression level. While this provides a significant benchmark, our approach might not directly apply. Nonetheless, we are actively exploring the feasibility of introducing randomness on the foundation of this method.\n\n**Finally, we thank the valuable suggestions from the reviewer again.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method proposes a randomized pruning mask generation strategy that can introduce controllable randomness in a principled way.\n2. The method achieves consistent improvement on benchmark compared with other state-of-art pruning methods.", "reasons_to_reject": "1. The phenomenon of how randomness outperforms determinism is not explained and analyzed in detail.\n2. No deeper analysis and comparison about the proposed method and other methods. ", "questions_to_authors": "1. How to evaluate and select the best one for pruning in MCSS? What if all of them are not good enough? Moreover, as more candidates lead to more complexity, I wonder the training cost of the proposed method as it is not reported and compared with other methods in the paper.\n2. How does ir work in your methods?\n3. Do you consider appling the method on larger models in addition to BERT?\n", "paper_topic_and_main_contributions": "This paper proposes a model pruning strategy that first generates several pruning masks in a designed random way. Along with an effective mask selection rule and an early mask evaluation strategy, the approach achieves state-of-the-art performance across eight datasets from GLUE.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a randomized pruning approach that is adapted from iterative magnitude pruning (IMP), where multiple masks are generated using weight magnitude as part of random selection and then independently evaluated for one epoch. The mask with the best validation score after that epoch is then selected.

Generally, reviewers liked the idea and agree that it improves performance when compared against a commonly used approach (IMP) across many evaluated settings. The method is also quite clear and the addition of pseudo-code for the approach makes it easy to implement.

One point raised by reviewers was the training overhead introduced by this mask selection process. In the rebuttal, the authors provided evidence (via code and floating-point op estimates) to demonstrate that these masks can be generated relatively efficiently (significantly less than one forward/backward pass of the evaluated model. However, the authors do not provide an estimate of the actual training cost of the approach (not just the mask generation). As the paper says that each mask is trained for one epoch (i.e., many forward and backward passes), it seems that this may lead to significant overhead depending on the number of masks per layer, number of layers, and number of pruning stages. A detailed analytical model of this cost and/or training time comparison of this approach compared to IMP would make it more clear how important of an issue this is.
Decision: Accept-Findings