Reweighted Proximal Pruning for Large-Scale Language Representation
Authors: Fu-Ming Guo
Sijia Liu
Finlay S. Mungall
Xue Lin
Yanzhi Wang
Venue:  (2020)
Keywords: Language Representation
Machine Learning
Deep Learning
Optimizer
Statistical Learning
Model Compression

Abstract: Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pre-training task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).

Under review as a conference paper at ICLR 2020
REWEIGHTED PROXIMAL PRUNING FOR LARGE-SCALE LANGUAGE REPRESENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, pre-trained language representation flourishes as the mainstay of the nat- ural language understanding community, e-g., BERT. These pre-trained language representations can create state-of tf-----rttt--tttt results on wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task trans- fer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for large-ccale language rep- resentation model. Through experiments on SQuAD and the GLUE benchmark suite, show that proximal pruned BERT keeps high accuracy for both the pre- training task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides new perspective to help us analyze what large-scale language rep- resentation might learn. Additionally, RPP makes it possible to deploy large state-of-the language representation model such BERT on a series of dis- tinct devices (c.g., online servers, mobile phones, and edge devices).
INTRODUCTION
Pre-trained language representations such as GPT (Radford et al. 2018 BERT (Devlin et al. 2019) and XLNet (Yang et al. 2019). have shown substantial performance improvements using self-supervised training on large-scale corpora (Dai & Le 2013 Peters et al. 2018 Radford et al,, 2018 Liu al.) 2019a). More interestingly, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-arr models for a wide range of tasks, such as question answering (Rajpurkar et al., 2016 2018), and language inference (Bowman et al.) 2015 Williams et al.) (017). without substantial task-specific architecture modifications. BERT is concep- tually simple and empirically powerful (Devlin et al.) 2019).
However, along with the significant performance enhancement, the parameter volume and com- plexity of these pre-trained language representations significantly increase. As result, it becomes difficult to deploy these large-scale language representations into real-life computation constrained devices including mobile phones and edge devices. Throughout this paper, we attempt answer the following questions.
Question 1: Is possible to compress large-scale language representations such as BERT via weight pruning?
How would the weight- prughh,pruued,, pre- pre-trained model affect the performance of the down- stream multi-task transfer learning objectives?
Question 2:
The problem of weight pruning has been studied under many types of deep neural networks (DNNs)
(Goodfellow et 2016), such as AlexNet (Krizhevsky et al., (012). VGG (Simonyan & Zisserman, 2014), ResNet (He et al. 2016), and MobileNet (Howard et al. [017]. is shown that weight prun- ing can result in notable reduction in the model size. A suite of weight pruning techniques have been developed, such as non- con-struct weight pruning (Han et al. 2013). structured weight pruning (Wen al. 2016). filter pruning (Li et al.) 2016), channel pruning (He et al.) 2017). ADMM-NN (Ren et al. 2019 and PCONV (Ma et al. 2019) to name few. Different from pruning CNN- type models, BERT not only considers the metrics the pre-training task, but also needs to make

---

Under review as a conference paper at ICLR 2020
allowance for the downstream multi-task transfer learning objectives. Thus, the desired weight prun- ing needs to preserve the capacity of transfer learning from a sparse pre-trained model to downstream fine-tuning tasks.
In this work, we investigate irregular weight pruning techniques on the BERT model, including the iterative pruning method (Han al. 2013) and one-shot pruning method (Liu et al.) 2019b). How- ever, these methods fail to converge to a sparse pre-trained model without incurring significant ac- curacy drop, in many cases do not converge at all (see supporting results in Appendix). Note that the aforementioned weight pruning techniques are built on different pparity-promoting regulariza- tion schemes (Han al. 2015 Wen et al. 2016). e.g., lasso regression (f1 regularization) and ridge regression (l2 regularization), We find that the failure of previous methods on weight pruning of BERT possibly due to the inaccurate sparse pattern learnt from the simple & or la based sparsity- promoting regularizer. In fact, the difficulty of applying regularization to generate weight sparsity coineides with the observation in (Loshchilov & Hutter 2018) on the imcompatibility of conven- tional weight decay (l2 regularization) for training super-deep DNNs as BERT. It is pointed out that the main reason that the direct optimization of a regularization penalty term causes divergence from the original loss function and has negative effect the effectiveness of gradient- based update. To mitigate this limitation, (Loshchilov & Hutter] (018) have modified the regularization in Adam by decoupling weight decay regularization from the gradient-based update, and have achieved state- of-the-art results on large-scale language pre-training and downstream multi-task transfer learning objectives (Devlin et 2019).
MHF MNL
CoLA, SQWAD
Mask Mask
Start/End
Identical
Universal Sparsity
NERT
Sparse BERT
Sparse BERT
BAB
KdM
Sentence
Masked
Masked Sentence
Masked Sentence
Maseed Sentence
Peragraph
Unlabeled Sentence and
Unlabeled Sentence A
Question Answer
Pre-training
Reweighted Proximal Pruning
Fine- Fine-Tuning
Overview of pruning BERT using Reweighted Proximal Pruning algorithm and then fine-tuning on a wide range downstream transfer learning tasks. Through RPP, we find identical universal sparsity Sw. The BERT model pruned with RPP could be fine-tuned over the downstream transfer learning tasks.
Figure 1:
In this work, we aim at more accurate universal sparse pattern search (see Figure for an overview of our approach) motivated by our experiments and the conclusion from Loshchilov & Hutter (2018). We propose Reweighted Proximal Pruning (RPP), which integrates reweighted ε minimization (Candes et al. 2008) with proximal algorithm (Parikh et al. 2014). RPP consists of two parts the reweighted & minimization and the proximal operator. Reweighted & minimization serves as a bet- ter method of generating sparsity in DNN models matching the nature of weight pruning, compared with & regularization. Thanks to the closed-form solution of proximal operation on a weighted l1 norm, in RPP the sparsity pattern search can be decoupled from computing the gradient of the training loss. In this way the aforementioned pitfall in prior weight pruning technique on BERT can be avoided. We show that RPP achieves effective weight pruning BERT for the first time to the best of our knowledge. Experimental results demonstrate that the proximal pruned BERT model keeps high accuracy on wide range of downstream tasks, including SQuAD (Rajpurkar al. 2016 2018) and GLUE (Wang et al. 2018).
We summarize our contributions follows.
We develop the pruming algorithm Reweighted Proximal Pruning (RPP), which acheives the first effective weight pruning result large pre-trained language representation model BERT. RPP achieves 59.3% weight sparsity without inducing the performance loss on both pre-training and fine-tuning tasks.
2

---

Under review as a conference paper at ICLR 2020
We spotlight the relationship between the pruning ratio of the pre-trained DNN model and the performance on the downstream multi-task transfer learning objectives. We show that many downstream tasks except for SQuAD allows at least 80% pruning ratio compared with 59.3% under the more challenging task SQuAD.
We observe that as the pruning ratio of the pre-trained language model increases, the perfor- mance on the downstream transfer learning tasks decreases. The descending range varies in different downstream transfer learning tasks. However, the proposed RPP approach is able to achieve consistently high pruning ratio compared to iterative pruning based methods.
We show that different from weight pruning in image classification tasks, RPP helps to find the structured sparsity pattern in transformer blocks used in BERT. Moreover, peer into the effect of network pruning on the language representation embedded in BERT.
2 RELATED WORK
BERT and prior work model compression BERT (Devlin et al.] 2019) is self-supervised approach for pre-training a deep transformer encoder (Vaswani et al.) 2017]. before fine-tuning it for particular downstream tasks. Pre-training of BERT optimizes two training objectives masked language modeling (MLM) and next sentence prediction (NSP) - which require a large collection of unlabeled text. We use BooksCorpus (800M words) (Zhu et al., 2015) and the English instance of Wikipedia (2,500M words) as the pre-training corpus, the same as Devlin et al. (2019). For detailed information about the BERT model, readers can refer to the original paper (Devlin et al. 2019).
Michel et al. 2019) mask some heads in multi-head attention modules in BERT, and then evaluate the performance on the machine translation task. Similarly, Hao et al. (019) eliminates certain heads in the multi-head attention module. First, the limited previous work do not consider the pre- training metrics and the other downstream multi-mask transfer learning objectives. They only con- sidered the specific machine translation task (out of over 10 transfer tasks), which is only a specific fine-tuning and is limited for the universal pre-trained language representation (BERT). Second, the multi-head attention module uses weight sharing mechanism (Vaswani al. 2017). So masking some heads does not reduce the weight volume. Finally, multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions, while single attention head inhibits this effect (Vaswani et al. 2017). As a result, masking some heads in multi-head attention harms the weight sharing mechamism, without weight volume reduction. In summary. the limited previous work in this area are not effective weight pruning method on BERT. Shen et al. (2019) reports the quantization result of BERT model, which is orthogonal our work and can be combined for further compression/acceleaatioo
Reweighted & and proximal algorithm Candes et al. (2008) present reweighted & algorithm and demonstrate the remarkable performance and broad applicability in the areas of statistical esti- mation, error correction and image processing. Proximal algorithms can be viewed as an analogous tool for non non-smooth, constrained, large-scale, or distributed versions these problems (Parikh et al.) 2014). To the best of our knowledge, ours is the first work that applies reweighted & minimization to network compression, particularly for BERT pruning.
3 REWEIGHTED PROXIMAL PRUNING FOR LARGE-SCALE LANGUAGE REPRESENTATION DURING PRE-TRAINING
Pruning for pre-trained language representations should not only consider the performance of pre- training objectives, but also make allowance for the downstream fine-tuning transfer learning tasks. Let fa denote the loss function of network for downstream task T/ ~ p(T), where p(T) denotes the distribution of tasks. Let denote the parameters of the pre-trained model (pre-training in BERT), and z; denote the i-th task-specified model parameters (fine-tuning in BERT). The downstream tasks have separate fine-tuned models, even though they are initialized with the same tre-trained parameters (Devlin et al.) 2019). Starting from the pre-trained parameters w, the parameters 2, (w) are obtained through tine-tuning
minimize fi(w) w we2d
(1)
3

---

Under review as a conference paper at ICLR 2020
3.1 PRUNING FORMULATION IN TRANSFER LEARNING
Following the conventional weight pruning formulation, we first consider the problem of weight pruning during pre-training:
minimize fo(w) + +///w//>
we2d
where fo is the loss function of pruning, E {0, 1] denotes the type of regularization norm, and y is regularization term. We note that the pparsity-promoting regularizer in the objective could also be replaced with a hard & constraint, |wlpp I T for some T.
Let w denote the solution to problem
2 and the corresponding sparse pattern Sw is given by S = ( 0, vi [d]
(3)
For a specific transfer task i, we allow an additional retrainingffne-tuningg step to train/fine-tune weights starting from the pre-training results w and subject to the determined, fixed sparse pattern Sw. denoted as Z,W S.). That is, we solve the modified problem equation
minimize fi (W; Sw))
Here, different from the task-specific fine tuning weights variable z, Sw) is now defined over Sa.
Our goal is to seek sparse (weight pruned) model during pre-training, with weight collection w and sparsity Sw: which can perform as well as the original pre-trained model over multiple new tasks (indexed by i). These fine-tuned models 2;W Sw) (for different i) share the identical universal sparsity Sw.
3.2 REWEIGHTED PROXIMAL PRUNING
In order enhance the performance of pruning pre-trained language representation over multi-task downstream transfer learning objectives, we propose Reweighted Proximal Pruning (RPP). RPP consists of two parts: the reweighted & minimization and the proximal operator. Reweighted L minimization serves as better method of generating sparsity in DNN models matching the natural objective of weight pruning, compared with & regularization. The proximal algorithm then sep- arates the computation of gradient with the proximal operation over weighted & norm, without directly optimizing the entire sparsity-penalizer loss, which requires gradient backpropagation of the involved loss. This is necessary in the weight pruning of super-deep language representation models.
3.2.1 REWEIGHTED & MINIMIZATION
In the previous pruning methods (Han et al. 2015 Wen al. 2016), & regularization is used to generate sparsity. However, consider that two weights w;, Wy (w) < wyy) in the DNN model are penalized through ε regularization. The larger weight wj is penalized more heavily than smaller weight w; in é regularization, which violates the original intention of weight pruning, "removing the unimportant connections" (parameters close zero) (Han et al. 2015). To address this imbal- ance, we introduce reweighted & minimization (Candes et al. 2008) to the DNN pruning domain. Our introduced reweighted E minimization operates process shown in Algorithm [ and the first iteration of reweighted ε minimization is & regular- ization. This designed mechanism helps us to observe the performance difference between & and reweighted i minimization. Meanwhile, this mechanism ensures the advancement of reweighted L minimization over & regularization, as the latter is the single, first step of the former.
systematic and iterative manner (detailed
Consider the regularized weight pruning problem (reweighted & minimization):
minimize w
fo + Σ as. [ws]
(S)
where a; (a; > 0) factor is a positive value. It is utilized for balancing the penalty, and different from weight w; in DNN model. a, factors will be updated in the iterative reweighted & minimiza- tion procedure (Step in Algorithm|] [ in a systematic way (Candes et al.] 2008). If we set T for reweighted &,, then it reduces to sparse training.
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 RPP procedure for reweighted é minimization
Input: Initial pre- pre-trained model wº, initial reweighted & minimization ratio 7, initial positive
value ax" =
2: for = .... do
(w((-1) w= a (t-1)
3: w
4: Step 1: Solve problem (5) to obtain a solution w via iterative proximal algorithm 6
5: Step 2: Update reweighted factors af [w
(the inside wf denotes the weight w; in iteration t, and the outside (t) denotes the exponent), is small constant, c.g., e = 0.001
6: end for
3.2.2 PROXIMAL METHOD
In the previous pruning methods (Han et al. 2015 Wen al., 2016), & regularization loss is di- rectly optimized through the pack-propngttion based gradient update of DNN models, and the hard- threshold is adopted execute the pruning action the step of pruning (all weights below the hard-threshold become zero). In our approach, we derive an effective solution to problem (5) for given {a,), namely, in Step of Algorithm 2, in which back-propagation based gradient update is only applied on fo(w) but not 7 >; lwsl-
We adopt the proximal algorithm (Parikh al. 2014) satisfy this requirement through decou- pling methodology. In this way, the sparsity pattern search can be decoupled from pack-propagation based gradient update of the training loss. The proximal algorithm is shown in (Parikh et al.) 2014) to be highly effective (compared with the original solution) on a wide set of non-convex opumiia- tion problems. Additionally, our presented reweighted & minimization has analytical solution through the proximal operator.
To solve problem (5) for given a, the proximal algorithm operates in an iterative manner:
Wk prox x (Wk-1 Ae V fo (wa-1))
where the subscript & denotes the time step of the training process inside each iteration of RPP, A (Ax 0) is the learning rate, and we set the initial to be w-(t-1) from the last iteration of reweighted f1. The proximal operator prox (a) is the solution to the problem
minimize y Σos lw
1
l|w all
2
where a W V (W:-1). The above problem has the following analytical solution (Liu et al. 2014)
We.k { Ta
(1
[ad| > Aayyass
la;
(8)
las| < A
We remark that the updating nule (6) can be interpreted the proximal step X over the gradient descent step Wk-1 >x V w (W:-1). Such a descent can also be obtained through optimizers such AdamW: We use the AdamW (Loshchilov Hutter (0188) as our optimizer, the same with (Dellnn et al. 2019). The concrete process of Adam W with proximal operator is shown in Algorithm
of Appendix C.
Why chooses Adam rather than Adam? Loshchilov & Hutter (0018) proposes Adam W to improve the generalization ability of Adam (Kingma & Bal 2014). Loshchilov & Hutter (2018) shows that the conventional weight decay is inherently not effective in Adam and has negative effect on the effectiveness of gradient-based update, which is the reason of the difficulty to apply adaptive gradi- ent algorithms to super-deep DNN training for NLU applications (like BERT).(Loschin (2018) mitigates this limitation and improves regularization of Adam, by decoupling weight de- cay regularization from the gradient-based update (Loshchilov & Hutter 2018). AdamW is widely adopted in pre- gre-trinning large language representations, e.g., BERT (Devlin et al. 2019), GPT (Rad-] ford et al., 2018) and XLNet (Yang al.) 2219)). Our proposed RPP also benefits from the decou- pling design ideology. The difference is that RPP is for the generation of sparsity, instead of avoiding over-fitting, like decoupled weight decay in AdamW.
& Hutter
5

---

Under review as a conference paper at ICLR 2020
Our new and working baseline: New Iterative Pruning (NIP). To get the identical universal sparsity Sw. we tried a series of pruning techniques. including the iterative pruning method (Han et al. 2015) and one-shot pruning method (Liu et al. 2019b). But these methods do not converge to a viable solution. The possible reason for non-convergence of the iterative pruning method is that di- rectly optimizing Ep (p {1,2)) pparsityprromotie
regularization makes the gradient computation involved and thus harms the loss convergence (We provide the loss curve and analysis in Appendix D). To circumvent the convergence issue of conventional iterative pruning methods, we propose new iterative pruning (NIP) method. Different from iterative pruning (Han et al., 2015). NIP reflects the naturally progressive pruning performance without any externlly introduced penalty. We hope that other pruning methods should not perform worse than NIP, otherwise, the effect of optimizing the newly introduced ppasity-promotiig regularization is negative. We will show that NIP is able to successfully prune BERT to certain pruning ratios. We refer readers to Appendix A for the full detail about NIP, our proposed baseline algorithm.
4 EXPERIMENTS
In this section, we describe the experiments on pruning pre-trained BERT and demonstrate the per- formance on 10 downstream transfer learning tasks.
4.1 EXPERIMENT SETUP
We use the official BERT model from Google as the startpoint. Following the notation from Devlin et al. (2019). we denote the number of layers (i.e., transformer blocks) as L, the hidden size H, and the number of self-attention heads as A. We prune two kinds of BERT model: BERTRASS RASE (L 110M) and BERT (L 24, H 1024, 16, total parameters = 340M). As the parameters of these transformer blocks take up more than 97% weights of the entire BERT, the weights of these transformer blocks are our pruning target.
12, H 768, A 12, total parameters
Data: In pre-training, we use the same pre-training corpora as Devlin et al. 2019): BookCorpus (800M words) (Zhu al. 2015) and English Wikipedia (2, 500M words). Based on the same cor- pora, we use the same preprocessing scrip to create the pre-training data. In fine-tuning, we report our results on the Stanford Question Answering Dataset (SQuAD) and the General Language Under- standing Evaluation (GLUE) benchmark (Wang et al. 2018). We use two versions of SQuAD: V1.1 and V2.0 (Rajpurkar et al.) 2016 (0088) The GLUE is collection of datasets/tasks for evaluating natural language understanding systems
Input/Output representations: We follow the input/output representation setting from Devlin et al. 2019) for both pre-training and fine-tuning, We use the WordPiece (Wu et al. 2016) embeddings with a 30, 000 token vocabulary. The first token of every sentence is always a special classification token (ICLS). The sentences are differentiated with special token ([SEP]).
Evaluation: In pre-training, BERT considers two objectives: masked language modeling (MLM) and next sentence prediction (NSP). For MIM, random sample of the tokens in the input sequence is selected and replaced with the special token ( (MASK)). The ML. objective is a cross-entropy loss on predicting the masked tokens. NSP is binary classification loss for predicting whether two segments follow each other in the original text. In pre-training, we use MLM and NSP as training objectives pre-train, retrain the BERT model, and metrics to evaluate the BERT model In fine- tuning, Fl scores are reported for SQuAD, QQP and MRPC. Matthew's 's Corr and Peerson-Speaamee Corr are reported for CoLA and SST2 respectively. Accuracy scores are reported for the other tasks.
All the experiments execute on one Google Cloud TPU V3-512 cluster, three Google Cloud TPU V2-512 clusters and 1 110 Google Cloud TPU V3-8/V2-8 instances.
httpss//gihubb.oom/gogggg comm wog.ggole-eeeecchbbeerr
"The datasets/tasks are: CoLA Warstadt 2018
Stanford Sentiment Treebank (SST) (Socher et al.
2013). j. Microsoft Research Paragraph Corpus (MRPC) (Dolan & Brockett (005). Semantic Texual Similarity
Benchmark (STS) (Agire & Soroa 2007). Quora Question Pairs (QQP), Multi-Genre NLI (MNLI) (Williams)
al. 2017). Question NI. (QNL.)) Rajpurkar et al. 2016), Recognizing Textual Entailment (RTE) and Wino-
grad NLI(WNL.) (Levesque al. 2012

---

Under review as a conference paper at ICLR 2020
Baseline: As there is public effective BERT pruning method, we use the proposed NIP pruning method on BERT as the baseline method. Th detail of NIP is shown in Appendix The progressive pruning ratio is Vp 10% (prune 10% more weights in each iteration). Starting from the official BERTBASE BASE, we use iterations. In each iteration of NIP, we get the sparse BERT BERTBASE BASE with specific spansity,
(w"; Sw:). Then we retrain the sparse BERT BASE w' over the sparsity Swi. In the retraining process, the initial learning rate is 2-10 the batch size is 1024 and the retraining lasts for 10, 000 steps (around 16 epochs). For the other hyperparameters. we follow the original BERT paper Devlin et al. (2019). In each iteration, the well retrained sparse BERTBASE is the starting point for the fine-tuning tasks and the next iteration.
4.2 REWEIGHED PROXIMAL PRUNING (RPP)
We apply the proposed Reweighted Proximal Pruning (RPP) method on both BERT BASE and BERTLARGEE and demonstrate performance improvement. Detailed process of RPP is in Ap- pendix
For BERT BASE, we use the hyperparameters exactly the same with our experiments using NIP. The initial learning rate = and the batch size is 1024. We iterate the RPP for six times (T=6), and each iteration lasts for 100, 000 steps (around 16 epochs). The total number of epochs in RPP is smaller than NIP when achieving 90% sparsity (96 < 144). There is no retraining process in RPP. We set y E {10-² 10 10-4, 10-5) and c=10- 9 in Algorithm Recall that RPP reduces to L sparse training as t=1.
In Figure 2 present the accuracy versus the pruning ratio for pre-training objectives MLM and NSP, and fine-tuning task SQuAD 1.1. Here we compare RPP with NIP. Along with the RPP contin- uing iterate, the performance of RPP becomes notably higher than NIP for both the pre-training task and the fine-tuning task. The gap further increases as the RPP iterates more times. In Figure find that the NSP accuracy is very robust pruning. Even when 90% of the weights are pruned, the NSP accuracy keeps above 95% in RPP algorithm and around 90% in NIP algorithm. For MLM accuracy and SQuAD F1 score, the performance drops quickly as the prune ratio increases. RPP slows down the decline trend to great extent. On SQuAD 1.1 dataset/task, RPP keeps the F1 score of BERT BASE at 88.5 (0 degradation compared with original BERT) at 41.2% prune ratio, while the F1 score of BERTBASE applied with NIP drops to 84.6 (3.9 degradation) at 40% prune ratio. At 80% prune ratio, RPP keeps the Fl score of BERT BASE at 84.7 (3.8 degradation), while the score of BERT'BASE applied with NIP drops to 68.8 (19.7 degradation compared with the original BERT). addition to the fine-tuning task SQuAD 1.1, the other transfer learning tasks show the same trend (RPP consistently outperforms NIP) and the detailed results reported in Appendix C
For BERTLARGE. we use the hyperparameters exactly the same with our experiments using NIP except for the batch size. The initial learning rate is 2- 10-5 and the batch size is 512. We iterate the RPP for four times (7=6), and each iteration lasts for 100,000 steps (around 8 epochs). There is
retraining process either. We 7 (10-2 2 10 10-4 10- 10- and 10-9 in Algorithm The
experimental results about pruning BERTLARGE and then fine-tuning are shown in Table
00
Acourssy (6) F1 scora SQUAD
80
VEMTNIIS
REF hars
EQWAST JIP
MLM IRPE
NSP IREP
BOWAD1 (APF)
80
0.00
0.10
0.20
0.50
0.48
0.50
0.50
0.70
0.50
0.00
Prune Ratio (%)
Figure Evaluate the performance of pruned BERT BaSe using NIP RPP, respectively (MLM and NSP accuracy on pre-training data and Fl score of fine- tun-tt on SQuAD 1.1 are reported).
7

---

Under review as a conference paper at ICLR 2020
Table 1: BERTTLACC pruning results of transfer learning tasks. The degradation is contrasted with the original BERT (without pruning) transfer learnin..
Method
Prune Ratio(%)
SQuAD 1.1
QQP
MNLI
MRPC
CoLA
NIP
50.0
85.3 (-5.6)
85.1 (-6.1)
77.0(-99.11
(3.55)
76.3 (-5.2) 69.13 (-12.37)
80.0
75.1 (-15.8)
81.1 (-10.1)
73.81 (-12.29)
68.4 (-20.5)
RPP
59.3
90.23 (-0.67)
(-0.0)
86.1 (-0.0)
88.1 (-1.2) 81.9 (-7.1)
82.8 (+1.3)
88.4
81.69 (-9.21)
89.2 (-2.0)
81.4 (-4.))
79.3 (-2.2)
Method
Prune Ratio(%)
SQuAD 2.0
QNLI
MNLIM
SST-2
RTE
NIP
50.0
75.3 (-6.6)
90.2 (-1.1)
82.5 (-3.4)
91.33 (-1.9)
68.6 (-1.5) 62.8 (-7.3)
80.0
70.1 (-11.8)
80.5 (-10.8)
78,4
88.7 (-4.5)
RPP
59.3
81.3 (-0.6)
(+1.0) 88.0 (-3.3)
85.7 (-0.2)
92.4 (-0.8)
70.1 (-0.0)
88.4
80.7 (-1.2)
81.8 (-4.1)
90.5 (-2.7)
67.5 (-2.)
4.3
VISUALIZING ATTENTION PATTERN IN BERT
We visualize the sparse pattern of the kernel weights in sparse BERT model applied with RPP, and present several examples in Figure 3 Since we directly visualize the value of identical universal sparsity Sw without any auxiliary function like activation map, the attention pattern universal and data independent.
Layer 2 Query matrix
Layer 3 Query matrix
Layer 11 Query matrix
Layer 2 Key matrix
Layer Key matrix
Layer 11 Key matrix
Figure Visualization sparse pattern S in pruned BERTBASE model w. We sample matrices query matrices at the top row and key matrices at the bottom row) from layer 2, layer and layer in the sparest pruned BERTBASE BASE-
BERT's model architecture a multi-layer, bidirectional transformer encoder based on the original implementation (Vaswani et al.) 2017). Following (Vaswani et al. (017), the transformer archi- tecture is based on "scaled dot-product attention." " The input consists queries, keys and values, denoted as matrices Q, K and V, respectively. The output of attention model is computed as
QKT ViT
Attention (Q, K.V) softmax
(9)
where de is the dimension. We visualize the sparse matrices Q and K of layer 2, layer and layer 11 respectively in Figure From Figure have the following observations and analyses.
8

---

Under review as a conference paper at ICLR 2020
Structured pattern: Figure demonstrates the structured pattern of non-zero weights in a pruned transformer block. More specifically, we found that the pruned Q and K matrices within each transformer yield interesting group-wise structures (column-wise non-sparsity for query matrix and row-wise non-sparsity for key matrix). Interestingly, we obtained these structured sparse patterns from our proposed RPP, irregular pruning method (namely, no group-wise sparsity is penalized). This is different from the irregular pruning on image classifiers, and thus shows the specialty of pruning on language models. We also believe that the use of reweighted & approach matters to find these fine ine-grained sparse patterns. Note that the structured sparsity pattern is more friendly to hardware implementation and acceleration than the non-structured pattern.
Semantic interpretation:
The structured pattern found by RPP (visualized in Figure 3 has the following semantic interpretation. What might the large-scale language representation learn? The answer becomes clear after the language representation is pruned by RPP. From the perspective of attention mechanism, the query matrix Q (column- information inside each sequence, while the key matrix K (row-wise non-sparsity) mainly models the attention information between different sequences in the context.
non-sparsity) mainly models the attention
4.4 t-SNE VISUALIZATION
t-Distributed Stochastic Neighbor Embedding (I-SNE) is technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets (Maaten Hinton 2008 Pre-trained word embeddings
an integral part of modern NLP systems (Devlin et al., 2019) and one contribution of BERT is the pre-trained contextual embedding. Hence, we visualize word embedding in the original BERT model and the BERT model applied with RPP in Figure 4 using t-SNE. Since BERT is different from commonly would like to examine if pruning BERT will lead to significant change on the low-dimensional manifold of the language representation. From Figure we obtain the following observations and insights.
image classifier in network pruning,
Low-dimensional manifold: Figure illustrates that, for both original BERT and BERT pruned with RPP, the low-dimensional manifolds of the language representations are similar, showing the similar projection. Taking the specific word "intelligent" in Figure as an example, the distribution of specific words and corresponding nearest words at the low-dimensional manifold (calculated using cosine/Euclidean distance) remain the high degree of similarity. This implies that the BERT applied with RPP keeps most of the language representation information similar to that from the original BERT.
Linguistic interpretation of proper noun: There is one salient ribbon on the upper left of the macroscopical t-SNE visualization of word embeddings in either the original BERT or the pruned BERT through RPP. Each point in the ribbon represents a year number in annals. There is also one salient short line on the lower left of the macroscopical t-SNE visualization of word embeddings in either the original BERT or the BERT applied with RPP. Each point in most of the lines represents an age number. Other proper nouns also reveal similar characteristics. Our proposed RPP remains the embedding information of these proper nouns from the perspective of linguistic interpretation.
nbbon
intelligent
intel
salient line
salient
Figure 4: 1-SNE visualization of word embeddings in the original BERT model and the pruned BERT model using RPP. From left to right: t-SNE of original BERT embedding, together with an enlarging region around word "intelligent"; t-S of embedding pruned BERT, together with enlarging region. These visualiza- tions are obtained by running t-SNE for 1000 steps with perplexity=100.

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS AND FUTURE WORK
This paper presents the pruning algorithm RPP, which achieves the first effective weight pruning result on large pre-trained language representation model BERT. RPP achieves 59.3% weight spar- sity without inducing the performance loss on both pre-training and fine-tuning tasks. We spotlight the relationship between the pruning ratio of the pre-trained DNN model and the performance on the downstream multi-task transfer learning objectives. We show that many downstream tasks except SQuAD allows at least 80% pruning ratio compared with 59.3% under task SQuAD. Our proposed Reweighted Proximal Pruning provides a new perspective to analyze what does large language representation (BERT) learn.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eneko Agirre and Aitor Soroa. Semeval-2007 task 02: Evaluating word sense induction and dis- crimination systems. Proceedings of the 4th International Workshop on Semantic Evaluations, pp. 7-12. Association for Computational Linguistics, 2007.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv: 2015.
Emmanuel J Candes, Michael B Wakin, and Stephen Boyd. Enhancing sparsity by reweighted minimization. Journal of Fourier analysis and applications, 14(5-6):877-905, 2008.
Andrew M Dai and Quoc Le. Semi-supervised sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-3087, 2015.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 -116. 2019.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1 1135-1143, 2015.
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. Multi-granularity self- attention for neural machine translation. arXiv preprint arXiv: 19999992222.... 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X4xv:700.0488000 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 0rXiv:1412.69880 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- teenth International Conference the Principles of Knowledge Representation and Reasoning, 2012.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1rXiv:1608.08771 2016.
Sijia Liu, Engin Masazade, Makan Fardad, and Pramod K Varshney. Sparsity field estimation via ordinary kriging. In 2014 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 3948-3995 IEEE, 2014.

---

Under review as a conference paper at ICLR 2020
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXhv:1907.11692, 2019a.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2018.
Xiaolong Ma, Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices. arXiv preprint 1xXiv:190000007... 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2665, 2008.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? preprint arXiv:1905. 10650, 2019.
arXiv
Neal Parikh, Stephen Boyd, et Proximal algorithms. Foundations and Trends in Optimization, 1(3):127-239, 2014.
Matthew H Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1FXXv:8802.05663 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language derstanding by generative pre-training. URI. mttp:://33-s--wstts amazonaws. com/openai- hate://eerrrleeeerrrgeggguureree
understanding paper. pdf. 2018.
Pranav Rajpurkar, Jian Zhang, Konstamtin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1706002600.05250,, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad, arXiv preprint arXiv:18066 03822, 2018.
Ao Ren, Tianyun Zhang, Shaokai Ye, Jiayu Li, Wenyao Xu, Xuchai Qian, Xue Lin, and Yanzhi Wang. Admm-nn: An seborithm-harddwaan co-design framework of dans using alternating direc- tion methods of multipliers. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 925-938. ACM, 2019.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 100.0010. 2018
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1001.80054141. /247/ 2018.
12

---

Under review as a conference paper at ICLR 2020
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082 2016.
Adina Williams, Nikita Nangia, and Samuel R Bowman. broad-coverage challenge corpus for 2017.
sentence understanding through inference. arXiv preprini
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint IrXiv: 00009.08144. 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13

---

Under review as a conference paper at ICLR 2020
A ALGORITHM OF NEW ITERATIVE PRUNING
Algorithm 2
shows the detail process of our proposed NIP algorithm.
Algorithm New Iterative Pruning (NIP) algorithm
1: Input: Initial model weights w, initial prune ratio p = 0%, progressive prune ratio Vp
2: for = 1,2,..... II' do
3: W == (w1
4: Sample batch of data from the pre-training data
5: Obtain sparsity Sw through hard threshold pruning, prune ratio p' = t Vp
6: Retrain
sparsity constraint Sw
7: for all tasks in {7:} do
8: Fine-tune z; (w; Sw) over sparsity Sw (if the desired prune ratio p' has been reached for
downstream task i)
9: end for
10: end for
B ALGORITHM OF REWEIGHTED PROXIMAL PRUNING (RPP)
Algorithm shows the detail process our enhanced AdamW (Loshchilov & Hutter 2018) with
proximal operator.
Algorithm 3 Our enhanced AdamW (Loshchilov & Hutter 2018) with proximal operator
1: Given 0.001, 0.9, B2 0.999, e 10-6 X E R
2:
Initialize time step lt <- 0, parameters pre-trained model w, first moment vector m,=0 <- 0, second moment vector vr=0 schedule multiplier Tı==0 R
3: repeat
4: k- k +1
5:
Vfx (Wk-1) - SelectBatch (Wa-1)
6:
gk Vfx (w
7:
my + 8im= - + +1 B1) gk
8:
Ok + B20 -1 + (1 (22 gt.
9:
m + m (1 g8)
10:
O + /k/ 10/(111111 (1- 35)
aK -- SelScchehueeWuutiplleeetk
a W 7/4 (ar/// (Vo bb + c) Awa-1)
- - 14: until stopping criterion is met
W& prox
return optimized sparse model in pre-training
14

---

Under review as a conference paper at ICLR 2020
C DOWNSTREAM TRANSFER LEARNING TASKS
As we mentioned in our main paper, we prune the pre-trained BERT model (using NIP and RPP) and then fine-tune the sparse pre-trained model different down-stream transfer learning tasks. In this section, we exhibit the performance of pruned BERT using NIP and RPP on wide range of downstream transfer learning tasks demonstrate our conclusions in the main paper.
Through finetuning the pruning BERT over different downstream tasks, we found that SQuAD the most sensitive to the pruning ratio, showing an evident performance drop after 80% pruning ratio. By contrast, the pruning can be made more aggressively when it is evaluated under other finetuning tasks. This not surprising, since SQuAD is much harder Question Answering (QA) tasks, than other simple classification tasks with limited solution space.
On the other hand, as the prune ratio the pre-trained BERT increases, the performances on differ- ent transfer learning tasks descend generally. The descending ranges differ in different downstream transfer learning tasks. The descending range on SQuAD the largest. Our proposed RPP mitigates the descending trend on all downstream transfer learning tasks to great extent, compared with NIP. The intrinsic reason of this descending trend left to future work.
C.I
QQP
Quora Question Pairs is a binary classification task where the goal to determine if two questions asked on Quora are semantically equivalent.
Accuracy (96 //1/F1 score for 00P
70
OGP
Figure A1: Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre-training data and F1 score of fine-tuning on QQP are reported).
Finetuning setting: for fine-tuning on QQP, we set learning rate 2 10-5 batch size 32 and fine tuned for epochs.
15

---

Under review as a conference paper at ICLR 2020
C.2 MRPC
Microsoft Research Paraphrase Corpus consists of sentence pairs automatically extracted from on- line sources, with human annotations for whether the sentences in the pair are semantically equivalent, Dolan & Brocketi (2005)
Finetuning setting: for fine-tuning on MRPC, we set learning rate 2 10-5, batch size 32 and fine-tune for 3 epochs.
100
Accuracy (%6 /F1 score for MRPC
(RPP)
MAPC (RPF))
Figure Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre-training data and F1 score of fine-tuning on MRPC are reported).
C.3 MNLI
Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classification task Williams al. (017). Given pair of sentences, the goal is predict whether the second sentence is an entailment, contradiction, or neutral with respect to the first one.
Finetuning setting: for fine-tuning MNLI, we set learning rate À 2 10-5 batch size 32 and fine-tune for 3 epochs.
100
Accuracy (95)
MSLINIP MNJ (RPP))
MLM (APP)
NSP (APP)
0.40
0.90
Figure Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM NSP accuracy on pre-traine training data and accuracy of fine-tunie fine-tuning on MNLI are reported).
16

---

Under review as a conference paper at ICLR 2020
C.4 MNLIM
Multi-Genre Natural Language Inference has a separated evaluation MNLIM. Following (Devlin al. 2019), the fine-tuning process on MNLIM is separated from MNLL So we present our results on MNLIM in this subsection.
Finetuning setting: for fine-tuning on MNLIM, we learning rate 2- 10-5, batch size 32 and fine-tune for epochs.
100
Accuracy (%)
70
(RPP)
(RPP)
MNLIM
0.30
Figure A4: Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre-training data and accuracy of fine-tuning on MNLIM are reported).
C.5 QNLI
Question Natural Language Inference a version of the Stanford Question Answering Dataset (Ra- jpurkar al., 2016) which has been converted to binary classification task Wanget al., 201 The positive examples are (question, sentence) pairs which do contain the correct answer, and the nega- tive examples are (question, sentence) from the same paragraph which do not contain the answer.
Finetuning setting: for fine-tuning on QNLI, we set learning rate À = 2 10-5, batch size 32 and fine-tune for 3 epochs.
Accuracy 19%)
M.M [NIP)
REST IMP) NSP (RPP)
MLM IRPP)
ONLIAPP))
t-0 t+2
E+4
1+5
1+5
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.90
0.90
Prune Flatio (36)
Figure A5: Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre-training data and accuracy of fine-tuning on QNLI reported).
17

---

Under review as a conference paper at ICLR 2020
C.6 SST-2
The Stanford Sentiment Treebank is binary single-sentence classification task consisting of sen- tences extracted from movie reviews with human annotations of their sentiment(Socher al. 2013).
Finetuning setting: for fine-tuning on SST-2, we set learning rate À == 2 10-5, batch size 32 and fine-tune for epochs. To consistent with the GLUE benchmark (Wang et 2018 the Pearson- Spearman Corr score is reported.
100
Plarson Pearson-Spiaame Spaaman Car
(RPP)
(RPP)
SST-2 (APP)
Figure A6: Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre-training data and accuracy of fine-tuning on SST-2 are reported).
C.7 COLA
The Corpus Linguistic Acceptability is a binary single-sentence classification task, where the goal to predict whether an English sentence is linguistically acceptable not (Warstadt et al. 2018).
Finetuning setting: for fine-tuning on CoLA, we set learning rate 10 5 batch size 32 and fine-tune for 3 epochs. To consistent with the GLUE benchmark (Wang al. 2018), the Matthew's Corr score is reported.
100
Accuracy 1%) Matthew'ss Coir for CoLA
INIP)
(NPP NSP (RPP)
(NP)
MLM (RPP)
Cel A (RPP)
0.00
0.10
0.00
0.50
0.40
0.50
0.60
0.70
0.80
0.00
Prune Rabio
Figure A7: Evaluate the performance of pruned BERT BASE using NIP and RPP, respectively (MLM and NSP accuracy on pre pe-training data and accuracy of fine-tuning on CoLA are reported).
18

---

Under review as a conference paper at ICLR 2020
D NON CONVERGENCE OF PRUNING BERT USING PREVIOUS METHODS
praring one pruning
RPP Tw1
RPP T-2
RPP Tw3
5-
4.
2-
10
15
20
25
30
Epoch
Figure A8: Training loss curve of applying iterative pruning and RPP BERT
As we mentioned in our main paper, we investigate a series of pruning techniques to prune BERT, include the iterative pruning method (Han et al. 2015) and the one-shot pruning (Liu et al. 2019b). However, most of the previous pruning techniques requires to directly optimize the & /2 regular- ization using the pack-propagation based gradient update in the original training of DNN models. We execute school of experiments and find that, this kind of method to optimize regularization might not be compatible with BERT. We show the experiment results about this incompatibility in this section. For the sake of fair comparison, we not only adopt the same hyperparameters (in our experiments about NIP and RPP) on iterative pruning and one-shot pruning, we execute wide set of hyperparamters to make the iterative pruning and one-shot pruning work. We set the learn- ing À E [2 10 10-4, 5 10 3 10- 2 10 10-6, 1 10-5,7 50- 10-6, 1 10-7
", 1 size E (256, 512, 1024, 2048). We execute the same hyperparameters (with NIP and RPP) and tempt more hyperparameters on the iterative pruning and one-shot pruning, but iterative and one-shot pruning could not converge to a valid solution. Figure A8 A8jillustrates training loss curve of applying iterative pruning, one-shot pruming and RPP BERT. It is clear that iterative pruning and one-shot pruning leads to a non-convergence result, while different settings of RPP (T 0, T 1,T = 2) converge well.
10 10-$), batch
From the perspective of optimization and convergence, we make the following analysis:
The previous method, such as Iterative Pruning (IP) and one-shot pruning, relies directly optimiz- ing the e / & penalized training loss to conduct DNN pruning (this is discussed by Han et al. (2015) on iterative pruning, Section 3.1). As result, simultaneous pack-propagation (for updating model weights) conducted over both the original training loss as well as the non-smooth sparsity regu- larizer. When the penalty term is coo-propagated together with the loss function, this affects the convergence of the original loss function. The convergence performance is significantly degraded for extremely large DNN model like BERT. This phenomenon is also observed in the training of BERT (ADAM weight decay) that decouples the regularization term with the original loss function, instead of using an overall back-propagattio.
For the super-deep DNN model (like BERT), it becomes harder to count on the one-time back- propagation flow to solve both the original training objective and the sparse regulariztion at the same time. Loshchilov & Hutter (2018) notices this limitation and improves regularization of Adam, by decoupling weight decay regularization from the gradient-basee update (Loshchilov & Hutter, 2018). A is widely adopted in pretraining large language representations, e.g., BERT (Devlin) et al. 2019). GPT [Radford et al. 2018) and XLNet (Yang et al.) 2019). The difference that, the decoupled weight decay in Adam is to avoid overfitting, while our purpose is to generate sparsity. Moreover, previous algorithms with directly optimizing é through the back propaagtton based gradient update penalty on TPU will easily lead to the gradient NaN.
19

---

Under review as a conference paper at ICLR 2020
Hence proposed New Iterative Pruning (NIP) as working baseline. We believe that NIP works since NIP reflects the naturally progressive pruning performance without any externally introduced penalty. As fix of IP, NIP simplifies the training objective by removing the non-smooth sparsity regularizer. This simple fix improves the convergence of the training process, and makes new itera- tive pruning doable for BERT. We hope that other pruning methods should not perform worse than NIP, otherwise, the effect of optimizing the newly introduced sparsity-promotiee regularization is negative.
To further improve the pruning performance, we need to find a better pruning method that exploits our composite objective structure (original training loss + sparsity regularization), that the back- propagation is not affected for the original training objective of BERT. Motivated by that, proximal gradient provides an elegant solution, which splits the updating rule into a) gradient descent over the original training loss, and b) proximal operation over non-smooth sparsity regularizers. Moreover, reweighted & minimization serves as better sparsity generalization method, which self-adjusting the importance of sparsity penalization weights. Furthermore, the incorporation of reweighted & will not affect the advantage of the proximal gradient algorithm. Thanks to the closed-form solution (equation 8) of proximal operation a weighted & norm, Reweighted Proximal Pruning (RPP) is a desired pruning method on BERT model. We hope RPP proves to be effective in more kinds of DNN models in the future.
20


Reviews:
[{"review_id": "rkgN_clcqr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach to prune weights that is designed keeping large scale pre-trained language representations like BERT. Such a method is desirable for deploying such models on devices with limited memory like phones etc. Experiments on Squad and Glue datasets show that a pruned version of the model maintains high accuracy for these tasks. \n\nPros\n1. Pretty high pruning ratios (80%) can be used for many datasets (except Squad). Its an encouraging result for low-memory requirement scenarios.\n\nWeakness:\n1. Modest technical contribution. The approach description also requires elaboration. Unclear what weights participate in the pruning objective. \n2. Figure 2 is difficult to understand. The paper says \"The sparse attention pattern exhibits obvious structured\ndistribution.\", but I do not know why that is desirable/useful. \n3. The t-SNE visualization appears perfunctory. What should I take away from this analysis?\n4. The baseline approach NIP was derived from the IP approach of Han et al. (2015). Explanation for not using IP is that it does not converge to a \"viable solution\". This needs more elaboration.\n5. Why not compare to teacher-student distillation approaches like DistilBERT? These approaches have the same motivation of compressing model size, though different approach than what the paper adopted. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJlU2stRYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a way to compress Bert by weight pruning with  L1 minimization and proximal method. This paper is one of the first works aiming at  Bert model compression.\nThe authors think the traditional pruning ways can not work well for Bert model, so they propose Reweighted Proximal Pruning and conduct experiments on two different datasets. According to their results, they successfully compress 88.4% of the original Bert large model and get a reasonable accuracy.\n\nStrong points:\n1. The authors propose a new method RPP for Bert model compression.\n2. The authors design experiments to show their RPP can get a very good prune ratio with reasonable accuracy.\n\nWeak points:\n1. The authors should provide a detailed and rigorous explanation for the drawback of existing pruning methods.\n2. In the experiments, the authors only compare RPP with self-designed method NIP instead of any existing pruning method.  The reason they said is “these methods do not converge to a viable solution''. It would be better if they are also compared and analyzed in detail.\n3. In the CoLA and QNLI datasets of Bert_large experiments, RPP can get a higher accuracy even than the original Bert_large model without pruning? This is counter-intuitive.  \n4. About the metrics, the authors use F1 score and accuracy, the standard metrics in the GLUE benchmark for different tasks, except for CoLA. It might make sense to also keep the metrics for CoLA consistent with GLUE benchmark for better comparison.\n5. It is not clear what the authors want to express in Figure 2. The generation of the figure needs more explanation, and the results need to be better interpreted. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlB_8x9cS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nModels such as BERT are pretrained language models which provide significant improvement for different tasks, however they suffer from high huge size and complexity. This paper has proposed using proximal gradient descent to find sparse weights for BERT to reduce the number of parameters and make the model smaller. They concentrate on the drawbacks of the previous sparse-based approaches and claimed that they have convergence issues (they have provided some evidence in the appendix). therefore, they propose to use reweighed sparse method and optimise it using proximal gradient descent which provides a closed form solution for sparse constraint. \n\nALthough proposing a minor novelty (reweighted sparse optimization ), they have provided interesting results for both pretrained structure and fine-tuning for several different tasks. they have also provided some visualisation for the weight matrices after sparsification.\n\nTheir results are notably stronger than simply adding the L1 regularizer to the optimisation method. \n\nThe paper is well written and easy to follow with nearly comprehensive related work.\n\nHowever, there are some drawbacks:\n\n1. They have claimed that “ To the best of our knowledge, we are the first to apply reweighted l1 and proximal algorithm in the DNN weight pruning domain, and achieve effective weight pruning on BERT. ”, however proximal optimization has been used for DNN in works like “Combined Group and Exclusive Sparsity for Deep Neural Networks, 2017”.  \n2. It should be explained clearly about all the matrices included in the sparsification steps, despite only saying “parameters of the model”.\n3. More analysis is required on the results, specially the diagrams for fine-tuning over different datasets.\n4. It is essential to compare the method with other related works for Bert and transformer compression, including quantisation-based, factorisation-based, pruning, knowledge distillation papers such as:\n--Prato, Gabriele, Ella Charlaix, and Mehdi Rezagholizadeh. \"Fully Quantized Transformer for Improved Translation.\" arXiv preprint arXiv:1910.10485 (2019).\n--Tang, Raphael, et al. \"Distilling Task-Specific Knowledge from BERT into Simple Neural Networks.\" arXiv preprint arXiv:1903.12136 (2019).\n--Sanh, Victor, et al. \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.\" arXiv preprint arXiv:1910.01108 (2019).\n--Ziheng Wang, et al. \"Structured Pruning of Large Language Models.\"\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel pruning method for use with transformer text encoding models like BERT, and show that it can dramatically reduce the number of non-zero weights in a trained model while only slightly harming performance.

This is one of the hardest cases in my pile. The topic is obviously timely and worthwhile. None of the reviewers was able to give a high-confidence assessment, but the reviews were all ultimately leaning positive. However, the reviewers didn't reach a clear consensus on the main strengths of the paper, even after some private discussion, and they raised many concerns. These concerns, taken together, make me doubt that the current paper represents a substantial, sound contribution to the model compression literature in NLP.

I'm voting to reject, on the basis of:

- Recurring concerns about missing strong baselines, which make it less clear that the new method is an ideal choice.
- Relatively weak motivations for the proposed method (pruning a pre-trained model before fine-tuning) in the proposed application domain (mobile devices).
- Recurring concerns about thin analysis.
Decision: Reject