Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Authors: Mitchell A Gordon
Kevin Duh
Nicholas Andrews
Venue:  (2020)
Keywords: compression
pruning
pre-training
BERT
language modeling
transfer learning
ML
NLP

Abstract: Universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. A common paradigm is to pre-train a feature extractor on large amounts of data then fine-tune it as part of a deep learning model on some downstream task (i.e. transfer learning). While effective, feature extractors like BERT may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.

Under review as a conference paper at ICLR 2020
COMPRESSING BERT: STUDYING THE EFFECTS OF WEIGHT PRUNING ON TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Universal feature extractors, such BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. A common paradigm is pre-train a feature extractor on large amounts of data then fine-tune it as part of deep learning model on some downstream task (i.e. transfer learning). While effective, feature extractors like BERT may be prohibitively large for some de- ployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning af- fects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-traine -ee-training information from being transferred downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, observe that fine-tuning BERT specific task does improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.
INTRODUCTION
tre-trained feature extractors, such as BERT (Devlin et al.) (018)) for natural language processing and VGG (Simonyan & Zisserman 2014) for computer vision, have become effective methods for improving the performance of deep learning models. In the last year, models similar to BERT have become state-of-the-ar
many NLP tasks, including natural language inference (NL.), named entity recognition (NER), sentiment analysis, etc. These models follow pre-training paradigm: they are trained on a large amount of unlabeled text via a task that resembles language modeling Yang al., 2019: Chan al. 2019) and are then fine-tuned data, which is labeled for a specific task. Pre-trained models usually achieve higher accuracy than any model trained on downstream data alone.
smaller amount of downstream
The pre-training paradigm, while effective, still has some problems. While some claim that lan- guage model pre-training is "universal language learning task (Radford al., 2019), there is no theoretical justification for this, only empirical evidence. Second, due to the size of the pre-training dataset, BERT models tend be slow and require impractically large amounts of GPU memory. BERT-Large can only be used with access to a Google TPU, and BERT-Basc requires some opti- mization tricks such as gradient checkpointing or gradient accumulation be trained effectively on consumer hardware (Sohoni et al.) 2019). Training BERT-Bare from scratch costs S7K and emits 1438 pounds of CO2 (Strubell et 2019)..
Model compression (Bucila et al. 2006), which attempts to shrink model without losing accuracy, viable approach to decreasing GPU usage. might also be used to trade accuracy for mem- ory in some low-resource cases, such deploying to smartphones for real-time prediction. The main questions this paper attempts to answer are: Does compressing BERT impede it's ability to transfer to new tasks? And does fine-tuning make BERT more or less compressible?
To explore these questions, we compressed English BERT using magnitude weight pruning (Han et al. 2015) and observed the results transfer learning the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2019), a diverse set of natural language understanding tasks including sentiment analysis, NLI, and textual similarity evaluation. We chose magnitude

---

Under review as a conference paper at ICLR 2020
weight pruning, which compresses models by removing weights close to O, because it is one of the most fine-graing mne-grained and effective compression methods and because there are many interesting ways to view pruning, which we explore in the next section.
Our findings are as follows: Low levels pruning (30-40%) do not increase pre-training loss or affect transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. This infor- mation is not equally useful to each task; tasks degrade linearly with pre-train loss, but different rates. High levels of pruning, depending on the size of the downstream dataset, may additionally degrade performance by preventing models from fitting downstream datasets. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability or change the order of pruning by a meaningful amount.
To our knowiedge, prior work had not shown whether BERT could be compressed in a task-generic way, keeping the benefits of pre-training while avoiding costly experimentation associated with compressing and re-training BERT multiple times. Nor had it shown whether BERT could be over- pruned for a memory / accuracy trade-off for deployment to low-resource devices. In this work, we conclude that BERT can be pruned prior to distribution without affecting it's universality, and that BERT may be over over-pruned during pre- pre-training for a reasonable accuracy trade- for certain tasks.
PRUNING: COMPRESSION, REGULARIZATION, ARCHITECTURE SEARCH
Neural network pruning involves examining trained network and removing parts deemed to be unnecessary by some heuristic saliency criterion. One might remove weights, neurons, layers, chan- nels, attention heads, etc. depending on which heuristic is used. Below, describe three different lenses through which we might interpret pruning.
Compression Pruning a neural network decreases the number of parameters required to specify the model, which decreases the disk space required to store it. This allows large models to be deployed on edge computing devices like smartphones. Pruning can also increase inference speed if whole neurons or convolutional channels are pruned, which reduces GPU usage
Regularization Pruning neural network also regularizes it. We might consider pruning be form of permanent dropout (Molchanov et al. 2017) or a heuristic-based LO regularizer (Louizos et al.) 2018). Through this lens, pruning decreases the complexity of the network and therefore narrows the range of possible functions it can express The main difference between LO or LI regularization and weight pruning is that the former induce sparsity via a penalty on the loss function, which is learned during gradient descent via stochastic relaxation. It's not clear which approach is more principled or preferred. (Gale al. 2019)
Sparse Architecture Search Finally, we can view neural network pruning as a type of sparse ar- chitecture search. [Luu al. (2019b) and Frankle Carbin (2019) show that they can train carefully re- re-initialized pruned architectures to similar performance levels as dense networks. Under this lens, stochastic gradient descent (SGD) induces network sparsity, and pruning simply makes that spar- sity explicit. These sparse architectures, along with the appropriate initializations, are sometimes referred to as lottery tickets]
2.1 MAGNITUDE WEIGHT PRUNING
In this work, we focus on weight magnitude pruning because is one of the most fine- ane-ggriiedd and effective pruning methods. It also has compelling saliency criterion (Han al., 2015): if a weight close to zero, then its input is effectively ignored, which means the weight can be pruned.
If weights are pruned. however, the weight matrices become sparse. Sparse matrix multiplication difficult to optimize on current GPU architectures (Han et al. (2016), although progress is heing made.
"Interestingly, recent work used compression not to induce simplicity but to measure it (Arora et al. 2018
Sparse networks are difficult to train from scratch (Evci et al. 2019). However, Detmers & Zettlemoyer 2019 and Mostafa & Wang (2019) present methods to do this by allowing SGD to search over the space of possible subnetworks. Our finding suggest that these methods might be used to train sparse BERT from scratch.
2

---

Under review as a conference paper at ICLR 2020
Magnitude weight pruning itself is simple procedure: 1. Pick a target percentage of weights to be pruned, say 50%. 2. Calculate a threshold such that 50% weight magnitudes are under that threshold. 3. Remove those weights, 4. Continue training the network to recover any lost accuracy. 5. Optionally, return to step and increase the percentage of weights pruned. This procedure is conveniently implemented in Tensorflow (Abadi et al. 2016) package" which we use (Zhu & Gupta 2017).
Calculating threshold and pruning can be done for all network parameters holistically (global prun- ing) or for each weight matrix individually (matrix-local pruning). Both methods will prune to the same sparsity, but in global pruning the sparsity might be unevenly distributed across weight matri- ces. We use matrix-local pruning because it is more popular in the community For information on other pruning techniques, we recommend Gale al. (2019) and Liu et al. 2019b).
EXPERIMENTAL SETUP
BERT a large Transformer encoder; for background, we refer readers to Vaswani et al. (2017) (017)or or one of these excellent tutorials (Alammar 2018 Klein et al. (017).
3.1 IMPLEMENTING BERT PRUNING
BERT-Base consists of 12 encoder layers, each of which contains 6 prunable matrices: for the multi-headed self-attention and 2 for the layer's output feed-forward network.
Recall that self seff-attention first projects layer inputs into key, query, and value embeddings via linear projections. While there is separate key, query, and value projection matrix for each attention head, implementations typically stack matrices from each attention head, resulting in only parameter matrices: for key projections, one for value projections, and one for query projections. We prune each of these matrices separately, calculating a threshold for each. We also prune the linear output projection, which combines outputs from each attention head into single embedding
We prune word embeddings in the same way we prune feed-foward networks and self-attention parameters
T The justification similar: if word embedding value is close to zero, we can assume it's zero and store the rest in sparse matrix. This is useful because token subword embeddings tend to account for large portion of a natural language model's memory. BERT-Base specifically, the embeddings account for 221 % of the model's 's memory.
Our experimental code for pruning BERT, based on the public BERT repository, available here
3.2 PRUNING DURING PRE- TRAINING
We perform weight magnitude pruning pre-trained BERT BERT-Base model We select sparsities from 0% to 90% in increments 10% and gradually prune BERT to this sparsity over the first 10k steps of training. We continue pre-training English Wikipedia and BookCorpus for another 90k steps to regain any lost accuracy 10 The resulting pre-training losses are shown in Table
We then fine-tune these pruned models tasks from the General Language Understanding Evalua- tion (GLUE) benchmark, which is standard set of 9 tasks that include sentiment analysis, natural
prunnig "The weights in almost every matrix in BERT-Bas are approximately normally distributed with mean 0 and variance between 0,03 and 0.05 (Table A). This similarity may imply that global pruning would perform similarly matrix-local pruning.
WW could have calculated single threshold for the entire self-attention layer or for each attention head scparately. Similar to global pruning vs. matrix-local pruning, it's not clear which one should be preferred. "Interestingly, pruning word embeddings is slightly more interpretable that pruning other matrices. See Figure for a heatmap of embedding magnitudes, which shows that shorter subwords tend to pruned more than longer subwords and that certain dimensions are almost never pruned in any subword.
*URL omitted for anonymity
thttps//github.com. Evaluation leveled 20k steps.
3

---

Under review as a conference paper at ICLR 2020
language inference, etc. We avoid WNLI, which is known be problematic We also avoid tasks with less than 5k training examples because the results tend to be noisy (RTE, MRPC, STS-B). We fine-tune separate model on each of the remaining 5 GLUE tasks for 3 epochs and try 4 learn- ing rates: [2, 3, 4,5) x 10-5, The best evaluation accuracies are averaged and plotted in Figure Individual task results in T
BERT can be used as a static feature-extractor or as a pre-trained model which is fine-tuned end-to- end. In all experiments, we fine-tune weights in all layers of BERT on downstream tasks.
3.3 DISENTANGLING COMPLEXITY RESTRICTION AND INFORMATION DELETION
Pruning involves steps: it deletes the information stored in weight by setting it and then regularizes the model by preventing that weight from changing during further training.
To disentangle these two effects (model complexity restriction and information deletion), repeat the experiments from Section 3.2 with an identical pre-training setup, but instead of pruning we simply set the weights pre- pre-training information associated with the weight but does not prevent the model from fitting downstream datasets by keeping the weight at zero during downstream training. We also fine-tune on downstream tasks until training loss becomes comparable to models with no pruning, We trained most models for 13 epochs rather than 3. Models with 70-90% information deletion required 15 epochs to fit the training data. The results are also included in Figure and Table [
and allow them vary during downstream training. This deletes the
3.4 PRUNING AFTER DOWNSTREAM FINE-TUNING
We might expect that BERT would be more compressible after downstream fine-tuning, Intuitively, the information needed for downstream tasks is a subset of the information learned during pre- training; some tasks require more semantic information than syntactic, and vice-versa. We should be able discard the "extra" information and only keep what we need for, say, parsing (Li Fisner 2019).
For magnitude weight pruning specifically, we might expect downstream training to change the distribution of weights in the parameter matrices. This, in turn, changes the sort-order of the absolute values of those weights, which changes the order that we prune them in. This new pruming order, hypothetically, would be less degrading to specific downstream task.
To test this, we fine-tuned pre-trained BERT-Base on downstream data for 3 epochs. We then pruned various sparsity levels and continued training for more epochs (7 for 80/90% sparsity), which point the training losses became comparable to those models pruned during pre-training. We repeat this for learning rates in (2,3, 4, 10-5 and show the results with the best development accuracy in Figure / Table [ We also measure the difference in which weights are selected for pruning during pre- pre-training vs. downstream fine-tuning and plot the results in Figure
PRUNING REGIMES
4.1
30-40% OF WEIGHTS ARE NOT USEFUL
Figure T shows that the first 30-40% of weights pruned by magnitude weight pruming do not impact pre-training loss inference any downstream task. These weights can be pruned either before or after fine-tuning. This makes sense from the perspective of pruning as sparse architecture search: when we initialize BERT-Base, we initialize many possible subnetworks. SGD selects the best one for pre-training and pushes the rest of the weights to O. We can then prune those weights without affecting the output of the network
" htttps//glubbbnchhatk com//aq
We know, however, that increasing the size of BERT to BERT-Large improves performance. This view does not fully explain why even an obviously under-panee parameterized model should become sparse. This may be caused by dropout, it may be general property of our training regime (SGD). Perhaps an extension of Tian al. 2019) parametaraeee models would provide some insight.
4

---

Under review as a conference paper at ICLR 2020
Average GLUE Dev Ace
Average GLUE Tre ning Loss
info deletion
0.5
downstrear
random pruning
BEAT 0% Pruna
Dey Aoc
0.4
0.75
0.3
prune
info dhletion
0.2
prane
random praning
Frune
0.8
0.4 Percentage
Percentage
Figure 1: (Blue) The best GLUE dev accuracy and training losses for models pruned during pre- training, averaged over 5 tasks. Also shown models with information deletion during pre-training (orange), models pruned after downstream fine-tuning (green), and models pruned randomly during pre-training instead of by lowest magnitude (red). 30-40% of weights can be pruned using mag- nitude weight pruning without decreasing dowsntream accuracy. Notice that information deletion fits the training data better than un- un-pruned models all sparsity levels but does not fully recover evaluation accuracy. Also, models pruned after downstream fine-tuning have the same worse development accuracy, despite achieving lower training losses. Note: none of the pruned models overfitting because un-pruned models have the lowest training loss and the highest development accuracy. While the results for individual tasks are in Table [ each task does not vary much from the average trend, with an exception discussed in Section
Pre-treining vs. Accuracy
1.5
Average Feeture Cosine sim with Prune 0
NNLI
0.90
DgP
ONU
FalA
Clue Acc
Loonoo 3mm
0.80
0.75
0.2
2.0
2.5
3.0
3.5
4.3
4.5
5.0
5.5
60
02
0.6
Pre Pre-Training Loss
prune Harcentage
Figure 2: (Left) Pre-training loss predicts information deletion GLUE accuracy linearly as sparsity increases. We believe the slope of each line tells how much bit of BERT is worth each task. (CoLA at 90% excluded from the line of best fit.) (Right) The cosine similarities of features extracted for subset of the pre-training development data before and after pruning. Features are extracted from activations of all 12 layers of BERT and compared layer-wise to model that has not been pruned. As performance degrades, cosine similarities of features decreases.
5

---

Under review as a conference paper at ICLR 2020
4.2 MEDIUM PRUNING LEVELS PREVENT INFORMATION TRANSFER
Past 40% pruning, performance starts to degrade. Pre-training loss increases as we prune weights necessary for fitting the pre-training data (Table []. Feature activations of the hidden layers start to diverge from models with low levels of pruning (Figure 2 Downstream accuracy also begins to degrade at this point.
Why does pruning at these levels hurt downstream performance? On one hand, pruning deletes pre- training information by setting weights to O, preventing the transfer the useful inductive biases learned during pre-training. On the other hand, pruning regularizes the model by keeping certain weights at zero, which might prevent fitting downstream datasets.
Figure 1 and Table [ show information deletion is the main cause of performance degradation between 40 60% sparsity, since pruning and information delction degrade models by the same amount. Information deletion would not be problem if pre-training and downstream datasets con- tained similar information. However, pre-training is effective precisely because the pre-training dataset much larger than the labeled downstream dataset, which allows learning of more robust representations.
We that the main obstacle to compressing pre-trained models is maintaining the inductive bias of the model learned during pre-training. Encoding this bias requires many more weights than fitting downstream datasets, and it cannot be recovered due to fundamental information gap between pre-training and downstream datasets [ The amount a model can be prumed is limited by the largest dataset the model has been trained on: in this case, the pre-training dataset. Practitioners should be aware of this; pruning may subtly harm downstream generalization without affecting training loss.
4.3 HIGH PRUNING LEVELS ALSO PREVENT FITTING DOWNSTREAM DATASETS
At 70% sparsity and above, models with information deletion recover some accuracy w.r.t. pruned models, so complexity restriction is a secondary cause of performance degradation. However, these models do not recover all evaluation accuracy, despite matching un un-pruned model's training loss.
Table shows that on the MNLI and QQP tasks, which have the largest amount of training data, information deletion performs much better than pruning. In contrast, models do not recover as well on SST-2 and CoLA, which have less data. We believe this is because the larger datasets require larger models to fit, so complexity restriction becomes an issue earlier.
We might be concerned that poorly performing models are over-fitting, since they have lower train- ing losses than unpruned models. But the best performing information-deleted models have the lowest training error of all, so overfitting seems unlikely.!!
4.4 How MUCH IS BIT OF BERT WORTH?
We've seen that ever-pruning BERT deletes information useful for downstream tasks. this infor- mation equally useful to all tasks? We might consider the pre-training loss as a proxy for how much pre-training information we' ve deleted in total. Similarly, the performance of information-de deletion models is proxy for how much of that information was useful for each task. Figure 2 shows that the pre-training loss linearly predicts the effects of information deletion on downstream accuracy.
For every bit of information we delete from BERT, it appears only a fraction is useful for CoLA, and an even smaller fraction useful for QQP 16 This relationship should be taken into account when
We believe this observation may point towards a more principled stopping criterion for pruning. Currently, the only way to know how much to prune is by trial and (dev-set) error. Predictors of performance degradation while pruning might help us decide which level of sparsity is appropriate for given trained network without trying many at once.
14 We might consider finding a lottery ticket for BERT, which we would expect to the GLUE training data just as well as pre-trained BERT (Morcos et al. 2019 Yu et al. 2019). However, we predict that the lottery- ticket will not reach similar generalization levels unless the lottery ticket encodes enough information close the information gap.
15 We are reminded of the double-descent risk curve proposed by Belkin et al. 2018 16 We can't 't quantify this now, but perhaps compression will help quantify the "universality" of the LM task.

---

Under review as a conference paper at ICLR 2020
Pre-train vs. Downstream Pruning
Offect of rine-tuning Before Pruning 60%
DOF
Pruning Mask Differ Diffkriieeettt renee 1%)
Praning Diference (%)
ONU
S
0.90
COLA
0.85
0.80
Aug
0.75
0.70
0.65
Sponsity
Downstream Esochs Before Pruning
Figure 3: (Left) The measured difference in pruning masks between models pruned during pre- training and models pruned during downstream fine-tuning. As predicted, the differences are less than 6%, since line-tuning only changes the magnitude sorting order of weights locally, not globally. (Right) The average GLUE development accuracy and pruning mask difference on downstream datasets before pruning 60% at learning rate 5e-5. After pruning, models are trained for an additional epochs to regain accuracy. We see that training between 3 and 12 epochs before pruning does not change which weights are pruned or improve performance.
models trained
considering the memory / accuracy trade-off of over-pruning Pruning an extra 30% of BERT's weights worth only one accuracy point on QQP but 10 points on CoLA. It's unelear, however, whether this is because the pre-training task is less relevant to QQP or whether QQP simply has bigger dataset with more information content
DOWNSTREAM FINE-TUNING DOES NOT IMPROVE PRUNABILITY
Since pre-training information deletion plays a central role performance degradation while over- pruning, we might expect that downstream fine-tuning would improve prunability by making im- portant weights more salient (increasing their magnitude). However, Figure [ shows that models pruned after downstream fine-tuning do not surpass the development accuracies of models pruned during pre-training, despite achieving similar training losses. Figure 3 which weights are pruned by less than 6%.
shows fine-tuning changes
showws shows that the magnitude sorting order of weights mostly preserved; weights move average 0-4% away from their starting positions in the sort order. We also see that high magnitude weights are more stable than lower ones (Figure
Why doesn't 't fine-tume fine-tuning change which weights are pruned much? Table 2
Our experiments suggest that training on downstream data before pruning is too blunt an instrument improve prunability. Even so, we might consider simply training on the downstream tasks for much longer, which would increase the difference in weights pruned. However, Figure 4 shows that even after an epoch of downstream fine-tuning, weights quickly re-stabilize meaning longer downstream training will have only marginal effect on which weights are pruned. Indeed, Figure 3 shows that the weights selected for 60% pruning quickly stabilize and evaluation accuracy does not improve with more training before pruning.
new sorting order,
RELATED WORK
Compressing BERT for Specific Tasks Section showed that downstream fine-tuning does not increase prunability. However, several alternative compression approaches have been proposed to discard non-task-specifee information. [i Eisner (2019) used information bottleneck to discard non-syntactic information. Tang et al. (2019) used BERT as a knowledge distillation teacher to
Hendrycks et al. 2019 suggest that pruning these weights might have hidden cost: decreasing model robustness.
7

---

Under review as a conference paper at ICLR 2020
% of Individual Attention Head Pruned
Weig it Sort Order Movement During Downstream Training
Avg % Movementt Weights in Sort Order
0.8
Afm Hoad Pruned
0.6
12 12
0.4
COP
0.2
SST
00
Sporsity
Downstrcam Epachs
Figure 4: (Left) The average, min, and max percentage of individual attention heads pruned at each sparsity level. We see 60% sparsity, each attention head individually is pruned strictly between 55% and 65%. (Right) We compute the magnitude sorting order of cach weight before and after downstream fine-tuning.
weight's 's original position is 59 100 before fine-tum fine-tnnig and 63 100 after fine-tuning, then that weight moved 4% in the sorting order. After even an epoch of downstream fine-tuning, weights quickly stabilize in new sorting order which not far from the original sorting order. Variances level out similarly.
compress relevant information into smaller Bi-LSTMs, while Kuncoro al. (2019) took a similar distillation approach. While fine-tuning does not increase prunability, task-specific knowledge might be extracted from BERT with other methods.
Attention Head Pruning Voita et al. (2019) previously showed redundancy in transformer models by pruning entire attention heads. Michel et al.] (2019) showed that after fine-tuning on MNLI, up to 40% of attention heads can be pruned from BERT without affecting test accuracy. They show redundancy in BERT after fine-tuning on a single downstream task; emphasizes the interplay between compression and transfer learning to many tasks, pruning both before and after fine-tuning. Also, magnitude weight pruning allows us to additionally prune the feed-foward networks and sub-word embeddings in BERT (not just self-attention) which account for ~72% of BERT's total memory usage.
contrast, our work
We suspect that attention head pruning and weight pruning remove different redundancies from BERT. Figure 4 shows that weight pruning does not prune any specific attention head much more than the pruning rate for the whole model. It is not clear, however, whether weight pruning and recovery training makes attention heads less prunable by distributing functionality to unused heads.
CONCLUSION AND FUTURE WORK
We've shown that encoding BERT's s inductive bias requires many more weights than are required to fit downstream data. Future work on compressing pre-trained models should focus on maintaining that inductive bias and quantifying its relevance to various tasks during accuracy/memory trade-off's.
For magnitude weight pruning, we've shown that 30-40% of the weights do not encode any useful inductive bias and can be discarded without affecting BERT's universality. The relevance of the rest of the weights vary from task to task, and fine-tuning on downstream tasks does not change the nature of this trade-off by changing which weights are pruned. In future work, we will investigate the factors that influence language modeling's relevance to downstream tasks and how improve compression in task-generas way.
It's reasonable to believe that these conclusions will generalize to other pre-trained language mod- els such Kermit (Chan et al. 2019), XLNet (Yang et al.) 2019), GPT-2 (Radford et al. 2019). RoBERTa (Liu et al.) [201 or ELMO (Peters et al., 2018). 8) All of these learn some variant of language modeling, and most use Transformer architectures. While it remains to be shown in fu- ture work, viewing pruning as architecture search implies these models will be prunable due to the training dynamics inherent to neural networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gre- gory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan J. Good- fellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore. Derek Gor- don Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467,, 2016. URL httt:://////xxivvoo.
11663644637 C4467
Jay Alammar. The illustrated transformer, 2018.
URL
1o/
illustzatee-traasooooom
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. CoRR, abs/1802.00 802.05296,, 2018. URL http://rxva
05296
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mand al. Reconciling modern machine leamn ing practice and the bias-variance trade-off. arXiv e-prints, art. arXiv:1812.111 1118, Dec 2018.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceed- ings of the Twelfih ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia. PA, USA, August 23, 2006, pp. 535 541, 2006. doi: 10.1145/1150402 1150464. URL httpt::t///oooo org/ wttp///....111/11111 1141/111144100 1tw.1150002.11101...0 1150464
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: gen- erative insertion-based modeling for sequences. CoRR, abs/1906.01604, 2019. URL http:
//axxiv.orr//bb//1966. 01604
Tim Dettmers and Luke S. Zettlemoyer. Sparse networks from scratch: Faster training without 2019.
losing performance. ArXiv,
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Utku Evci, Fabian Pedregosa, Aidan N. Gomez, and Erich Elsen. The difficulty of training sparse neural networks. CoRR, Abs/1906.10734 2019. URL http:://axttt wwwpaaiiii......aaaa/aa wtt.:ogg/abbs/loooo
10732]
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://
openreview .nt/ forum? 2drooummmmddr3--33F2
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, abs/1902.09574, 2019. URL http::/ //txxia.avvvv///////a/eaaeaaavvv org/ abs aabs/1902 1902.11122.2 09574
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1135- 1143. Curran Associates. Inc., 2015. URL http::/papee:: nips cc/paper/ 5784 1244-eearning-bt... pdf
and wwl--conneiiionss for
-oaneettooos-fo-eeffiio. wow.eifiieeettt--ooooerr---ee-ee neural network.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J.
Dally. Eic: Efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture, ISCA '16, pp. 243-25% Piscataway, NJ, USA, 2016. IEEE Press. ISBN 14771-44673-894711
doi: (0.1109/SCA.221133333 URI.
https: //ooi.d////. ww..rrg/l. / 1009 TSCA 201 6.30
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training improve model robust- ness and uncertainty. In ICML, pp. 2712-2721, 2019. URL http://proceednngs
press/v97/hendrycee 9a. html

---

Under review as a conference paper at ICLR 2020
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Proc. ACL, 2017. doi: 10.18653/v1/ P17-4012. Utthttps: sttetps://do... org 11w..11///////1/0000000
Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. Scalable syntax- aware language models using knowledge distillation. CoRR, abs/1906.06438, 2019. URLhttp:
//aaxiv..rg/abbb.....a 906 6 06438
Xiang Lisa Li and Jason Eisner. Specializing word embeddings (for parsing) by information bottle- neck. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and 9th International Joint Conference Natural Language Processing, Hong Kong, November 2019. URL htLLtttp://se. cs. http.hhh.//aaaaso//aaaaeeeee http:///aaaaatttteeeeeeeeee----e 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019a. OtLtttt://aaxix org. wwwp////////////o//////// 11692
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference Learning Representations, 2019b. URL
https::/cppenrevieet net. forum?1d=::1111333
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through I- 0 regularization. In International Conference on Learning Representations, 2018. URL https:
//openzeview. http://wweetttoorreeeeeeeaaa.eee
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? ArXiv, abs/1905.10650, 2019.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov, Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 2498 2507. JMLR.org, 2017. URL http: hwtp:///dlaccc.../oommmmmmm. wwtp://.ggggaatitieiieeeeeeeitce
cfm2id=3305899 3305939
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuand ong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv e-prints, art. 1XXiv:1906007777 Jun 2019.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization, 2019. URL /tRR///hpps//peeneeeeee.eee////e
id S1 x85K
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. CoRR, abs//802.00365, 2018. htrgg/abs18802 05365
URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large- Scale Image Recognition. arXiv e-prints, art. arXiv: 1556, Sep 2014.
Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszezynski, Jian Zhang, and Christo- pher Ré. Low-memory neural network training: A technical report. CoRR, abs/ 1005//9944.1631,, 2019. URL http::://axxii..o/////aaaa hbrggg//bs/19// 10631
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. CoRR, abs/1906.02244 2019, URL http://arxxvv.rr3/2b////19
02243
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from BERT into simple neural networks. CoRR, 1tt./193..1116,, 2019. URL
http:://rxiii
http:aawwwwwoogaabb.b//abbooaae/ www.:voorrggg... /19/a 12136
10

---

Under review as a conference paper at ICLR 2020
Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari S. Morcos. Luck matters: Understanding training dynamics of deep relu networks. CoRR, 0bbs/1905.13400 2019. URL http::/aaax.
org/abs/ 1905. 13405
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
URLhttp: wttp//ii.orooveeoooeee httpa/as..11.0..0 03762
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1 1580. URL https: www. http:/wwwwwwww..c..//wpwppwww.pt org org./anooooogg/111----11
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL
net/Forumridd-14kkn222
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237,, 2019. URL httEt://axxixx wwteeiiiiirrraaeooooooo httpg//bs/11000 08237
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets RL and NLP. arXiv e-prints, art. 190.:190...2266. Jun 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, arXiv e-prints, art, arXiv:1710.01878 Oct 2017.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
Pre-train Loss
MNLI
QQP
QNLI
SST-2
CoLA
Pruned
AVG
392k
363k
108k
67k
8.5k
0
1.82
0.25
90.5|0.10
91.1
92.1 0.06
0..1
85.2
10
1.82
8.23
90.4|0.10
91.0
91.6|0.07
7.34
87.2 16.0
20
1.83
0.23
90.5
91.1
91.6
79.1
87.1 16.0
30
1.86
8..3
90.2|0.12
90.7
9..9
79.5
87.1 87.1
40
1.93
8.25
90.1
90.4
91.5
78.4
86.7
50
2.03
82.6
89.8|0.13
90.2
90.9/0.07
77.4/0.30
86.2
60
2.25
0.38
89.4)0.16
89.3
91.4|0.07
75.9(0.44
85.6
70
2.62
79.5
88.6/0.18
88.4
90.1
72.7
83.9
80
3.44
0.49
86.9
8..3
88.1
69.1
81.1
90
5.83
0.76
8..1
7.57
80.3
69.1
73.4|49.8
Information Deletion
0
1.82
0.20
90.6
90.0
92.1 0.03
80.6/0.18
81.3
10
1.82
8..8
90.5
90.5
9..2
80.8
87.4
20
1.83
8.09
90.5
90.5
91.5
80.3
87.2
30
1.86
8..3
90.6|0.04
90.5
90.8/0.05
80.0
86.9
40
1.93
0.12
90.5
90.1
92.0
79.0
86.7
50
2.03
8.15
90.3|0.05
90.2
91.2
77.9(0.19
86.4
60
2.25
0.29
90.1
89.5
90.8
76.4
85.7
70
2.62
0.08
90.2
88.7
9..3
74.4
84.9
80
3.44
7..6
8.03
86.0
88.8
70.0
81.5
90
5,83
72.9
0.05
76.8/0.06
83.0
69.1
77.9
Pruned after Downstream
Fine-tuning
0
82.6
9..6
90.1
92.1
7..7
86.8
10
0.
90.6
90.3
91.6|0.05
79.0
86.9
20
82.7
90.6
90.2
92.0
79.0
86.9
30
8.27
90.4
89.7
91.6|0.04
78.5
86.6
40
8..7
90.5
89.9
91.7
78.8(0.17
86.7
50
8.16
90.3
89.7
90.8
78.0/0.22
86.3 13.0
60
8..8
90.2
89.3
90.6
76.1
85.6
70
8..5
89.4
86.2
88.2
69.5
82.7
80
0.53
87.8
80.4
86.4
69.1
79.5 30.5
90
58.7 0.86
8..5
0.52
81.5 0.16
69.1
71.4
Random Pruning
0
1.82
83.3
90.5
90.6|0.15
92.4
78.7
85.3
10
2.09
8.20
90.1
90.3
92.3
77.0/0.32
86.3
20
2.46
8..6
89.8
88.5
91.1
73.5
84.7
30
2.98
0.36
89.2
86.9|0.23
89.3
71.8(0.47
83.3/25.9
40
3.76
75.4
88.2
84.5/0.23
88.6/0.09
69.3/0.57
81.2
50
4.73
71.6/0.60
86.6
81.5
85.0/0.10
69.1
78.8
60
5.63
7..4
85.2
71.7
81.5|0.21
69.1
75.6
70
6.22
6.71
81.4
63.0/0.62
80.6/0.20
69.1
71.6
80
6.87
58.8
76.6/0.46
61.1
80.6/0.23
69.1
69.3
90
7.37
49.8
74.3
6..2
75.1 0.33
69.1
65.7
Table 1: Pre-training development losses and GLUE task development accuracies for various levels of pruning. Each development accuracy is accompanied its right by the achieved training loss, evaluated on the entire training set. Averages are summarized in Figure [ Pre-training losses are omitted for models pruned after downstream fine-tuning because it is not clear how to measure their performance on the pre-training task in a fair way.
12

---

Under review as a conference paper at ICLR 2020
Sum of Weights Pruned
Absol.00
7500000
2000000
1500020
1000000
500000
Spanity
Figure 5: The of weights pruned at each sparsity level for shot pruning of BERT. Given the motivation for our saliency criterion, seems strange that such a large magnitude of weights can be pruned without decreasing accuracy.
Figure 6
LR
MNLI
QQP
QNL
SST-2
CoLAA
2e-5
1.91 + 1.81
1.82 1.72
1.27
1.22
1.06
1.03
0.79 0.77
3e-5
2.68 2
2.56
1.79 +
1.69
1.54 1.47
1.06 1.03
4e-5
3.18
3.30 4.30
2.19
1.99 1.89
1.11 + 1.09
5e-5
3.83
3.77
2.77
2.62
2.38 + 2.29
1.47 = 1.43
Table We compute the magnitude sorting order each weight before and after downstream fine- tuning. If weight's original position is 59 / 100 before fine-tuning and 63 100 after fine-tuning, then that weight moved 4% in the sorting order. We then list the average movement of weights in each model, along with the standard deviation. Sorting order changes mostly locally across tasks: weight moves, on average, 0-4% away from its starting position. As expected, larger datasets and larger learning rates have more movement (per epoch). We also see that higher magnitude weights are more stable than lower weights, see Figure
Distribution of Sort Order Movements
0.024
0.110
0.000
0.004
0.000
Saatiing Magn tocko Sort Crier Poudson
Figure 7: We show how weight sort order movements are distributed during fine-tuning, given weight's starting magnitude. We that higher magnitude weights more stable than lower magnitude weights and do not move as much in the sort order. This plot is nearly identical for every model and learning rate, so we only show it once.
13

---

Under review as a conference paper at ICLR 2020
Paroreter Hottre Hestrep
Figure 8: heatmap of the weight magnitudes of the horizontally stacked self-attention key projection matrices for layer 1. A banding pattern can be seen: the highest values of the matrix tend to cluster in certain attention heads. This pattern appears in most of the self-attention parameter matrices, but it does not cause pruning to prune one head more than another. However, it may prove to be a useful heuristic for attention head pruning, which would not require making many passes over the training data.
Figure 9: A heatmap of the weight magnitudes of BERT's subword embeddings. Interestingly. pruning BERT embeddings are more interpretable; can see shorter subwords (top rows) have smaller magnitude values and thus will be pruned earlier than other subword embeddings.
14

---

Under review as a conference paper at ICLR 2020
Weight Matrix
Weight Mean
Weight STD
embeddings word embeddings
-0.0282
0,042
layer O attention output FC
-0.0000
0.029
layer 0 self attn key
0.0000
0.043
layer O self attn query
0.0000
0.043
layer 0 self attn value
-0.0000
0.029
layer 0 intermediate FC
-0.0000
0.037
layer 0 output FC
-0.0012
0.036
layer attention output FC
0.0001
0.028
layer self attn key layer 1 self attn query layer self attn value
0.0000
0.043
-0.0003
0.043
-0,0000
0.029
layer I intermediate FC
0.0001
0.039
layer output FC
-0.0014
0.038
layer 10 attention output FC
-0.0000
0.033
layer 10 self attn key
-0.0000
0.046
layer 10 self attn query
0.0002
0.046
layer 10 self attn value
-0.0000
0.036
layer 10 intermediate FC
0.0000
0.039
layer 10 output FC
-0.0011
0.038
layer 11 attention output FC
-0.0000
0.037
layer 11 self attn key
0.0002
0.044
layer 11 self attn query
-0.0001
0.045
layer 11 self attn value
-0.0000
0.039
layer 11 intermediate FC
0.0004
0.039
layer 11 output FC
-0.0008
0.036
layer attention output FC
0.0000
0.027
layer self attn key
0.0000
0.047
layer 2 self attn query
0.0000
0.048
layer 2 self attn value
-0,0000
0.028
layer 2 intermediate FC
0.0001
0.040
layer output FC
-0.0015
0.038
layer 3 attention output FC
0.0001
0.029
layer self attn key
0.0000
0.043
layer 3 self attn query
0.0003
0.043
layer 3 self attn value
-0.0001
0.031
layer 3 intermediate FC
-0.0001
0,040
layer output FC
-0.0014
0.039
layer 4 attention output FC
0.0000
0,033
layer self attn key
0.0000
0.042
layer 4 self attn query
-0.0001
0.042
layer 4 self attn value
0.0001
0.035
layer 4 intermediate FC
0.0001
0.041
layer 4 output FC
-0.0014
0.040
layer attention output FC
-0,0000
0.033
layer self attn key
-0.0001
0.043
layer 5 self attn query
-0.0000
0.043
layer 5 self attn value
-0.0000
0.035
layer intermediate FC layer 5 output FC
0.0000
0.041
-0.0014
0.039
15

---

Under review as a conference paper at ICLR 2020
layer attention output FC
0.0001
0.032
layer 6 self attn key -0.0000
0.043
layer 6 self attn query
0.0001
0.043
layer 6 self attn value
0.0000
0.034
layer intermediate FC
-0.0000
0.041
layer output FC
-0.0014
0.039
layer attention output FC
0,0000
0.032
layer 7 self key
-0,0000
0.044
layer 7 self attn query
00,000
0.044
layer 7 self value
0.0001
0.033
layer intermediate FC
0.0003
0.039
layer output FC
-0.0013
0.038
layer attention output FC
0.0000
0.034
layer 8 self attn key
-.0000
0.044
layer 8 self attn query
0.0001
0.044
layer 8 self attn value
0.0000
0.035
layer intermediate FC
0.0004
0.039
layer 8 output FC
-0.0013
0.037
layer attention output FC
0.0001
0.033
layer 9 self attn key
0,0000
0.046
layer self attn query
-0.0001
0.046
layer 9 self attn value
0.0000
0.035
layer intermediate FC
0.0005
0.040
layer output FC pooler FC
-0.0012
0.039
0.0000
0.029
Table The values of BERT's weights are normally distributed in each weight matrix. The means and variances are listed for each.
16


Reviews:
[{"review_id": "HJeqpvgaYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conducts a series of interesting experiments on compressing BERT and makes several conclusions. The compression technique is magnitude weight pruning based on an existing work. The paper mainly tested different compression rates and the stages when the compression can be applied. Compared to the existing work, one main contribution of the paper is to show that the BERT model can be pruned prior to fine-tuning any specific downstream tasks by 30%-40% without affecting all tested downstream tasks much. The paper is well motivated and presents interesting experimental results and conclusions. I have some concerns on their experiment details, which needs some clarification.\n\n1. the observation in 3.4 is a little counter-intuitive to me. The model has all pre-trained weights and should be able to determine, during fine-tuning, which weights to decrease to nearly zero or to abandon. However, the experimental results show that the pruning at that point produces a worse dev accuracy. For the experiments, 3 epochs is used for fine-tuning and then the pruning is applied. I was wondering what happen if you first fine-tune the model to get the best dev accuracy and prune the weights at that point. How did you choose the number 3? I am guessing that the pruning in the middle of fine-tuning process may throw away useful information too early.   \n2. It will be helpful to show the thresholds of pruning and how these thresholds relate to the training loss and accuracy. I think the value of the thresholds can tell whether some pruning ratios are reasonable.\n3. when the authors continue training the model, for example in 3.4, the training stops when the training losses are comparable. Why did the training loss is used as the metric instead of the dev accuracy? Figure 1 right seems to show that those models are overfitting. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xSiSF6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is an empirical study of testing how pruning at the pre-training stage affects subsequent transfer learning (through fine-tuning) stage. The main idea is to carefully control the amount of sparsity injected into BERT through weight magnitude pruning and study the impact on accuracy. The experimental setup is mostly well done, especially the part that disentangles the complexity restriction and information deletion. During the exploration, the authors made several interesting observations, such as 30-40% model weights do not encode any useful inductive bias, which could help shed some light for future work on both training and compressing BERT-like models.\n\nOverall, the paper is well written and explained. The goal is meaningful, and this is a sensible contribution to the ongoing interests of compressing BERT-like large models for efficient training and inference. \n\nMy major concern is on its novelty and how directly it can provide benefit to computation.  First, although the findings are interesting, the methods used in this paper are not new.  Various pruning techniques have been explored in prior work, which makes the novelty contribution of this paper somewhat limited. \n\nFurthermore, the study has mostly focused on the impact of random sparsity to accuracy. However, as it is known that it is really difficult for modern hardware to benefit from random sparsity because it leads to irregular memory accesses, which negatively impact the performance. It has been observed that speedups are very limited or can be negative even the random sparsity is >95% [1]. Therefore, it is hard to judge how inference or training can benefit from 30-40% weight sparsity. Going forward, the authors are encouraged to choose pruning methods that lead to regular memory access to avoid adversely impacting practical acceleration in modern hardware platforms.\n\n[1] Learning Structured Sparsity in Deep Neural Networks. Wen et al. NeurIPS 2016", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeAGC3pYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work explores weight pruning for BERT. It finds that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of\npruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation.\n\nMy major concern about this work is its technical innovation and value to the community.\n1. This is simply a study of model pruning for BERT. There is nothing new technically.\n\n2. It shows BERT can be pruned for 30-40% parameters. Actually, this is not surprising; instead I'm even disappointed about this result. 30-40% weight reduction does not really speed up inference much or save model size much. Besides, to handle sparse weight matrixes, one may need additional operations to use the pruned models on a modern GPU.\n\n3. Several other submissions show that BERT models can be compressed for 5-10x without accuracy loss. Comparing with this work, this paper seems to tell me that pruning is not suitable for BERT.   ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work explores weight pruning for BERT in three broad regimes of transfer learning: low, medium and high.

Overall, the paper is well written and explained and the goal of efficient training and inference is meaningful. Reviewers have major concerns about this work is its technical innovation and value to the community: a reuse of pruning to BERT is not new in technical perspective, the marginal improvement in pruning ratio compared to other compression method for BERT, and the introduced sparsity that hinders efficient computation for modern hardware such as GPU. The rebuttal failed to answer a majority of these important concerns.

Hence I recommend rejection.
Decision: Reject