Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This work explores weight pruning for BERT. It finds that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of
pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation.

My major concern about this work is its technical innovation and value to the community.
1. This is simply a study of model pruning for BERT. There is nothing new technically.

2. It shows BERT can be pruned for 30-40% parameters. Actually, this is not surprising; instead I'm even disappointed about this result. 30-40% weight reduction does not really speed up inference much or save model size much. Besides, to handle sparse weight matrixes, one may need additional operations to use the pruned models on a modern GPU.

3. Several other submissions show that BERT models can be compressed for 5-10x without accuracy loss. Comparing with this work, this paper seems to tell me that pruning is not suitable for BERT.   
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work is an empirical study of testing how pruning at the pre-training stage affects subsequent transfer learning (through fine-tuning) stage. The main idea is to carefully control the amount of sparsity injected into BERT through weight magnitude pruning and study the impact on accuracy. The experimental setup is mostly well done, especially the part that disentangles the complexity restriction and information deletion. During the exploration, the authors made several interesting observations, such as 30-40% model weights do not encode any useful inductive bias, which could help shed some light for future work on both training and compressing BERT-like models.

Overall, the paper is well written and explained. The goal is meaningful, and this is a sensible contribution to the ongoing interests of compressing BERT-like large models for efficient training and inference. 

My major concern is on its novelty and how directly it can provide benefit to computation.  First, although the findings are interesting, the methods used in this paper are not new.  Various pruning techniques have been explored in prior work, which makes the novelty contribution of this paper somewhat limited. 

Furthermore, the study has mostly focused on the impact of random sparsity to accuracy. However, as it is known that it is really difficult for modern hardware to benefit from random sparsity because it leads to irregular memory accesses, which negatively impact the performance. It has been observed that speedups are very limited or can be negative even the random sparsity is >95% [1]. Therefore, it is hard to judge how inference or training can benefit from 30-40% weight sparsity. Going forward, the authors are encouraged to choose pruning methods that lead to regular memory access to avoid adversely impacting practical acceleration in modern hardware platforms.

[1] Learning Structured Sparsity in Deep Neural Networks. Wen et al. NeurIPS 2016
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper conducts a series of interesting experiments on compressing BERT and makes several conclusions. The compression technique is magnitude weight pruning based on an existing work. The paper mainly tested different compression rates and the stages when the compression can be applied. Compared to the existing work, one main contribution of the paper is to show that the BERT model can be pruned prior to fine-tuning any specific downstream tasks by 30%-40% without affecting all tested downstream tasks much. The paper is well motivated and presents interesting experimental results and conclusions. I have some concerns on their experiment details, which needs some clarification.

1. the observation in 3.4 is a little counter-intuitive to me. The model has all pre-trained weights and should be able to determine, during fine-tuning, which weights to decrease to nearly zero or to abandon. However, the experimental results show that the pruning at that point produces a worse dev accuracy. For the experiments, 3 epochs is used for fine-tuning and then the pruning is applied. I was wondering what happen if you first fine-tune the model to get the best dev accuracy and prune the weights at that point. How did you choose the number 3? I am guessing that the pruning in the middle of fine-tuning process may throw away useful information too early.   
2. It will be helpful to show the thresholds of pruning and how these thresholds relate to the training loss and accuracy. I think the value of the thresholds can tell whether some pruning ratios are reasonable.
3. when the authors continue training the model, for example in 3.4, the training stops when the training losses are comparable. Why did the training loss is used as the metric instead of the dev accuracy? Figure 1 right seems to show that those models are overfitting. 
review_title: Official Blind Review #1

Meta-review: This work explores weight pruning for BERT in three broad regimes of transfer learning: low, medium and high.

Overall, the paper is well written and explained and the goal of efficient training and inference is meaningful. Reviewers have major concerns about this work is its technical innovation and value to the community: a reuse of pruning to BERT is not new in technical perspective, the marginal improvement in pruning ratio compared to other compression method for BERT, and the introduced sparsity that hinders efficient computation for modern hardware such as GPU. The rebuttal failed to answer a majority of these important concerns.

Hence I recommend rejection.
Decision: Reject