Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new approach to prune weights that is designed keeping large scale pre-trained language representations like BERT. Such a method is desirable for deploying such models on devices with limited memory like phones etc. Experiments on Squad and Glue datasets show that a pruned version of the model maintains high accuracy for these tasks. 

Pros
1. Pretty high pruning ratios (80%) can be used for many datasets (except Squad). Its an encouraging result for low-memory requirement scenarios.

Weakness:
1. Modest technical contribution. The approach description also requires elaboration. Unclear what weights participate in the pruning objective. 
2. Figure 2 is difficult to understand. The paper says "The sparse attention pattern exhibits obvious structured
distribution.", but I do not know why that is desirable/useful. 
3. The t-SNE visualization appears perfunctory. What should I take away from this analysis?
4. The baseline approach NIP was derived from the IP approach of Han et al. (2015). Explanation for not using IP is that it does not converge to a "viable solution". This needs more elaboration.
5. Why not compare to teacher-student distillation approaches like DistilBERT? These approaches have the same motivation of compressing model size, though different approach than what the paper adopted. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: 
Models such as BERT are pretrained language models which provide significant improvement for different tasks, however they suffer from high huge size and complexity. This paper has proposed using proximal gradient descent to find sparse weights for BERT to reduce the number of parameters and make the model smaller. They concentrate on the drawbacks of the previous sparse-based approaches and claimed that they have convergence issues (they have provided some evidence in the appendix). therefore, they propose to use reweighed sparse method and optimise it using proximal gradient descent which provides a closed form solution for sparse constraint. 

ALthough proposing a minor novelty (reweighted sparse optimization ), they have provided interesting results for both pretrained structure and fine-tuning for several different tasks. they have also provided some visualisation for the weight matrices after sparsification.

Their results are notably stronger than simply adding the L1 regularizer to the optimisation method. 

The paper is well written and easy to follow with nearly comprehensive related work.

However, there are some drawbacks:

1. They have claimed that “ To the best of our knowledge, we are the first to apply reweighted l1 and proximal algorithm in the DNN weight pruning domain, and achieve effective weight pruning on BERT. ”, however proximal optimization has been used for DNN in works like “Combined Group and Exclusive Sparsity for Deep Neural Networks, 2017”.  
2. It should be explained clearly about all the matrices included in the sparsification steps, despite only saying “parameters of the model”.
3. More analysis is required on the results, specially the diagrams for fine-tuning over different datasets.
4. It is essential to compare the method with other related works for Bert and transformer compression, including quantisation-based, factorisation-based, pruning, knowledge distillation papers such as:
--Prato, Gabriele, Ella Charlaix, and Mehdi Rezagholizadeh. "Fully Quantized Transformer for Improved Translation." arXiv preprint arXiv:1910.10485 (2019).
--Tang, Raphael, et al. "Distilling Task-Specific Knowledge from BERT into Simple Neural Networks." arXiv preprint arXiv:1903.12136 (2019).
--Sanh, Victor, et al. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv preprint arXiv:1910.01108 (2019).
--Ziheng Wang, et al. "Structured Pruning of Large Language Models."

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper proposes a way to compress Bert by weight pruning with  L1 minimization and proximal method. This paper is one of the first works aiming at  Bert model compression.
The authors think the traditional pruning ways can not work well for Bert model, so they propose Reweighted Proximal Pruning and conduct experiments on two different datasets. According to their results, they successfully compress 88.4% of the original Bert large model and get a reasonable accuracy.

Strong points:
1. The authors propose a new method RPP for Bert model compression.
2. The authors design experiments to show their RPP can get a very good prune ratio with reasonable accuracy.

Weak points:
1. The authors should provide a detailed and rigorous explanation for the drawback of existing pruning methods.
2. In the experiments, the authors only compare RPP with self-designed method NIP instead of any existing pruning method.  The reason they said is “these methods do not converge to a viable solution''. It would be better if they are also compared and analyzed in detail.
3. In the CoLA and QNLI datasets of Bert_large experiments, RPP can get a higher accuracy even than the original Bert_large model without pruning? This is counter-intuitive.  
4. About the metrics, the authors use F1 score and accuracy, the standard metrics in the GLUE benchmark for different tasks, except for CoLA. It might make sense to also keep the metrics for CoLA consistent with GLUE benchmark for better comparison.
5. It is not clear what the authors want to express in Figure 2. The generation of the figure needs more explanation, and the results need to be better interpreted. 

review_title: Official Blind Review #3

Meta-review: This paper proposes a novel pruning method for use with transformer text encoding models like BERT, and show that it can dramatically reduce the number of non-zero weights in a trained model while only slightly harming performance.

This is one of the hardest cases in my pile. The topic is obviously timely and worthwhile. None of the reviewers was able to give a high-confidence assessment, but the reviews were all ultimately leaning positive. However, the reviewers didn't reach a clear consensus on the main strengths of the paper, even after some private discussion, and they raised many concerns. These concerns, taken together, make me doubt that the current paper represents a substantial, sound contribution to the model compression literature in NLP.

I'm voting to reject, on the basis of:

- Recurring concerns about missing strong baselines, which make it less clear that the new method is an ideal choice.
- Relatively weak motivations for the proposed method (pruning a pre-trained model before fine-tuning) in the proposed application domain (mobile devices).
- Recurring concerns about thin analysis.
Decision: Reject