Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper sets to solve the problem that many language models such as GPT-2 despite having achieved great success in their language tasks are too big to deploy in applications. They propose LightPAFF, a framework that allows to transfer knowledge from a big teacher model to a lightweight student model, thus solving the deployability issue at hand. They conduct experiments showing LightPAFF achieves a 5x reduction in the number of parameters and a 5x-7x improvement on the inference speed, while roughly preserving performance. 

The distillation framework the authors use in their methods is not new. It has been proposed in previous work, as the authors noted. As opposed to previous works, where distillation is performed in the fine tuning stage, the authors of LightPAFF propose a two stage distillation procedure instead that performs distillation at the pre training phase and fine tunes this distilled model for use in a downstream task using a big fine tuned teacher model and the dataset of the task. The experimental results show these results to be meaningful. They achieve better accuracy and similar compression than previous approaches. It's main weakness is that the method doesn't seem to be particularly new. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
Summary: This work leverages knowledge distillation both in pre-training and fine-tuning stages to learn a more compact student model that approximates the performance of a teacher model. Extensive experiments with different knowledge distillation loss functions are conducted on a number of representative language representation models including BERT, GPT-2 and MASS to show the benefit of the method. 

Strengths:

The numerical results show some promise of the proposed method to achieve competitive performance to the teacher network and outperform the considered baselines. The evaluated 9 downstream tasks cover a wide range of language understanding and generation problems and they do prove the effectiveness of knowledge distillation in boosting the accuracy.

Weaknesses:

My main concern is about the limited novelty of approach. The knowledge distillation method developed in the paper seems not particularly novel in principle. Although extensive numerical results are provided to show the effectiveness of knowledge distillation, the value-added beyond applying different distillation loss functions in different NLP tasks is still insufficient. Particularly, the loss function that combines maximal likelihood loss with KL divergence looks similar to those used in the paper [1]. The connection and difference to that related work need to be clarified. 

[1] Yu, Dong, et al. "KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition." 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013.

=== update after author response ===

Thank you for the response. Although interesting and showing some promise in a bunch of applications, I am still not convinced that the proposed approach is novel enough in principle.  I thus stick to my assessment of this paper as borderline leaning to rejection.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors propose a method for distilling language models such as BERT into much smaller versions with similar accuracy. The approach, therefore, saves both memory and computation resources. 

The difference to existing methods is that the authors method also performs distillation during pretraining. The authors apply their method to BERT and run experiments, comparing the distilled BERT model with the original one as well as existing distillation methods.

The experiments are extensive with impressive results. I'm torn. On the one hand, these results show that this method (which is simple and straight-forward) reduces memory and computation time (latency) substantially while maintaining similar accuracy. It can be very useful in practice. On the other hand, the idea is so simple and straight-forward, that I'm not sure ICLR (a top machine learning conference) is the right place for it. I tend to think that the paper should get accepted though since the empirical work is really exhaustive and impressive. 
review_title: Official Blind Review #1

Meta-review: This paper proposes a two-stage distillation from pretrained language models, where the knowledge distillation happens in both the pre-training and the fine-tune stages.  Experiments show improvement on BERT, GPT and MASS.  All reviewers pointed that the novelty of the work is very limited.
Decision: Reject