Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: What is the task?
Knowledge distillation of BERT

What has been done before?
Unlike prior works such as Distilled BiLSTMSOFT (Tang et al., 2019), BERT-PKD (Sun et al., 2019) and DistilBERT, this work

i) Do knowledge distillation at pre training stage also in addition to fine tuning stage.
ii) Student learns from all - embedding layers, attention matrices, hidden states, and final prediction layers. 

In BERT-PKD, student learns from the [CLS]  hidden states of the teacher.

What are the main contributions of the paper?
Novel Transformer distillation method that is specially designed for knowledge distillation of the Transformer-based models.
Novel two-stage learning framework which performs Transformer distillation at both the pre-training and task-specific learning stages
Resulting TinyBERT being 7.5x smaller and 9.4x faster on inference and significantly outperforms other state-of-the-art baselines on BERT distillation.

What are the key techniques used to tackle this task?
Novel Transformer distillation method that is specially designed for knowledge distillation of the Transformer-based models.
Novel two-stage learning framework which performs Transformer distillation at both the pre-training and task-specific learning stages

What are the main results? Are they significant?
Resulting TinyBERT being 7.5x smaller and 9.4x faster on inference and significantly outperforms other state-of-the-art baselines on BERT distillation with only ∼28% parameters and ∼31% inference time of them.

Results show that three key procedures: TD (Task-specific Distillation), GD (General Distillation) and DA (Data Augmentation) are crucial for the proposed KD method.

Proposed distillation objectives - Transformer-layer distillation (attention matrices and hidden states), embedding-layer distillation and prediction layer distillation  are crucial for the proposed KD method.

Weaknesses
experimental results were not easily comparable to prior work so it is hard to say if claims are well-supported experimental results

Questions
Did authors try other values of lambda

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes a new knowledge distillation method for BERT models. A number of modifications to the vanilla knowledge distillation method of Hinton et al (2015) are proposed. First, authors suggest adding L2 loss functions between alignment matrices, embedding layer values and prediction layer values. Second, authors propose run knowledge-distillation twice, once with the original pre-trained BERT model as teacher, and then again with task specific fine-tuned BERT as a new teacher. Third, authors emphasize the use of data augmentation for successful knowledge distillation. In Table 2, authors claim a significant lift across GLUE benchmarks with respect to other baseline methods with comparable model size.

While the main contribution of this paper is the proposal of empirically useful techniques than theoretical development, the empirical results reported in this paper are somewhat puzzling. 

First of all, GLUE benchmark scores reported in Table 2 don't seem to be consistent with Table 1 of Sun et al (2019) for BERT-PKD ( https://arxiv.org/pdf/1908.09355.pdf ) or DistilBERT ( https://medium.com/huggingface/distilbert-8cf3380435b5 ). Indeed, BERT-PKD in Sun et al seems to significantly outperform TinyBERT on QNLI (89.0 vs 87.7) and RTE (65.5 vs 62.9), and the gap between BERT-PKD and TinyBERT on other tasks are much smaller if we take numbers reported in the original paper.

In Table 6, ablation studies with different distillation objectives are reported. Quite surprisingly, without Transformer-layer distillation (No Trm) the performance drops quite significantly. This is unexpected, because baselines such as Sun et al and DistilBERT do not use the Transformer-layer distillation but much more competitive to full TinyBERT than TinyBERT without Transformer-layer distillation. Would there be a reason why TinyBERT is so critically dependent on Transformer-layer distillation? Similarly, the removal of data augmentation (Table 5, No DA) is so detrimental to the performance of the model that it makes me to suspect whether the most of gain is from successful data augmentation. Indeed, 'No DA' row of Table 5 is very close to the performance BERT-PKD in Table 4, although the number of layers is different (4 vs 6). 

In order for the research community to understand the contribution of proposed techniques more thoroughly, I suggest authors to conduct ablation studies with the simplest baseline. That is, rather than starting with the full TinyBERT model, start with a simple but competitive baseline like BERT-PKD, and only add one technique (DA, GD, Transformer-layer distillation) at a time so that readers shall understand what technique is the most important to be added to the baseline, and also whether some of the proposed techniques should always be used in combination.

--- 
After Author Rebuttal: authors have addressed all of my concerns quite clearly. Additional experiments which targeted a specific design choice at a time made me much more convinced that the techniques proposed in this paper are useful not only for this particular context but also more broadly applicable.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose TinyBERT, a smaller version of BERT that is trained with knowledge distillation. The authors evaluate on the GLUE benchmark.

Overall, I find the direction of this work exciting and making these large models smaller for practical use is an important research area. The authors provide various ablation experiments that provide insight into their method. The main contribution is experiments comparing various existing distillation methods to different parts of the model (embeddings, layers, prediction layer), so is not particularly novel in contributing new techniques for distillation. That being said, there is importance in contributing these results as they are very useful for others working in the area and on making smaller models. But I would expect the authors to be much more detailed in their experimental description and make it clear in the paper that the comparative baselines are fair and well tuned. 

Comments:

1. Can the authors please add details for how the model has been trained, such as the datasets used, the number of update steps, the batch size, etc. as well as the finetuning parameters that were cross validated for GLUE? It is difficult to tell in the current setting if the models are comparable to the baselines. The current paper doesn't seem like it could be reproduced. It is particularly important to detail how the finetuning was done, as this is very important for the smaller datasets in GLUE.

2. Is the learning of the distilled model only done on the training dataset, or there is data augmentation beyond the training set? What is the effect without data augmentation?

3. Unfortunately, the performance drop on the GLUE benchmark as shown in Table 2 is fairly large. The authors compare to BERT Small and DistilBERT and I like the baselines, but the claim that the model achieves comparable performance to BERT Base is not true. 

4. Was the BERT Small model tuned, or the same learning parameters from BERT Base were used? 

5. Can the authors clarify the inference time of BERT Small? The speed improvement of TinyBERT should be the same as BERT Small based on parameter size.

6. The authors experiment with distilling the embedding layer to reduce the number of parameters, why not reduce the parameter size by reducing the vocabulary size? Existing approaches to BERT training use BPE with ~30k vocabulary size or RoBERTa with ~50k vocabulary size, but large gains could be applied here by reducing the size or using softmax reduction techniques that were popular on full vocabulary language modeling datasets like wikitext-103 or billion word. 

7. Can the authors please clarify the construction of Table 2? Are those results on the test set (e.g. evaluated on the official GLUE benchmark), or on the dev set? Where are the DistilBERT numbers on the test set coming from, as it is not reported in their paper? 
review_title: Official Blind Review #3

Meta-review: This paper proposes a new distillation-based method for using large pretrained models like BERT to produce much *smaller* fine-tuned target-task models. 

This paper is low-borderline: It has merit and meets our basic standards, but owing to capacity limitations we had to give preference to papers we see as having a higher potential impact. Reviewers had some concerns about experimental design, but those seem to have been fully resolved after discussion. Reviewers were not convinced, even after some discussion, that the method and results were sufficiently novel and effective to have a substantial impact on the state of practice in this area.
Decision: Reject