Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to pre-train a student before training with a teacher, which is easy to understand. Although the authors provide extensive empirical studies, I do not think they can justify the claims in this paper. 


** Argument

One concern is that compared to other baselines such as "Patient knowledge distillation" [1], the proposed method is not consistently better. The authors argue that [1] is more sophisticated in that they distill task knowledge from intermediate teacher activations. However, the proposed method introduces other extra complexities, such as pre-training the student. I do not agree that the proposed method is less elaborate than previous methods. 


Although the investigation on influence of model size and the amount/quality of unlabeled data is interesting in itself, this does not help justify the usefulness of pre-training a student. I hypothesize that when considering the intermediate feature maps as additional training signals, randomly initialized students can catch up with pre-trained students. 

Furthermore, the mixed results shown in Table 3 do not justify the proposed method well enough. 

[1] Patient Knowledge Distillation for BERT Model Compression, https://arxiv.org/abs/1908.09355
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This submission revisits the student-teacher paradigm and shows through extensive experiments that pre-training a student directly on masked language modeling is better than distillation (from scratch). It also shows that the best is to combine both and distill from that pre-trained student model.

My rating is Weak Accept. I think the submission highlights a very useful observation about knowledge distillation that I imagine is overlooked by many researchers and practitioners. The decision of Weak as opposed to a Strong accept is because the submission does not introduce anything truly novel, but simply points out observations and offers a recommended training strategy. However, I do argue for its acceptance, because it does a thorough job and presents many interesting findings that can benefit the community.

Comparison with prior work:

The submission focuses on comparison with Sun et al. and Sanh. These comparisons are important, but not the most compelling part of the paper. Comparison with more prior work that show large benefits would make the paper even stronger.

Interesting experiments:

The paper presents many interesting experiments useful for anyone trying to develop a compressed model. First, it shows that distillation (from scratch) by itself may be overrated, since simply repeating the pre-training+fine-tuning procedure on the small model directly is effective. However, distillation remains relevant since it also shows that pre-training the student, then distilling against a teacher, is a potent combination. In the case when the transfer set is the same size as the pre-training set, it surprisingly still has some benefits. This is not experimentally explained, but I suspect there are optimization benefits that are hard to pin down exactly. The paper hypothesizes that the two methods learn different “linguistic aspects,” but I think it is a bit too speculative to put it in such terms.

The experiments are thorough, with many student sizes, transfer set sizes, transfer set/task set correlation, etc. It also compares against the truncation technique, where the student is initialized with a truncated version of the teacher. There are no error bars in the plots, but there are so many plots with clear trends, that this is not a big concern. I can’t think of any experiments that are obviously missing.

Misc:

- The introduction says that the pre-training+fine-tuning baseline has been overlooked. It would be great to point out papers that has actually overlooked this baseline. Including this in the results would be even better.
- During my first read-through, I got confused because I didn’t realize “pre-training” in most of the paper refers to “student pre-training” (as opposed to simply training the teacher). Making this a bit more explicit here and there can avoid this confusion.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors investigate the problem of training compact pre-trained language model via distillation. Their method consists of three steps: 
1. pre-train the compact model LM
2. distill the compact model LM with a larger model (teacher)
3. fine-tune the compact model on target task 

This idea is not significantly new since it is quite common to apply distillation to compress models, and the results are largely empirical. From Table 3 the results on test sets are better than previous works, but not by much. The authors spend quite a of space on ablation studies to investigate the contribution of different factors, and on cross-domain transfers. They do manage to show that using a teacher for distilling a compact student model does better than directly pre-training a compact model on the NLI* task in section 6.3. It would be better if they could show it for other tasks on the benchmark as well. 

Overall I think this work is somewhat incremental, and falls below the acceptance threshold. 

review_title: Official Blind Review #1

Meta-review:  Though the reviewers thought the ideas in this paper were interesting, they questioned the importance and magnitude of the contribution.  Though it is important to share empirical results, the reviewers were not sure that there was enough for this paper to be accepted.
Decision: Reject