Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a contextual knowledge distillation approach by leveraging two types of contextual knowledge: word relations and layer transforming relation. Recent advancement in this area emphasizes the promising effect of this area in language modeling. 

The paper is well-written and well-structured. The experiment section shows a complete set of experiments including the baselines, benchmark and ablation study. The results are relatively incremental in comparison with TinyBert. Considering that the improvement has been relatively incremental, it would be helpful to compare the models with respect to FLOPs and speedup.
Novelty: It seems that the notion of structural knowledge distillation have been used previously by Wang et al [1]. It would be great if the authors clarify about their contribution. Also,    the related work section can be enriched by new publications such as PoWER-BERT [2], FastBert [3] and TextBrewer[4]
1)	Structure-Level Knowledge Distillation For Multilingual Sequence Labeling, ACL 2020 
2)	PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination , ICML 2020
3)	FastBERT: a Self-distilling BERT with Adaptive Inference Time , acl 2020
4)	TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing 


review_title: CKD Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new knowledge distillation (KD) method for distilling BERT. This area is pretty active given that deploying BERT based models is of keen interest to many industrial applications.

This paper proposes distillation via modeling "Word Relation and Layer Transforming Relation" which essentially aims to "capture the knowledge of relationships between word representations and LTR defines how each word representation changes
as it passes through the network layers". Not to mention *pairwise* and triple-wise relations are being modeled

My biggest question in the paper (which the doesn't paper address) is that this is bound to be expensive. Yet there is hardly any mention of training time (or time needed to cache these values from the teacher). 

This seems even worse when there is not much gain over existing baselines and can be attributed to simply noise/variance. 

Overall, I don't think this method will be impactful at all and it is probably not worth having over existing approaches. It is far too complex. Experiments on the GLUE benchmark alone is also not convincing. 

The authors can try other tasks and perhaps SuperGLUE to make their experiments more convincing. 

I think a runtime analysis could help to make the paper stronger to understand the differences. But I would like to make it clear that I want a honest, runtime analysis of how long this distillation would take or the practical considerations. How much more expensive is it to align and compute these values during training of the student. Please make this clear. 

I would also like to see a runtime analysis of the baselines as well.
review_title: Doesn't seem too impactful 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an interesting knowledge distillation method based on a newly defined contextual knowledge of transformer-based models. The proposed contextual knowledge models the pair-wise or triple-wise relations across BERT-based contextual representations, based on which the local structures between Teacher and Student models are encouraged to be well aligned.     

The main contribution of this work lies in the newly proposed two types of contextual knowledge: Word Relation and Layer Transforming Relation. By using this new contextual knowledge, the contextual representations of Teacher model can be well transferred to Student model. Compared with existing BERT compression methods, like MobileBERT/DistilBERT/TinyBERT, this CKD has the advantage of being directly applied on top of other pre-trained small BERT models without conducting time-consuming pre-training process. 

The authors evaluate their approach on GLUE datasets and compare it to other state-of-the-art models.

The paper is well-written and organized, the experiments are thorough. However, I have several concerns:

* This proposed KD method is designed for the distillation on downstream tasks, so the whole distillation process should be conducted for each task, while the task-agnostic KD method, like MobileBERT, can be directly used with fine-tuning, please identify this fact in the introduction part. It would be more interesting, if experiments can be conducted during the pre-training stage and further evaluated. 

* More experiments on challenging tasks like QA should be added. 

* In the Table 1, the performance of TinyBERT on MNLI-mm is 82.6, while in an old version of tinybert paper, (https://arxiv.org/pdf/1909.10351v4.pdf), in the Table 10, the corresponding value is 83.2. And on the official GLUE benchmark the TinyBERT has comparable performance as the proposed CKD(w/DA).

* In the section 5.2, the MobileBERT is further improved by the proposed CKD with self-distillation, that is the MobileBERT is used as its own teacher on the downstream tasks. This comparison is not that fair, since MobileBERT can also be improved by other self-distillation method.  

* In the section 5.3, “we observe that the BERTMINI trained with the CKD shows the higher average
score even though BERTMINI has fewer model parameters.” this comparison is unfair since BERTMINI has 6 layers and TinyBERT is a 4-layer model, and less number of model parameters does not always mean fast inference. 
review_title: Well written paper on KD for Transformer-based models, the experiments can be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes two new distillation objective, word relations and layer transforming relations. Word relations constrain the pairwise/triplet relations of embeddings at each layer to be closer to the teacher network. The layer transforming relations constrain that the pairwise/triplet relations of embeddings between different layers should match the teacher network. 

pros:

1. The methods in this paper consider the pairwise or higher order (i.e. triplet)  relations to constrain  the student embeddings, while previous methods usually consider embeddings separately. The methods shall provide more constrained information from the teacher network.

2. Comparison with previous methods, ablation study and other experiments like model sizes, etc demonstrate the effectiveness of the proposed method. In some cases, the student network even outperforms the teacher network (more explanation about this might be needed).

Cons (or questions): 

1. Why the angle-based method is adopted, instead of other methods (e.g. the maximum/average distance between the triplet)? Is there any experiments studying the effect of the choice of these functions?

2. Previous methods sometimes use a short network (fewer layers) or thin network (smaller hidden sizes). As a result, I am not sure whether the number of parameters are comparable when comparing to the baselines. Could the authors also show the number of parameters of previous methods and their own methods?

Typo:
, etc. (Devlin et al., 2018; Lan et al., 2019; Liu et al., 2019a; Raffel et al., 2019; Yang et al., 2019): citations should be put before the punctuation. 
review_title: A paper that proposes two new distillation objective based on word relations and layer transforming relations that outperform previous distillation methods.

Meta-review: This paper proposes a new method to perform knowledge distillation (KD) for transformer compression, where two types of contextual knowledge, namely, word relations and layer-transforming relations, are considered for KD. Both pair-wise and triple-wise relations are modeled. 

This paper receives two weak reject and two weak accept recommendations. On one hand, the reviewers appreciate that the authors have added more results into the paper to solve their concerns. On the other hand, several concerns still exist. (i) With regards to the compute-performance trade-off, the gains of the method does not seem too great. One reviewer feels that the authors tried to downplay the cost of their method too much. Though we care more about the inference time, the development time in practice should also not be underestimated. (ii) Compared with TinyBERT, the performance gain looks marginal on the GLUE benchmark (Table 1). (iii) It will make the paper more convincing if pre-training experiments can be performed. 

Overall, after reading the paper, the AC thinks that the novelty of the proposed method is somewhat limited. The AC is also hesitant about whether modeling word relations and layer-transforming relations simultaneously are needed. The choices for ablation study are also not totally clear. 

For example, in Figure 2, it is not clear why the authors choose SST-2 to plot the figure; in Table 5, it is unclear why SST-2, MRPC and QNLI are selected, but not others. When looking at Table 5, it is not totally convincing it is needed to model both WR and LTR, or it is needed to introduce both pair-wise and triple-wise relations. More careful ablation studies are needed. It also remains unclear what kind of word relations or layer-transforming relations are learned. 

In summary, this is a borderline paper, and the rebuttal unfortunately did not fully address the reviewers' main concerns. On balance, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject