Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper try to distill large pretrained language models into a bidirectional CMOW/CBOW-Hybrid model. The proposed architecture was designed to output per-token representations, so as to perform knowledge distillation on the masked language modeling pretraining task. A two-sequence encoding scheme (DiffCat) was designed for the downstream task on sentence pairs. Empirical results show that the proposed method can achieve comparable results to ELMo while using only half of the parameters and providing three times faster inference speed.
main_review: Strengths:
1. The proposed idea is novel: the CMOW model can only be used in sentence-level tasks. The authors extend CMOW using the bidirectional variant to extract token-level representations, so now it can be used for the masked language modeling training objective.
2. DiffCat is designed specifically for two-sequence tasks. It improves the performance by 32% compared to naive joint encoding.
3. The proposed method can achieve comparable results to ELMo, while having a much higher encoding speed.

Weaknesses:
1. The evaluation of the proposed method is focused on the GLUE task, which is a collection of classification tasks. I encourage the authors to test the model on other tasks as well. For example, DistilBERT can also perform well on question answering datasets like SQuAD. It's interesting to see whether this shallow model can perform appropriately on more complex tasks.
2. How can we trade-off between this fast inference speed and good performance? It seems that this model is very fast at inference speed, but the performance was compromised. It would be better if we are allowed to find a balance between speed and performance according to our demand. For example, TinyBERT provides a 4-layer version and a 6-layer version, so we can find a balance between the inference time and performance. In this paper, it seems that the author only provides 20x20 matrix embeddings for the CMOW model.
summary_of_the_review: The paper presents a novel and useful idea: distill a complex pretraining model into a simple CMOW model. The initial result is promising while some limitations can be further explored. I think this paper is worth accepting because it can motivate the research on smaller distilled PreLMs.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a knowledge distillation method for BERT, where BERT-base acts as a teacher network and  CMOW is used as a student network. To get a relatively strong student network, the authors extend the vanilla CMOW to bidirectional CMOW. The authors also adopt a two-stage distillation method consisting of general distillation and task-specific distillation to further improve the performances. Finally, the authors evaluate the proposed method on GLUE dataset.     
main_review: The main contribution of this work mainly lies in the bidirectional CMOW and a two-stage knowledge distillation method. However, the technical contribution in bidirectional CMOW is quite limited, the related extension has been widely used in existing works, like bidirectional LSTM. Furthermore, the two-stage knowledge distillation method was originally proposed in TinyBERT, a well-known work on BERT distillation, which is directly used in this work. The proposed method has poor performances on several GLUE tasks, e.g., CoLA and MRPC, and many related baselines are not included, e.g. TinyBERT, BERT-PKD, MiniLM. The proposed method even does not have advantages on #parameters and inference time with comparison to 4-layer TinyBERT.      
summary_of_the_review: The bidirectional extension of CMOW is straightforward, and the two-stage knowledge distillation method is originally  proposed by TinyBERT. The final performances of the proposed method performs worse than most baselines, e.g., TinyBERT, DistilBERT, MiniLM, BERT-PKD.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to distill a BERT model to a smaller one,  a CMOW/CBOW-Hybrid. Its performance is better than ELMo, but worse than the typical distillation method using transformer-based students like DistillBERT. The authors claim that CMOW/CBOW-Hybrid is much faster. The paper is clearly written.
main_review: I am curious how it performs with a single CBOW or a single CMOW, not a hybrid one. 

Did you try other sizes of matrix embeddings of size? How does it perform? Should the CBOW and CMOW embeddings have the same dimension size? What are their individual functions for CBOW and CMOW? Are they complementary or just redundant? I know CMOW is from another paper, but as a reader, we might want to read a self-contained paper.


summary_of_the_review: 
To distill a better model to a weaker model, one can try different student models, either using shallower/narrower Transformer-based models or other models. If we could tolerate a big performance drop (as in this paper, the authors were satisfied that it exceeds ELMo), there are much more choices, of which CMOW/CBOW-Hybrid might be a good one. My concern is that using CMOW/CBOW-Hybrid is not that interesting, one might also try CBOW with convolutions, for example. It may help if the authors could provide more insights to choose CMOW/CBOW-Hybrid,  or why the investigation of  CMOW/CBOW-Hybrid is important, as we might have so many lightweight models as the student model.
review_title: null

Meta-review: This paper presents a method for distilling pretrained models (such as BERT) into a different student architecture (CMOW), and extend the CMOW architecture with a bidirectional component.  On a couple of datasets, results are comparable to DistilBERT a previous baseline. This paper is nice, but can be stronger with more empirical experiments on non-GLUE tasks (TriviaQA, Natural Questions, SQUAD for example).  Furthermore, I agree with Reviewer M3tk that there are many empirical comparisons with baselines such as TinyBERT missing and the argument of not needing the teacher model to be super convincing.
Decision: Reject