Question: What are the two strategies they propose for the distillation process?
Evidence: Language model pre-training has proven to be highly effective in learning universal language representations from large-scale unlabeled data. ELMo BIBREF0, GPT BIBREF1 and BERT BIBREF2 have achieved great success in many NLP tasks, such as sentiment classification BIBREF3, natural language inference BIBREF4, and question answering BIBREF5.
Despite its empirical success, BERT's computational efficiency is a widely recognized issue because of its large number of parameters. For example, the original BERT-Base model has 12 layers and 110 million parameters. Training from scratch typically takes four days on 4 to 16 Cloud TPUs. Even fine-tuning the pre-trained model with task-specific dataset may take several hours to finish one epoch. Thus, reducing computational costs for such models is crucial for their application in practice, where computational resources are limited.
Motivated by this, we investigate the redundancy issue of learned parameters in large-scale pre-trained models, and propose a new model compression approach, Patient Knowledge Distillation (Patient-KD), to compress original teacher (e.g., BERT) into a lightweight student model without performance sacrifice. In our approach, the teacher model outputs probability logits and predicts labels for the training samples (extendable to additional unannotated samples), and the student model learns from the teacher network to mimic the teacher's prediction.
Different from previous knowledge distillation methods BIBREF6, BIBREF7, BIBREF8, we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from previous layers of the teacher network. We call this `Patient Knowledge Distillation'. This patient learner has the advantage of distilling rich information through the deep structure of the teacher network for multi-layer knowledge distillation.
We also propose two different strategies for the distillation process: ($i$) PKD-Last: the student learns from the last $k$ layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and ($ii$) PKD-Skip: the student learns from every $k$ layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.
We evaluate the proposed approach on several NLP tasks, including Sentiment Classification, Paraphrase Similarity Matching, Natural Language Inference, and Machine Reading Comprehension. Experiments on seven datasets across these four tasks demonstrate that the proposed Patient-KD approach achieves superior performance and better generalization than standard knowledge distillation methods BIBREF6, with significant gain in training efficiency and storage reduction while maintaining comparable model accuracy to original large models. To the authors' best knowledge, this is the first known effort for BERT model compression.
We also propose two different strategies for the distillation process: ($i$) PKD-Last: the student learns from the last $k$ layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and ($ii$) PKD-Skip: the student learns from every $k$ layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.
Answer: ($i$) PKD-Last: the student learns from the last $k$ layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and ($i$) PKD-Skip: the student learns from every $k$ layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.