Controlling the Difference between Teacher and Student Models for Language Model Compression
Authors: Renyuan Liu, Xuejie Zhang, Jin Wang, Xiaobing Zhou


Reviewer paper summaries: This paper focuses on addressing the challenge of controlling the difference between teacher and student models in the context of knowledge distillation for language model compression. The authors propose two novel methods, Feature Disentanglement Conditional Mutual Information Constraint (FD-CMIC) and Cross Model Mutual Information Constraints (CM-MIC), to enhance the knowledge distillation process and improve the performance of student models. The authors demonstrate the effectiveness of their methods using the GLUE dataset, showing improved results over baseline models and other methods.
Summary of strengths: - Novel Methodology: The paper introduces two innovative methods, FD-CMIC and CM-MIC, which address the critical issue of controlling the difference between teacher and student models in knowledge distillation. These methods are novel and could be useful for future research.
- Theoretical Analysis: The authors provide a theoretical analysis of knowledge distillation from an information theory perspective. This analysis forms a solid foundation for the proposed methods.
Summary of weaknesses: - Weak writing: The proposed methods, FD-CMIC and CM-MIC, exhibit complexity with numerous terms in their equations. However, several terms lack explanations in Section 4 (e.g., the meaning of Z^), rendering the paper extremely hard to follow. Additionally, there are many typos and grammar issues.
-  Insufficient implementation details: Section 5 lacks any mention of implementation details. Without referring to the Appendix, it is even unclear which models were used for the student model. 
- Computational Efficiency: The paper may not provide enough information on the computational efficiency of the proposed methods. Adding details on training time, resource requirements, and comparisons with baseline models could be advantageous.
Comments, suggestions and typos: Comments:

Explain Abbreviations: Please ensure that all abbreviations are defined the first time they appear in the text.

Typos:  
- [Line 153] m aking --> making
- [Line 161] missing “.”
- [Line 336] missing “.”
Overall assessment scores: 2.5