# Autocorrelation Matrix Knowledge Distillation: A Task-Specific Distillation Method for BERT Models
## 1 Introduction  
Pre-trained language models based on Transformer architecture have consistently achieved exceptional performance across a wide range of natural language processing (NLP) tasks in recent years. Examples of these models include GPT [1], BERT [2], ET-BERT [3], BPCN [4], and PAL-BERT [5]. Despite excellent performance, these models are limited in their applicability to edge devices due to their large parameter count and high computational demands [6, 7]. Consequently, reducing computational costs and model size while maintaining original performance has become a vital area of research [8, 9]. Pre-trained language models perform well in many tasks, but in specific subjects or tasks further training and fine-tuning are usually required to improve performance, often consuming significant computing resources and time. To address this, researchers have developed various model compression and acceleration techniques for efficient deployment on edge devices, including pruning [10], quantization [11], knowledge distillation (KD) [12], and low-rank decomposition [13]. Among these, KD significantly reduces computing time and resource consumption by transferring the knowledge of a complex teacher model to a simplified student model. Although the student model may perform slightly worse than the teacher model, it can still maintain sufficient accuracy in resource-limited environments [14]. The key to efficient deployment is balancing computing requirements and model performance [15, 16]. Task-specific knowledge distillation has shown significant advantages in improving model performance [17; 18]. [PERSON] successfully compressed the BERT model by distilling task-specific knowledge into a lightweight BiLSTM model [19]. BERT-PKD guides the training of student models by extracting knowledge from the intermediate layers of BERT [20]. In addition, the researchers enabled the student model to more comprehensively capture the abstract knowledge of the teacher model by matching the internal representation of BERT [21]. DynaBERT achieves dynamic compression of the BERT model by adaptively adjusting the model width and depth [22]. BERT of Theseus reduces model complexity by gradually replacing BERT layers [23]. TinyBERT employs a two-stage distillation method to significantly reduce model size and accelerate inference [24]. However, these methods have certain limitations. Previous methods primarily focused on learning the features of the teacher model, while overlooking the interrelationships between these features, which are crucial in many complex natural language processing tasks [25]. In addition, when calculating the hidden states loss, the hidden states dimension of the student model must match that of the teacher model, otherwise additional calculations are required to align the dimensions of the two, which limits the flexibility of the student model structure. The main research question of this paper is how to enable the student model to more effectively capture and learn the characteristics of the teacher model during the knowledge distillation process, while fundamentally solving the hidden states dimension mismatch between the teacher and student models. The goal is to significantly reduce the model parameters while maintaining high performance. We propose the Autocorrelation Matrix Knowledge Distillation (AMKD) approach to address the limitations of traditional KD methods.Unlike existing techniques such as BERT-PKD, DistilBERT, or TinyBERT, AMKD excels at capturing complex feature interactions while directly resolving hidden states dimension mismatches without additional projection layers. AMKD not only allows the student model to learn the features of the teacher model but also captures the complex relationships among these features. The relationships between features play a critical role in NLP tasks.