# Autocorrelation Matrix Knowledge Distillation: A Task-Specific Distillation Method for BERT Models
## 5 Conclusions  
BERT\({}_{\text{TINY}}\)-AMKD significantly outperforms traditional distillation methods and other four-layer baseline models. In the GLUE task, BERT\({}_{\text{TINY}}\)-AMKD achieves an average score of 83.6%, which is 4.1% higher than BERT\({}_{\text{TINY}}\)-KD, 2.6% higher than BERT\({}_{\text{T}}\)-PKD, and 3.9% higher than DistilBERT\({}_{4}\). In the SQuAD benchmark test, the EM and F1 scores of BERT\({}_{\text{TINY}}\)-AMKD in SQuAD v1.1 increased to 72.1% and 81.8%, respectively, also surpassing other four-layer baseline models. Ablation experiments show that attention matrix distillation and hidden states distillation are crucial to performance improvement, \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **MNLI-m** & **MNLI-mm** & **SST-2** & **MRPC** & **Avg** \\ \hline BERT\({}_{\text{TINY}}\)-AMKD & 82.1 & 81.7 & 92.1 & 86.9 & 85.7 \\ \hline w/o Embed & 81.4 & 81.3 & 91.2 & 86.2 & 85.0 \\ w/o Atten & 79.3 & 78.7 & 89.5 & 83.6 & 82.8 \\ w/o Hidden & 79.5 & 78.9 & 91.0 & 84.2 & 83.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Ablation studies on distillation components in AMKD learning. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **SQuAD v1.1** & **SQuAD v2.0** \\  & **EM** & **F1** & **EM** & **F1** \\ \hline BERT\({}_{\text{BASE}}\) (Teacher) & 79.8 & 87.9 & 73.2 & 76.4 \\ [PERSON]\({}_{\text{4-PKD}}\) & 68.7 & 78.1 & 59.6 & 63.9 \\ [PERSON]\({}_{\text{4}}\) & 70.2 & 79.8 & 59.4 & 63.2 \\ [PERSON]\({}_{\text{TINY}}\)-AMKD & **72.1** & **81.8** & **65.2** & **68.4** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comprehensive evaluation results of baseline models and BERT\({}_{\text{TINY}}\)-AMKD on SQuAD. Evaluation metrics: EM (Exact Match) and F1 (F1 Score). \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **MNLI-m** & **MNLI-mm** & **QQP** & **QNLI** & **SST-2** & **MRPC** & **RTE** & **Avg** \\ \hline BERT\({}_{\text{TINY}}\) (AMKD-Last) & 81.1 & 80.5 & 88.2 & 86.3 & 91.8 & **87.0** & 65.0 & 82.8 \\ BERT\({}_{\text{TINY}}\) (AMKD-Skip) & **82.1** & **81.7** & **89.4** & **87.9** & **92.1** & 86.9 & **65.4** & **83.6** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance comparison between AMKD-Last and AMKD-Skip on the GLUE benchmark. while the layer-skipping distillation strategy can more comprehensively obtain different levels of knowledge from the teacher model. These experimental results demonstrate that the AMKD method can significantly improve the accuracy of small models in natural language processing tasks. By introducing the autocorrelation matrix, AMKD not only effectively captures the complex relationships between the features of the teacher model, but also skillfully handles the dimensional differences between the student model and the teacher model, reducing information loss. This method is suitable not only for classification tasks but also for a range of applications, including reading comprehension and regression tasks. AMKD significantly improves the performance of small BERT models in NLP tasks, providing an efficient and flexible solution for deploying pre-trained language models in resource-constrained environments. The AMKD method we proposed is mainly used for the distillation of specific tasks. If conditions permit, distillation can be performed in the pre-training stage in the future to generate a general small BERT model suitable for a wider range of tasks. In addition, it is difficult to adapt the fixed temperature parameter \(t\) in the distillation process to different samples. In the future, it will be possible to consider dynamically adjusting the temperature to better adapt to the needs of different samples. Conceptualization, K.Z. ; methodology, J.L. ; software, J.L. ; validation, J.L. and B.W. ; formal analysis, J.L. ; investigation, H.M.; resources, K.Z. ; data curation, B.W. ; writing--original draft preparation, J.L. ; writing--review and editing, K.Z. ; visualization, B.W. ; supervision, K.Z. ; project administration, K.Z. ; funding acquisition, K.Z.