# Autocorrelation Matrix Knowledge Distillation: A Task-Specific Distillation Method for BERT Models
## 4 Experiments
### Experimental Results on GLUE  
The experimental results are presented in Table 3. BERTBASE (Teacher) represents our implementation of the BERT teacher model [2]. BERTTINY-FT shows the results of directly fine-tuning the pre-trained BERTTINY model on each task. BERTTINY-KD represents the results of applying the prediction distillation method to the BERTTINY model [14]. BERTTINY-AMKD is our proposed method. The results of TinyBERT4 are obtained by \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **\#Layer** & \begin{tabular}{c} **Hidden** \\ **Size** \\ \end{tabular} & \begin{tabular}{c} **Feed-** \\ **Forward** \\ \end{tabular} & \begin{tabular}{c} **Speedup** \\ \end{tabular} & \
\begin{tabular}{c} **\#Params** \\ \end{tabular} &
\begin{tabular}{c} **Relative** \\ \end{tabular} \\ \hline BERTBASE & 12 & 768 & 3072 & 1.0\(\times\) & 109M & 100\% \\ BERTTINY & 4 & 312 & 1200 & 9.4\(\times\) & 14.5M & 13.3\% \\ BERTSMALL & 4 & 512 & 2048 & 5.7\(\times\) & 29.2M & 26.8\% \\ BERT4-PKD & 4 & 768 & 3072 & 3.0\(\times\) & 52.2M & 47.9\% \\ DistilBERT4 & 4 & 768 & 3072 & 3.0\(\times\) & 52.2M & 47.9\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of architecture and parameter size between the BERT teacher model and student models. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & \begin{tabular}{c} **Teacher** \\ **Model** \\ \end{tabular} & \begin{tabular}{c} **Prediction** \\ **Layer** \\ \end{tabular} & \begin{tabular}{c} **Embedding** \\ **Layer** \\ \end{tabular} & \
\begin{tabular}{c} **Hidden** \\ **States** \\ \end{tabular} &
\begin{tabular}{c} **Attention** \\ **Matrix** \\ \end{tabular} \\ \hline BERT-KD & BERTBASE & ✓ & & & & \\ DistilBERT & BERTBASE & ✓ & & & & \\ BERT-PKD & BERTBASE & ✓ & & ✓ & & \\ BERT-AMKD & BERTBASE & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of distillation components in different approaches. directly calculating the MSE of the hidden states. Other baseline data are obtained by directly fine-tuning the publicly available pre-trained models [19; 20]. Accuracy is used as the evaluation metric for all tasks except for the MRPC task, which uses F1 score. The number of training samples for each dataset is provided below the dataset name. The results from the four-layer student models suggest that a substantial reduction in model size results in a significant performance gap between \(\text{BERT}_{\text{TINY}}\text{-FT}\) (or \(\text{BERT}_{\text{SMALL}}\)) and \(\text{BERT}_{\text{BASE}}\). \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) outperforms \(\text{BERT}_{\text{TINY}}\text{-FT}\) in all GLUE tasks, with an average improvement of 4.7%, indicating that the proposed AMKD method can significantly enhance the performance of small models in various downstream tasks. In addition, \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) achieved an average score of 83.6% in the GLUE task, which is significantly higher than the 79.5% of the traditional knowledge distillation method \(\text{BERT}_{\text{TINY}}\text{-KD}\). \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) achieves 96.3% of the performance of the \(\text{BERT}_{\text{BASE}}\) teacher model while using only 13.3% of the parameters. This demonstrates its ability to maintain high accuracy while significantly reducing computational resources. \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) outperforms the four-layer KD baseline models \(\text{BERT}_{\text{4}}\text{-PKD}\) and \(\text{DistilBERT}_{\text{4}}\) by 2.6% and 3.9%, respectively, while utilizing only 28% of their parameters. Compared to \(\text{TinyBERT}_{\text{4}}\), \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) has obvious advantages in performance and flexibility. \(\text{TinyBERT}_{\text{4}}\) uses MSE loss directly on hidden states, while AMKD not only learns individual features but also captures high-order dependencies through the autocorrelation matrix. This makes \(\text{BERT}_{\text{TINY}}\text{-AMKD}\) outperform \(\text{TinyBERT}_{\text{4}}\) on average in all GLUE tasks, an improvement of 1.7%.