# Autocorrelation Matrix Knowledge Distillation: A Task-Specific Distillation Method for BERT Models
## 2 Preliminaries
### Transformer Architecture  
The Transformer architecture, introduced by [PERSON] et al. in 2017 [26], is a widely adopted deep learning model, particularly in natural language processing and machine translation. Unlike traditional models such as RNN [27] and CNN [28], the Transformercaptures long-range dependencies through self-attention. The model consists of an encoder, which converts input sequences into hidden representations, and a decoder, which generates outputs from these representations. The key components include multi-head attention, feed-forward networks [29], residual connections [30], and layer normalization [31]. **Multi-Head Attention Mechanism. ** The multi-head attention mechanism is a fundamental component of the Transformer architecture, enabling the model to attend to multiple segments of the input sequence concurrently. The computation of attention involves several steps, starting with the calculation of scaled dot-product attention. Given a query matrix \(\mathbf{Q}\), a key matrix \(\mathbf{K}\), and a value matrix \(\mathbf{V}\), the attention scores are derived using the following equation:  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\Bigg{(}\frac{Q\mathbf{K}^ {\mathsf{T}}}{\sqrt{d_{k}}}\Bigg{)}\mathbf{V}, \tag{1}\]  
where \(\mathbf{Q}\mathbf{K}^{\mathsf{T}}\) denotes the dot product between the query and key matrices, reflecting their similarity, and \(d_{k}\) represents the dimensionality of the key, which is used to scale the dot product to prevent gradient instability. The softmax function is subsequently applied to transform these similarities into weights, which are then multiplied by the value matrix \(\mathbf{V}\) to yield the weighted output. In the multi-head attention mechanism, the query, key, and value matrices \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) are each projected into \(h\) different subspaces through separate linear projections. These subspaces allow the model to perform attention computations in parallel across multiple different representation spaces. Specifically, for each head \(i\), the input matrices are transformed into the subspaces using different projection matrices \(\mathbf{W}_{i}^{Q}\), \(\mathbf{W}_{i}^{K}\), and \(\mathbf{W}_{i}^{V}\), and the attention computation is performed as follows:  
\[\text{head}_{i}=\text{Attention}(\mathbf{Q}\mathbf{W}_{i}^{Q},\mathbf{K}\mathbf{W}_{i}^{K}, \mathbf{V}\mathbf{W}_{i}^{V}), \tag{2}\]  
where \(\mathbf{W}_{i}^{Q}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}\), \(\mathbf{W}_{i}^{K}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}\), and \(\mathbf{W}_{i}^{V}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}\) are the linear transformation matrices that project the queries, keys, and values into different subspaces. The outputs of all the heads are concatenated and then processed through a linear transformation matrix \(\mathbf{W}^{O}\) to produce the final output:  
\[\text{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Concat}(\text{head}_{1},\dots, \text{head}_{h})\mathbf{W}^{O}, \tag{3}\]  
where \(\mathbf{W}^{O}\in\mathbb{R}^{hd_{z}\times d_{\text{model}}}\) is the linear transformation matrix that maps the concatenated multi-head outputs back to the original dimensionality. This process allows the multi-head attention mechanism to capture different aspects of the input sequence from multiple subspaces, enhancing the model's representational capability. **Feed-Forward Networks. ** In each encoder and decoder layer of the Transformer, there is also an independent feed-forward network applied to every position in the sequence. This network consists of two linear transformations with a ReLU activation function in between:  
\[\text{FFN}(\mathbf{x})=\text{max}(0,\mathbf{x}\mathbf{W}_{1}+\mathbf{b}_{1})\mathbf{W}_{2}+\mathbf{b} _{2} \tag{4}\]  
**Residual Connections and Layer Normalization.