## 2 Theoretical Basis
### Transformer Encoder  
The Transformer encoder is composed of many transformer blocks, including two sublayers, MSA, and a positionwise fully connected feedforward network, as presented on the left side of Figure 1. The core operation of Transformer is a multihead self-attention mechanism, which has achieved better results than RNN or LSTM structures. In summary, attention aggregates information according to the similarity of input information given by the current query with differences, which is to perform a weighted average operation on all input. In the calculation of the self-attention mechanism, the actual operations are three vectors: query, key, and value, where key and value are paired. The self-attention mechanism performs certain linear operations on these three vectors and obtains the results considering the global information. The specific process of self-attention is shown in Figure 2, which corresponds with the scaled dot-product attention presented on the right side of Figure 1.  
As shown in the figure, the input vector is a series of sequences in the form of \([x^{1},\ x^{2},\ x^{3},\ \dots,\ x^{t}]\). Each input vector multiplies a weight parameter matrix \(W\) to obtain a string of embedding vectors \(\left[a^{1},\ a^{2},\ \dots,\ a^{t}\right]\). Then, each is fed into the self-attention layer, which multiplies each embedding vector by three different transformation matrices \(W_{q},\ W_{k},\ W_{v}\) to obtain three corresponding vectors, namely, \(q^{i},\ k^{i},v^{i}\). Next, the self-attention mechanism performs vector point multiplication for \(q^{i}\) (query) and each \(k^{i}\) (key) to measure the similarity of any two vectors. The calculation process is called the scaled inner product (dot product), which is defined as follows:  
\[\alpha_{n,i}=q^{n}\cdot\frac{k^{i}}{\sqrt{d}} \tag{1}\]  
where \(\alpha_{n,\ i}\) is used to weigh the similarity between the \(i\)th key and the \(n\)th query, \(d\) is the dimension of \(q^{n}\) and \(k^{i}\), and \(\sqrt{d}\) is a scaling factor used to balance the dimensional changes resulted from vector dot product operations. The \(\alpha_{n,\ i}\) is then sent to obtain the weights \(\widehat{\alpha}_{n,\ i}\) on the values by applying the softmax function:  
\[\widehat{\alpha}_{n,i}=exp(\alpha_{n,i})/\sum_{j}exp(\alpha_{n,\ j}) \tag{2}\]  
where \(i\), \(j\in\{1,\ 2,\ \dots,\ n\}\). Then, vectors are constructed with global information by summing the product of the weight parameter \(\widehat{\alpha}_{n,\ i}\) with the corresponding \(v^{i}\). \[b^{n}=\sum_{i}\widehat{\alpha}_{n,\ i}v^{i} \tag{3}\]  
Figure 1: The structure of the Transformer encoder. Figure 2: The schematic calculation process of self-attention. where \(b^{n}\) is the reconstructed vector, which can be computed in parallel. In contrast with RNN, the self-attention layer can not only achieve the same results but also significantly improve the computational efficiency due to parallel operation. The current requirement for the input data type of Transformer is a tensor of shape \(\mathbb{R}^{n\times d}\), a sequence containing \(n\) vectors whose dimensions are \(d\). Therefore, the matrix of self-attention is described as:  
\[Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{4}\]  
where \(d_{k}\)= \(d\) is the dimension of the queries and keys. Equation (4) is an overview of the single-head self-attention mechanism, and its relationship with the MSA is shown in the middle of Figure 1.  
Since single-head self-attention can only focus on one aspect of the global information, it is beneficial to perform the attention function in parallel on each of the different, learned linear projections of queries, keys, and values. Meanwhile, the output of each subspace is connected and projected again to obtain the final value. This is exactly what the MSA does, which enables the model to further analyze the output of subspaces with different attention cores [39].