## 4 Natural Language Interfaces
### Seq2 seq Paraphrase Model with BERT
#### 4.2.1 Self-Attention Network  
The attention mechanism is widely used in various tasks of natural language processing based on deep learning, and the self-attention mechanism has become a hot research topic of neural network attention. The calculation process of **attention** and **multi-head-attention** are shown in Figure 8a,b. The calculation formula of attention is shown below.  
\[Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{6}\]  
where \(Q\in R^{n\times d_{k}}\), \(K\in R^{m\times d_{k}}\), \(V\in R^{m\times d_{v}}\). Without considering the softmax layer, we can consider the attention layer as encoding a sequence \(Q\) of \(n\times d_{k}\) into a new sequence of \(n\times d_{v}\). Let us make the metric more detailed, \(Q\), \(K\), and \(V\) are shorthand for query, key, and value, respectively, where \(K\) and \(V\) is a one-to-one correspondence, and they are key-value relationships. The above equation calculates the similarity between \(Q\) and each \(V\) by calculating the inner product of \(Q\) and each \(K\) and performing softmax, and then weighting and summing to get a \(d_{v}\)-dimensional vector, where the factor \(\sqrt{d_{k}}\) plays a regulating role of not making the inner product too large.  
Regarding the multi-head attention mechanism, which is a refinement of the attention mechanism, the approach is to map \(Q\), \(K\), and \(V\) through the parameter matrix, and then in performing the attention calculation, the process is repeated h times, and finally, the calculation results are stitched together. The formula is as follows:  
Figure 7: BERT-based Seq2 Seq language model.  
\[head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}) \tag{7}\]  
Here \(WQ_{i}\in Rd_{k}\times d^{k}\), \(WK_{i}\in Rd_{k}\times d^{k}\), \(WV_{i}\in Rd_{k}\times d^{k}\) and then we have :  
\[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\text{head}_{2},\dots, \text{head}_{h}) \tag{8}\]  
Finally, a sequence \(n\times hd_{v}\) is obtained.  
Self-attention is a kind of \(attention\), as \(attention(X,X,X)\), \(X\) is also the input sequence; that is, the attention is calculated inside the sequence to find the connection inside the sequence. Its calculation formula is as follows:  
\[Y=\text{MultiHead}(X,X,X) \tag{9}\]  
In summary, it can be seen that the attention mechanism, compared to recurrent neural networks (RNN), requires global information to be obtained by step-by-step recursion and takes the direct way of computing global information. \(PositionEmbedding\), which numbers each position and then each number corresponds to a vector, introduces certain position information to each word by combining the position vector and the word vector; in this way, the attention mechanism can distinguish words in different positions. The formula for calculating the position vector is shown below.  
\[PE_{(pos,2i)}=\sin\Bigl{(}pos/10,000^{2i|d_{\text{mod}\,\text{d}}}\,\Bigr{)} \tag{10}\]  
\[PE_{(pos,2i+1)}=\cos\Bigl{(}pos/10,000^{2i|d_{\text{mod}\,\text{el}}}\,\Bigr{)} \tag{11}\]  
The location-embedding matrix is shown in Figure 9, and to clearly describe this variation, we define the maximum sentence length as 50 and the vector dimension as 50 dimensions and find that this period variation will become slower and slower as the vector dimension increases while generating texture-containing location information with the period of the location function from 2 \(\pi\) to 10,000*2 \(\pi\) and each location in the embedding dimension becomes a combination of \(\sin\) and \(\cos\) function values with different periods. Thus generating unique texture location information allows the model to learn the dependencies between locations and the temporal properties of natural language.  
Figure 8: (**a**) The attention mechanism. (**b**) The abstract computational flow of the multi-headed attention mechanism.