## The Proposed Model  
### Multi-Head Self-Attention Mechanism  
Scale dot product attention is the most important self-attention mechanism that was studied in this paper. As shown in Figure 2, it involves three main parameters: Q (Query), K (Key), and V (Value). Each of the three parameters performs linear conversion and is represented by a vector matrix. After calculating the dot product of Q and K, we can attain the attention weight of the input sequence, which is, then, multiplied by V to obtain the final attention context vector.  
\[\text{Attention}(\text{Q},\text{ K},\text{ V})=\text{softmax}\Bigg{(}\frac{ \text{QK}^{\text{T}}}{\sqrt{d_{k}}}\Bigg{)}\text{V} \tag{1}\]  
As shown in Equation (1), the transposed matrices of K and Q are multiplied to calculate the similarity between each other, and the obtained result is the attention weight matrix of the current input sequence. The importance of the sequence information can be learned through this method. Then, this result is divided by sqrt (\(d_{k}\)) to prevent the result of SoftMax from being only 1 and 0. Finally, it is multiplied by V to obtain the corpus attention matrix of the last calculated sequence value.  
The multi-head self-attention mechanism divides the dimension of the sub-word embedding equally and then calculates it by the scale dot product attention. Considering that the Head number H = 8 and the model dimension \(d_{model}\) = 512 are the hyperparameters of the X-Transformer model set in this study, there is a 64-dimensional vector (512/8 = 64) in each head. The eight heads are, then, combined after the attention matrix is obtained. The reason behind separating the model dimension to calculate the self-attention mechanism is that the model in the multi-head mode can learn more easily the key point on which it must be focused. The eight heads can bring attention to at least eight tokens to let the  
Figure 2: Scale dot product attention.  
training of the NMT model be performed more precisely [4]. Both the encoder and decoder of the X-Transformer use double multi-head self-attention mechanisms.  
In the encoder, the parameters Q, K, and V of the first self-attention mechanism were used as inputs of the encoder after the sub-word embedding process. The residual result of the first self-attention mechanism and the input of the first self-attention mechanism were added and normalized, which were the parameters Q, K, and V of the second self-attention mechanism.  
In the decoder, the parameters Q, K, and V of the first self-attention mechanism were used as the input of the decoder after the sub-word embedding process and the correct answer (i.e., ground truth). More specifically, in order to prevent the NMT model from knowing the correct answer of the future training sequence in advance, an additional look-ahead mask was added.  
The look-ahead mask was implemented by multiplying a negative infinity value that is, assuming that the input sequence of the decoder is \(d,\ d+1,\ d+2,\ \ldots,\ d+n\). Additionally, in training step \(t\), except for input sequence \(d\), the other sequence \(d+1,\ d+2,\ \ldots,\ d+n\) was multiplied by the negative infinity value. Then, in training step \(t+1\), except for input sequence \(d,\ d+1\), the other sequence \(d+2,\ \ldots,\ d+n\) was multiplied by the negative infinity value, too. This took place until training step \(t+n\), which was the last training step, when the look-ahead mask ended.  
The second self-attention mechanism in the decoder calculated the similarity between the output of the first self-attention mechanism in the decoder and the output of the encoder. The parameter Q was added up and normalized the residual result of the first self-attention mechanism and the input of the first self-attention mechanism in the decoder. Additionally, the parameters K and V were the output of encoder.