# Multi-Source Neural Model for Machine Translation of Agglutinative Language
## 3 Materials and Methods  
### Standard NMT Model  
In this paper, we followed the NMT model proposed by [PERSON] et al. [12], which was implemented as a single-source Transformer model with an encoder-decoder framework as shown in Figure 1. We employed the basic model as our baseline. We will briefly summarize it in this section.  
Given the word sequence \(x=(x_{1},\dots,x_{m})\), both the input and output embedding layers map it into a word-embedding matrix \(e=(e_{1},\dots,e_{m})\), where \(e_{i}\) is computed by  
\[e_{i}=x_{i}\cdot E_{x}, \tag{1}\]  
where \(x_{i}\in\mathbb{R}^{1\times K_{x}}\) is the one-hot vector, \(E_{x}\in\mathbb{R}^{K_{x}\times d_{x}}\) is the word-embedding matrix, \(K_{x}\) is the vocabulary size and \(d_{x}\) is the word-embedding size. In order to make use of the word order in the sequence, Transformer provides the \"positional encoding\" function for the embedding layers, which uses sine and cosine functions of different frequencies by  
\[PE(pos,2i)=sin\Big{(}pos/10,000^{2i/d_{x}}\Big{)}, \tag{2}\]\[PE(pos,2i+1)=cos\Big{(}pos/10,000^{2i/dz}\Big{)},\]  
where _pos_ is the position of the word in the sequence and \(i\) is the dimension of the embedding matrix. The positional embedding matrix is the sum of the word-embedding matrix and the positional encoding matrix.  
The encoder is composed of a stack of \(N\) identical layers and each layer has two sub-layers consisting of the multi-head self-attention and the fully connected feed-forward network. The multi-head self-attention maps the query set \(Q\), the key set \(K\), and the value set \(V\) into an attention matrix by  
\[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h}) W^{E}, \tag{3}\]  
\[\text{head}_{i}=\text{Attention}\Big{(}QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\Big{)},\]  
\[\text{Attention}(Q,K,V)=\text{softmax}\Bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \Bigg{)}V,\]  
where \(Q\), \(K\), and \(V\) are the vector sets of query, key, and value. The dimension of query, key, and value is \(d_{k}\), \(d_{k}\), \(d_{v}\), respectively, and they are initialized by applying a linear transformation on the output of the previous sub-layer. Then, the fully connected feed-forward network performs the position-wise computation on the output \(X\) of the previous multi-head self-attention layer by  
\[\text{FFN}(X)=\text{max}\Big{(}0,XW_{1}^{E}+b_{1}^{E}\Big{)}W_{2}^{E}+b_{2}^{ E}, \tag{4}\]  
The decoder is also composed of a stack of \(N\) identical layers, where each layer has three sub-layers consisting of the multi-head self-attention, the multi-head attention and the fully connected feed-forward network. The multi-head self-attention was first applied on the target-side positional embedding matrix as the same with the encoder. Then, the multi-head attention builds an attention model between the encoder and the decoder, which utilizes the \(K\) and \(V\) vector sets from the outputs of the encoder and the \(Q\) vector set from the previous layer of the decoder as inputs to generate a context representation by  
\[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h}) W^{D}, \tag{5}\]  
Then, the fully connected feed-forward network is applied on the context representation \(Y\) by  
\[\text{FFN}(Y)=\text{max}\Big{(}0,YW_{1}^{D}+b_{1}^{D}\Big{)}W_{2}^{D}+b_{2}^{ D}, \tag{6}\]  
Figure 1: The structure of the single-source Transformer model with the encoder–decoder framework.  
Finally, a linear network layer followed by a softmax layer was applied on the above generated vector to predict the target word sequence. In addition, the residual connection [46] was employed around each sub-layer followed by layer normalization [47].