# Improving the Performance of Vietnamese-Korean Neural Machine Translation with Contextual Embedding
## 2 Transformer Model
### The Encoder of the Transformer  
A transformer contains a stack of encoder blocks. Each encoder block involves two sub-layers, including a multi-head attention mechanism (which computes multiple attentions instead of a single attention) and a feed-forward network, as shown in Figure 1. Each encoder sends its output to the next encoder, and the last encoder generates a matching representation of the corresponding input sentence.  
First, the input sentence is converted to input embedding. This embedding matrix is fed into the first encoder block (encoder 1). Encoder 1 receives the input and puts it into a multi-head attention layer, which generates the attention matrix as output. The attention matrix is then sent to the feed-forward network as the input for this layer, which returns the encoder representation as output. In the next step, the encoder representation of encoder 1 is fed into encoder 2 as input, and encoder 2 performs similar steps as encoder 1. This process ends when the final encoder block (encoder L) is computed and the representation of the given input sentence is generated.  
As an example, suppose \(n\) is the number of the single attention matrix \(Z\) and the multi-head attention mechanism is the combination of all the attention matrices (attention theads) multiplied by a new weight matrix, \(W_{0}\), as follows:  
\[multi-head\ attention=Concatenate\ (Z_{1},Z_{2},\dots,Z_{n})W_{0} \tag{1}\]  
Each attention matrix \(Z_{x}\) can be computed as  
\[Z_{x}=softmax\left(\frac{Q_{x}K_{x}^{T}}{\sqrt{d_{k}}}\right)V_{x} \tag{2}\]  
where \(Q_{x}\), \(K_{x}\), and \(V_{x}\) are, respectively, the query, key, and value matrix of input word \(x\). The query is the information we are looking for, the key represents relevance to the query, and the value is the actual content of the input; \(d_{k}\) represents the dimension of the key vector. Instead of computing a single attention head, multiple attention heads are calculated. As indicated in Equation (2), a single attention head is computed by the self-attention mechanism as follows: first, the dot product between the query matrix \(Q\) and the key matrix \(K^{T}\) is computed; the value of \(QK^{T}\) is then divided by the square root of the dimension of the key vector; next, the values obtained in the previous step are fed into the softmax function to normalize the scores and achieve the score matrix; finally, the attention matrix \(Z\) is computed by multiplying the score matrix by the value matrix \(V\).  
The feed-forward network in the encoder of the transformer contains two dense layers with a rectified linear unit (ReLU) activation function. The parameters of the feed-forward network differ in the encoder blocks, but they are the same over the various positions of the sentence.  
In addition to the feed-forward network and multi-head attention, the add-and-norm process is also an important component of the encoder. The add-and-norm component is used to connect the input and output of each feed-forward network and multi-head attention layer. In addition, a positional encoding (an encoding announcing word order) provides information about the position of each word before feeding the information into the transformer model, as shown in Figure 2.  
Figure 1: A simple demonstration of the encoder.  
Figure 2: Encoder x with the add-and-norm component.  
Another crucial component of the encoder is the add-and-norm component. It connects the input and output of the multi-head attention layer and the input and output of the feed-forward network layer. This component consists of a residual connection and a normalization layer. The add-and-norm component is fed into the encoder to make the training process faster, by avoiding heavy changes in value from layer to layer.