## 3 Neural Machine Translation  
Neural machine translation (NMT) has lately been introduced as a promising methodology with the potential of handling the shortcoming of traditional statistical machine translation systems. The power of NMT models lies in its ability to learn directly, end-to-end fashion, the mapping from input text to associated output text. Neural models are not new in the field as [PERSON] and [PERSON] [19] proposed such a solution many years ago. Other models by [PERSON] et al. [20] and [PERSON] and [PERSON] [21] have also been proposed, but for the first time [PERSON] [22] and [PERSON] [23] were able to design a powerful architecture for machine translation.  
In the encoder-decoder architecture which was discussed by [PERSON] [24], two recurrent neural networks (RNNs) are trained together to maximize the conditional probability of a target sequence (candidate translation) \(y=y_{1}\),... \(y_{m}\), given a source sentence \(x=x_{1}\),... \(x_{n}\). Input words are sequentially processed consecutively until the end of the input string is reached. An encoder reads words and maps the input sequence into a representation with a fixed-length. At each time in step \(t\), an input word is taken and the hidden state is further updated. This process can be expressed as in Equation (1):  
\[h_{t}\ =\ f(E_{x}[x_{t}],h_{t-1}) \tag{1}\]  
where \(h_{t}\epsilon R^{d}\) the hidden state (a vector) is at the time step \(t\) and \(f(.)\) is a recurrent function such as long short-term memory (LSTM) [25] or gated recurrent unit (GRU). \(f(.)\) is responsible for updating the hidden state of the layer and other associated units (if there are any, such as memoryunits, etc.) \(E_{x}\in R^{|V_{x}|\times d}\) is an embedding matrix for source symbols (\(d\) is the embedding size). The embedding matrix is a Look-up table (LUT) whose cells are treated as network parameters and updated during training. The embedding (numerical vector) for the \(v\)th word in \(v_{x}\) (vocabulary) resides in the \(v\)th row of the table. In the next step, the model undertakes processing for all words in the source sequence; \(h_{n}\) is a summary of the input sequence which is referred to as the context vector (\(c\)). Another RNN is initialized by \(c\) and seeks to produce a target translation. There is one word sampled from a target vocabulary \(v_{y}\) at each step of the process. The decoder conditions the probability of picking a target word \(y_{t}\) on the context vector, the last predicted target symbol and the decoder's state. This can be expressed in Equations (2):  
\[y_{t}\ =\ g(E_{y}[y_{t-1}],S_{t},c) \tag{2}\]  
\(S_{t}\ =\ f\big{(}E_{y}[y_{t-1}],S_{t-1},c\big{)}\) where \(S_{t}\) is the decoder's hidden state. Since we compute the probability of selecting \(y_{t}\) as the target word, \(g(.)\) should give a value in the range \([0,1]\). The most common function for \(g(.)\) is Softmax. The encoder and decoder RNNs are trained together to maximize the log probability of generating a target translation and are given an input sequence \(x\), so the training standards can be defined as in Equation (3):  
\[\underset{\theta}{\max}\frac{1}{K}\sum_{k\ =\ 1}^{k}\log(y_{k}|x_{k}) \tag{3}\]  
where \(\theta\) is a collection of network parameters and \(k\) designates the size of the training set. As mentioned before, recurrent functions in encoder-decoder models are not usual mathematical functions. RNNs are not powerful enough to capture all features about sequences, so more powerful choices, such as LSTM RNNs are needed.