# Domain Adaptation for Arabic Machine Translation: Financial Texts as a Case Study
## 2 Background  
### Neural Machine Translation  
NMT models based on deep neural networks (DNN) have been proposed in early NMT research [17]. A DNN-based NMT model employs a neural network system to perform the required machine translation tasks using an encoder-decoder network [18]. The encoder neural network inputs and encodes a source language sentence into a fixed-length vector in each hidden state. Then, given the final hidden state of the encoder, the decoder does the reverse work by transforming the hidden state vector to the target sentence word by word. A translation probability of a source sentence is modeled into the target sentence. Given a source sentences \(S=\left\{s_{1},s_{2},..s_{n}\right\}\) and a target sentence \(T=\left\{t_{1},t_{2},..t_{n}\right\}\), the encoder encodes all the words from the source sentence \(S\) into a set of hidden states \(\left(h_{1},h_{2},..h_{n}\right)\) and passes the fixed-size vector \(v\), which represents the source sentence, to the decoder. The translation probability with a single neural network is given by the following formula [19]:  
\[P(S)=\prod_{i=1}^{n}P(t_{<i},S) \tag{1}\]  
where \(t<i\) stands for the sequence preceding the \(i\)-th target word. Hence, each predicted word \(t_{i}\) is based on the previously predicted word \(t_{i-1}\) and the previous hidden states \(h_{i-1}\). However, when the sentences become long the performance deteriorates. This limitation is due to the limited feature representation ability in a fixed-length vector [17]. To overcome this issue and to provide additional word alignment information in translating long sentences, [PERSON] et al. [20] introduced the idea of the attention mechanism. Concretely, the attention mechanism is an intermediate component between the encoder and decoder, which can help to determine the word alignment dynamically. The decoder pays attention to input or to any part of the input sentence. Attention is calculated using each encoder output and the current hidden state, resulting in a vector of the same size as the input sequences using score functions [20]. There are three different architectures for constructing NMT, namely Recurrent neural network (RNN), Convolution neural network (CNN), and Self-attention-based Transformer. The use of RNN-based models has demonstrated good-quality translation results. This type of network is composed of an encoder and decoder with similar uses of sequence-to-sequence learning. Multiple variants of RNN architectures include, i.e., LSTM [21], BiLSTM [20] and GRU [22]. The second approach to developing NMT systems is based on convolution neural network (CNN) architecture. Work using CNN has generally reported good results, especially for word-based MT [23]. This work applied a convolution layer on the bottom of the recurrent layer which hinders the performance. The bottleneck was handled by implementing the fully convolutional model as suggested by [24]. The performance and accuracy were improved with a number of models; word-based [25], character-based [11], and recently with attention [26]. Recently, the use of transformers has resulted in well-performing machine translation systems. This model is a sequence-to-sequence model [27], which consists of a stack of layers. Each layer first utilizes self-attention to extract information from the whole sentence and then follows a point-wise feed-forward network to provide non-linearity. The novel idea of self-attention is to extend the mechanism to the processing of input sequences and output sentences as well. In general form, the Transformer attention function uses three vectors: queries (\(Q\)), keys (\(K\)) and values (\(V\)). The output is a weighted sum of values, where weights are computed by a similarity score between \(n\) query vectors and \(m\) keys [27]. The attention is defined as follows:  
\[Attention(Q,K,V)=softmax(score(Q,K))V \tag{2}\]  
where score \(Q\), \(K\) is an n x m matrix of similarity scores. A straightforward choice for score \(Q\), \(K\) proposed by [PERSON] et al.