## 6 Proposed Model
### Arabic Dialect Encoding with Bi-LSTM  
Long-Short Term Memory network is a type of recurrent neural network RNN that specifically address the issue of learning long-term dependencies by boosting the RNN with memory vector \(m_{t}\in R^{d}\). In the proposed Multitask NMT architecture, we used Bi-LSTM as an encoder for the translation tasks and part of speech POS tagging task. An LSTM unit takes \(x_{t}\), \(h_{t-1}\) and \(m_{t-1}\) as its input and produce \(h_{t}\) and \(m_{t}\) by computing the following Equation (6):  
Figure 2: The Architecture of the proposed model.  
\[\begin{array}{l}i_{t}\ =\ \sigma(W_{i}x_{t}+U_{l}h_{t-1}+b_{i}),\\ f_{t}\ =\ \sigma\Big{(}W_{f}x_{t}+U_{f}h_{t-1}+b_{f}\Big{)}.\\ o_{t}\ =\ \sigma(W_{o}x_{t}+U_{o}h_{t-1}+b_{o}),\\ g_{t}\ =\ \tanh\big{(}W_{g}x_{t}+U_{g}h_{t-1}+b_{g}\big{)},\\ m_{t}\ =\ f_{t}\odot\ m_{t-1}+i_{t}\odot\ g_{t},\\ h_{t}\ =\ o_{t}\cap\tanh(m_{t})\end{array} \tag{6}\]  
where \(i_{t}\), \(f_{t}\) and \(o_{t}\) designate the input, forget and output gates respectively. These gates collectively determine how to update the current memory cell \(m_{t}\) and the current hidden state \(h_{t}\). The parameter \(d\) is used to indicate the memory dimension in the LSTM were all the vectors in the defined architecture has the same dimension. \(\sigma(.)\) is an element-wise sigmoid function with an output range between \([0,1]\). Subsequently, \(tanh\) indicates the hyperbolic tangent function that has an output range between \([-1,1]\) and \(\odot\) denotes the element-wise multiplication function and \(W_{p}\), \(U_{p}\) and \(b_{p}\), \(p\in\ \{i,f,o,g\}\) are considered as network parameters. The function \(f_{t}\) is set to have a better understanding of mechanisms involved in the architecture and to control distinct type of information that is needed to be discarded from old memory cell. In addition, the use of \(i_{t}\) to control in the amount of information that is stored in the current memory cell and \(o_{t}\) to control the parameters required to be provided as an output based on the memory cell \(m_{t}\). LSTM was designed to deal with vanishing and exploding gradient problem when training traditional recurrent neural network. LSTM structure is illustrated in Figure 3.  
For a given sequence \((x_{1},x_{2},\ldots,x_{n})\) contains \(n\) words, each is represented as \(d\) dimensional vector, LSTM computes a representation \(\overrightarrow{h_{t}}\) of the left context of the sentence at every word \(t\). Gathering a representation of the right context \(\overleftarrow{h_{t}}\) will add useful information. This can be done using second LSTM that reads the same sentence in reverse. We will refer to the former as the forward LSTM and the latter as the backward LSTM. These are two distinct networks with different parameters. This forward and backward LSTM pair is referred to as a bidirectional LSTM (Bi-LSTM). The representation of a word using this model is obtained by concatenating its left and right context representations \([\overrightarrow{h_{t}};\overleftarrow{h_{t}}]\). These representations effectively include a representation of a word in context, which is useful for numerous tagging applications.