# Attention-Based RU-BiLSTM Sentiment Analysis Model for Roman Urdu
## 4 Recurrent Neural Network  
We empirically evaluated both classic LSTM and its variant BiLSTM. RNN has the power to capture historical information. In other words, RNNs can preserve the previous outputs, whereas the simple feed-forward networks just forward the new input and lack memory. Simple RNNs can only preserve the information of short-time stamps and fail to remember long time stamps. This problem is known as long-term dependencies, which can be dealt with using LSTM. The state-of-the-art LSTM units has the capability to deal with this long-term dependency problem. The recurrent architecture of LSTM can not only preserve the contextual information but also overcome the exploding and vanishing issue of the gradient descent. The gating units of LSTM enables it to control the flow and to decide what to ignore and what to update. The simple RNN as depicted in Figure 3 has a single layer recurrent module with a _tanh_ squashing function. Provided that we have an input neuron \(x_{t}\) and hidden output state \(h_{t}\) and the previous hidden output state \(h_{t-1}\), then the simple RNN can mathematically be represented as follows:  
\[h^{t}=g_{h}(W_{i}x^{t}+W_{R}h^{(t-1)}+b_{h}) \tag{3}\]  
\[y^{t}=g_{y}(W_{y}h^{t}+b_{y}) \tag{4}\]  
Here, \(h^{t}\) is the hidden output, \(g_{h}\) is a squashing function, \(W\) is the weighted matrix, \(b\) is a bias, and \(y^{t}\) represents the final output.  
The key idea behind _LSTM_ is cell states that act like conveyor belts. This conveyor belt transfers the information down in a chain of sequences and passes through different cell states with some minor linear interaction. During this journey, some information is added or removed to the cell states using gates. The classic LSTM consists of four memory blocks, each having a cell state \(c\) and three multiplicative units called gates. The gates are input gate \(i_{t}\), forget gate \(f_{t}\), and output gate \(o_{t}\), as illustrated in Figure 4 (source: [[https://www.edureka.co](https://www.edureka.co)]([https://www.edureka.co](https://www.edureka.co))) (accessed on 14 February 2022). A single memory block of LSTM can mathematically be demonstrated as follows:  
\[f^{t}=\sigma(W_{xf}x_{t}+W_{hf}h^{t-1}+b_{f}) \tag{5}\]  
\[i^{t}=\sigma(W_{xi}x_{t}+W_{hi}h^{t-1}+b_{i}) \tag{6}\]  
\[c_{t}^{\sim}=\tanh(W_{xc}x_{t}+W_{hc}h^{t-1}+b_{c}) \tag{7}\]  
Figure 3: Simple RNN architecture.  
Updating the cell state  
\[c_{t}=f_{t}\otimes c_{t-1}+i_{t}\otimes c_{t}^{\sim} \tag{8}\]  
\[o_{t}=\sigma(W_{xo}x_{t}+W_{hc}h^{t-1}+b_{o}) \tag{9}\]  
\[h_{t}=o_{t}\otimes\tanh(c_{t}) \tag{10}\]  
Here, \(\sigma\) represents the sigmoid function, and \(f_{t},i_{t},c_{t},o_{t}\) and \(h_{t}\) show the forget gate, input gate, cell state, output gate, and hidden state, respectively. In contrast, \(W\) denotes that the weight matrix corresponds to each separate layer and that \(b\) is the bias. The very first step in LSTM is to determine which information is not needed and may be ousted from the cell state. The sigmoid layer, also known as the forget layer, is responsible for this ruling. Second, to determine what new information is to be added, we have two-pass layers: a sigmoid layer called input gate layer, which decides the values to be updated, and a tanh layer, which generates a vector of new candidate values to be stored in cell state \(c_{t}^{\sim}\). Subsequently, the old cell state \(c_{t-1}\) is updated with the new cell state \(c_{t}\), as mentioned in Equation (8). Eventually, we run a sigmoid layer [0, 1] to determine what portion of the cell state should be included as output \(o_{t}\), which is further element-wise multiplied (denoted with \(\otimes\)) with a tanh layer having cell state as an argument. The tanh function restricts the values in between \(-1\) and \(1\). Consequently, we obtain a hidden output \(h_{t}\).