# Attention-Based RU-BiLSTM Sentiment Analysis Model for Roman Urdu
## 6 Results and Discussion
### Discussion  
Deep learning methodologies usually entail more trainable time, data, and fine-tuning of parameters compared with machine learning approaches. Machine learning schemes are much faster and require less time for training and typically use bag-of-words for feature engineering. The feature engineering that is generally adopted for machine learning classifiers suffers from the curse of dimensionality owing to a sparse matrix and the absence of word vectors. In contrast, deep networks are found to be much more reliable in feature engineering as well as in extracting hidden patterns. The deep neural network ability of pattern recognition enables us to semantically empower the model. Furthermore, such networks, if powered by neural word embedding, could yield promising results. We also employed three neural word embedding schemes: word2 vec, Glove, and FastText that were trained on an ample amount of Roman Urdu text. In comparison with pre-trained word embedding, the random word embedding entails more parameters to train and attains less classification accuracy [64]. In sentiment analysis task, feature extraction and the design of classifier are two significant factors. The LSTM has shown excellent performance in many NLP problems; however, the performance can still be ameliorated. We \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Word Embedding**} & \multicolumn{5}{c}{**Evaluation Metric**} \\ \cline{3-6}  & & **PRE** & **REC** & **F1** & **ACC** \\ \hline \multirow{3}{*}{BiLSTM [62]} & Word2 vec & 0.88 & 0.87 & 0.87 & 0.87 \\  & FastText & 0.87 & 0.86 & 0.86 & 0.86 \\  & Glove & 0.77 & 0.75 & 0.74 & 0.75 \\ \hline \multirow{3}{*}{
\begin{tabular}{c} RUBLSTM \\ (Our implementation) \\ \end{tabular} } & Word2 vec & 0.88 & 0.88 & 0.88 & 0.88 \\  & FastText & 0.87 & 0.86 & 0.86 & 0.86 \\ \cline{1-1}  & Glove & 0.76 & 0.75 & 0.74 & 0.74 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Sentiment analysis of IMDB Movie Reviews dataset. proposed an attention based bidirectional LSTM-based deep neural network RU-BiLSTM for Roman Urdu sentiment analysis. In our series of experiments for Roman Urdu sentiment analysis, we observed various factors affecting the classification accuracy. It is observed that the attention layer has a significant effect on ameliorating the classification accuracy owing to concentrating on what is more important. On the other hand, the bidirectional LSTM's capability of learning the network in both directions is vivid in the results. It is quite evident that traditional RNN lacks the capability of handling long-term temporal dependencies, which restricts the classification accuracy of this model. By fine-tuning the embedding dimensions, it is observed that word embedding size also plays a vital role in establishing the semantic relationship between the features and, consequently, improves the performance. Moreover, among three neural word embedding, word2 vec has shown superior performance in comparison with Glove and FastText. We also performed experiments by comparing words' weights before and after the attention layer. It is observed that the attention layer significantly modifies weights of many words, especially words that are subjective, i.e., conveying sentiment in Urdu. That is likely the reason why adding an attention layer to BiLSTM improves the performance of the model. Experiments and careful analysis also show that the accuracy of the proposed model could not be further improved because of the main limitations that are discussed in the Introduction section, i.e., several variations in spelling for the same words and multilinguality since Roman Urdu is very informal in writing and people use different spelling for the same words. Sometimes, people express their feelings in words and change the spelling unnecessarily. For example, for the sentence \"really enjoyed (in English)\", in Roman Urdu, it is written as \"boot maza aya\". This same sentence can be written by some people as boooooooot maaaaaaaaa ayaaaaa\".