## 3 Materials and Methods
### Dataset Preparation
#### 3.1.3 Building Ground-Truth Dataset  
Some users, as stated earlier, assigned rating scores for their reviews inconsistently. Thus, a new \"Sentiment Polarity\" feature was added to the dataset. This feature will be used as a sentiment class label used for training the ML classifiers. To avoid errors caused by users when they assign rating scores incorrectly, we asked three annotators to voluntarily judge 1135 reviews by giving, based on their opinions, a score for each review they revised.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Features** & **Description** \\ \hline UserName & The name of the user who wrote the review \\ ReviewContent & Content of review \\ DateOfPost & The date when the review was posted \\ ThumbsUpCount & Number of users who have the same feelings \\ Rating & Five-scale rating represents a score given by the user (0 = low, 5 = high) \\ Lang & Language of review \\ Sentiment Polarity & Sentiment label corresponding to each review (\(-1\) = negative, 0 = neutral, 1 = positive) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dataset description.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**App** & **\# Downloads** & **Rating** & **Current Version** & **Size (Mb)** & **Total No. of Reviews** \\ \hline Tawakalna & 10M & 4.5 & 3.3.0 & 131 & 575,842 \\ Tetaman & 1M & 2.7 & 1.7 & 26 & 6088 \\ Tabaud & 5M & 4.6 & 1.2.0 & 6.6 & 23,992 \\ Sehhaty & 10M & 4.4 & 2.13.2 & 42 & 263,004 \\ Mawid & 1M & 4.7 & 10.10.0 & 61 & 126,941 \\ Sehhah & 1M & 4.0 & 1.0.35 & 73 & 6121 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Google Play Store metadata of mobile apps.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**No.** & **Review Content** & **Translated Review** & **Score** \\ \hline
1 & _existed_ & _existed_ & _existed_ & _existed_ \\
2 & _existed_ & _existed_ & _existed_ & _existed_ \\
3 & _existed_ & _existed_ & _existed_ & _existed_ \\
4 & _existed_ & _existed_ & _existed_ & _existed_ \\
5 & _existed_ & _existed_ & _existed_ & _existed_ \\ \hline \hline \end{tabular}
\end{table}
Table 3: Review sample from the dataset with a misleading score assigned by a user.  
As each annotator assigned a sentimental score separately, we followed the same guideline that was used in [32; 33].  
The final sentiment polarity class label was assigned by computing the majority voting algorithm and labeling the review as positive, neutral, or negative.  
Figure 2 shows the distribution of polarity scores in the baseline dataset, which includes 522 instances that were labeled as positive reviews, 375 instances as negative reviews, and 238 instances as neutral reviews. Later, the Camel tool was used to predict the remaining instances in the dataset. To ensure that the tool assigned a correct sentiment polarity score for each review, the confession matrix was computed by calculating the values predicted by the Camel tool and those assigned by the annotators. The accuracy score shows that the Camel tool can predict class labels with an accuracy of 98.26%. Hence, employing the Camel tool on the remaining instances allowed us to extend the baseline dataset and the final distribution of sentiment polarity, as shown in Figure 3.