## 4 Results and Discussion  
After cleaning the data, the data set was manually annotated by annotators. Five machine learning (ML) algorithms along with two vectorization techniques were applied for the sentiment classification. The data were divided into a ratio of 80% for training and 20% for testing purposes, respectively. The classifiers used were RF, DT, MNB, GNB, and SVM. All classifiers were implemented in Python. At first, the models were applied to a data set of one thousand (1000) annotated tweets, divided into 80-20 ratios for testing and training purposes. Considered a small data set, the accuracy of RF was better as compared to other classifiers on small data sets consisting  
Figure 4: Trigram with positive words. Figure 5: Trigram with negative words. Figure 3: Bigram with positive words. of Roman Urdu. As shown in Figure 6, the accuracy of the RF count vectorization was 0.82 and RF-TFIDF showed an accuracy of 0.80. In our investigation, classifiers on small data sets produced stable results using both vectorization techniques. RF with fewer features in a smaller data set in combination with count vectorization produced better outcomes than RF with TFIDF. When using the DT classifier on a small data set, the results accuracy outperformed those of SVM, MNB, and GNB. Although results with vectorization in combination with DT were stable, DT with count vectorization showed better accuracy (0.78) compared to DT with TFIDF (0.72). SVM performed better than MNB and GNB on the small data set. As shown in Figure 6, SVM-TFIDF gave an accuracy of 0.75 and the SVM count showed an accuracy of 0.67. MNB in the small data set performed better than GNB. MNB-TFIDF yielded better results compared to the MNB count vectorization, i.e., 0.72 and 0.67, respectively. The GNB results with both vectorization techniques were not outstanding; GNB-TFIDF was 0.54 and the GNB count was 0.51 in the results. In the second phase, we increased the data set to four thousand (4000) annotated tweets in Roman Urdu, considered a large data set; the tweet data set was divided into 80-20% ratios for testing and training sets, i.e., 3200 and 800, respectively. Figure 7 illustrates the accuracy of the large data set. The accuracy of the large data set showed variations when compared to the small data set as Bayesian classifiers outperformed trees. MNB with count vectorization outperformed all other classifiers with an accuracy of 0.75 in the large data set while MNB-TFIDF vectorization showed an accuracy of 0.45. Other than MNB, there was no significant difference in accuracy when applying the same classifier but there was a different vectorization technique for the large data set. The accuracy of GNB was improved in the large data set, although it showed stable results with both vectorization techniques. The GNB count performed better in the large data set with an accuracy of 0.73 and a GNB-TFIDF of 0.71. In the large data set, the RF count showed an accuracy of 0.71; with RF-TFIDF, it had an accuracy of 0.69. DT with the large data set was not considered the best compared to the small data set; the DT-count and DT-TFIDF showed accuracies of 0.63 and 0.61, respectively. The SVM count and SVM-TFIDF showed accuracies of 0.62 and 0.63, respectively. Figure 6: Accuracy of the small data set (1000 tweets). We found that random forest (RF) and count vectorization performed best on the small data set related to anti-government gatherings in Roman Urdu, obtained from Twitter, for the task of sentiment analysis. The accuracy achievement was 0.82. In our study, we found that random forest (RF) performed well when applied to the small data set of Roman Urdu tweets related to protests, with an accuracy of around 80%. This is in line with previous research that suggests that RF is a suitable algorithm for handling high-dimensional and sparse data, as well as unbalanced data sets [32]. Additionally, by using count vectorization, our study is able to select the most informative features, which can further improve the performance of the classifier [33].