# Detecting Suspicious Texts Using Machine Learning Techniques
## 3 A Novel Suspicious Bangla Text Dataset
### Development of SBT Corpora  
Bengali is the resource-constrained language due to its scarcity of digitized text contents and unavailability of benchmark datasets. By considering the explanation of SBT and the characteristics of suspect activity defined by the U. S. department of homeland security, we accumulated the text data from various online sources [34]. We endorsed the same technique of developing datasets, as explained by [PERSON] et al. [35]. Figure 1 illustrates the process of dataset development. **Data crowd-sourcing:** Figure 2 shows the total number of texts collected from different sources in terms of suspicious (S) and non-suspicious (NS) classes. We have crawled a total of 7115 texts among them 3557 texts are S, and 3558 texts are NS. In the case of the suspicious class, 12.2% of source texts collected from the website (W), 12% data collected from the Facebook comment (FC), and 10.2% from the newspaper (N). Other sources such as Facebook posts (FP) and online blogs (OB) contributed 8.9% and 5.4% of text data. On the other hand, a significant portion of non-suspicious source texts collected from the newspapers (30.4%). A total of 7.8% of non-suspicious texts were collected from the OB, 5.6% from the W and 3.2% from the FC. A tiny portion of the texts was accumulated from various sources (such as novels and articles) in both classes. As the sources of the newspapers, the three most popular Bangladesh newspapers are considered (such as the daily Jugantor, the daily Kaler Kontho, and the daily Prothom Alo) for accumulating the texts. **Data labelling:** Crowd-sourced data are initially labelled by five undergraduate students of Chittagong University of Engineering and Technology who have 8-12 months of experience in the BLP domain. They are also doing their undergraduate thesis on BLP and attended several seminars, webinars, and workshops on computational linguistics and NLP. **Label verification:** The expert verifies the data labels. A professor or a PhD student having more than five years of experience or any researcher having vast experience in the BLP domain can be considered as an expert. The final labels (\(C_{ns},C_{s}\)) of data are decided by pursuing the process described in Algorithm 1. Figure 1: Process of dataset accumulation. ```
\(T\leftarrow\) Text Dataset; \(A\leftarrow\) Set of annotators; \(Final\_Label\leftarrow\) will contain final labels; \(Initial\_Label,Expert\_Label\); fori\(eT\)do \(Count_{S}\gets 0,Count_{NS}\gets 0\) (initialization of suspicious and non-suspicious count); for\(a_{j}eA\)do if\(a_{j}==C_{s}\)then \(Count_{S}=Count_{S}+1\); else \(Count_{NS}=Count_{NS}+1\);  end if  end for \(Initial\_Label=Count_{NS}>Count_{S}\)? (0) : (1); \(Expert\_Label=(0)\) || (1); ifInitial\(\_Label==Expert\_Label\)then \(Final\_Label[i]=Initial\_Label\); else \(Final\_Label[i]=\)'\(x\)';  end if \(i=i+1\);  end for
```  
**Algorithm 1**Process of data labelling  
For each text in \(T\), the annotator labels are counted. \(a_{j}\) indicates the \(j^{th}\) annotator label for the \(ith\) text. If the annotator label is suspicious (\(C_{s}\)), then suspicious count (\(Count_{S}\)) will be increased; otherwise, the non-suspicious count (\(Count_{NS}\)) will increase. Majority voting [36] will decide the initial label. If the non-suspicious count is greater than the suspicious count, then the initial label will be non-suspicious (\(0\)); otherwise, suspicious (\(1\)). After that, the expert will label the text as either  
Figure 2: Source texts distribution in suspicious (S) and non-suspicious (NS) categories. The acronyms FP, FC, W, OB, N, and M denote Facebook pages, Facebook comments, websites, online blogs, newspapers, and miscellaneous, respectively. non-suspicious or suspicious. If the initial label matches the expert label, then it will be the final label. When disagreement increased, the label marked with '\(x\)', and the final label will be decided by a discussion between the experts and the annotators.