# An Empirical Approach for Extreme Behavior Identification through Tweets Using Machine Learning
## 4 Experiments
### Data Collection and Preprocessing  
A _Twitter Streaming API_ was used to collect tweets from the geo-location of Afghanistan from 2 nd April 2018 to 8 th April 2018 after Kunduz Madrassa attack by Afghan forces. A list of 60 specific query terms, like _Taliban, airstrike, peace, Afghan_, etc. were used to extract tweets from the profile of users (see Section 3.1.1 for query words details). A total of 7500 tweets were downloaded. Many tweets were repeated in the corpus because of retweet feature of Twitter, and we only considered unique tweets in our dataset. We also filtered out tweets that are not relevant to the context of our problem and they appear in a dataset because of some query words in them but their text is not related to our problem. Finally, the resulting dataset has 3380 tweets for the analysis. For the classification task, the labels were required to be given using most appropriate and accurate approach. As mentioned earlier in Section 3.1.1, automated data annotation is often observed to be not reliable with around 70% error rate when compared with humanly assigned classes for our dataset (see Table 3 for sample labels of tweets using _WordNet, SentiWordNet_ and manual labelling by humans). The dictionaries assign labels in numerical form where range of scores is between {-ve, 0, +ve}. The -ve and +ve score represent the negative and positive sentiment correspondingly and zero denote to a neutral sentiment. Based on these scores it is insufficient to distinguish between anti or pro entities because both writers use similar words. Therefore, we adopted an approach where five different persons from various walks of life (i.e., a journalist, an academic at the Islamia University of Bahawalpur (IUB) from political science department, an academic from computer science department of the IUB and first two authors of this paper) were asked to label tweets. More than 95% of the tweets were given class labels based on the consensus of all five persons. For the remaining 5% of the tweets, the three or more persons agreed to assign a one class label whereas the remaining one or two persons had different views of class label assignments, so in such cases the class label was assigned based on the majority. The main disadvantage of the involvement of humans to label each tweet is a time taking process for preparing a training set for machine learning models. To fully automate the process of detecting extreme contents, the research community needs to focus on developing methods to automatically label each tweet that not only involves the occurrence of the words in a tweet content but also needs to consider the semantics in naturally written text. After the labeling process, our dataset was divided into four categories (i.e., pro-Afghan, pro-Taliban, Neutral, Irrelevant) based on the sentiment that it contains. The irrelevant class consists of a few random tweets, which are not related to our specified topic. They are in the corpus due to some query-words, but their tweet text is immaterial with respect to the context. The irrelevant tweets are filtered out from the corpus in further experiments. The description of each class and sub-class in a dataset is presented in Table 4.