# An Evolutionary Computation Approach for Twitter Bot Detection
## 3 Data Preparation
### Feature Extraction
Once this pipeline is run, we obtain for each user profile in _TwiBot-20_ a list of tokenized tweets, where each of these consists of a list of tokens that eventually includes hashtags and mentioned users. Moreover, we decide to divide each of these lists of tweets into two sub-lists, one containing only tweets that have been directly published by the user and one containing only the retweets. The main reason for this choice can be attributed to the fact that published tweets are probably more interesting for doing a meaningful analysis of word frequency. This expedient may enable an ML model to determine information related to the writing behavior of a given user. Including retweets in this kind of analysis would probably lead to misleading conclusions. However, it is also worth investigating the frequency of retweets compared to the frequency of authentic tweets. From now on, we are referring to the list of tweets as the list of published tweets that are not retweets, if not differently specified. By using user metadata along with tokenized tweets and retweets, it is possible to derive additional properties that can be adopted as hand-made features in the training phases of ML models. The set of features that we derive for each user account independently starting from user metadata, tweets, and retweets, are presented as follows. * _Statistical measures over tweet lengths_: a set of characteristics derived by applying statistical measures, namely, mean, median, maximum minimum range, and standard deviation, to an array containing the lengths, in terms of the number of characters, of the original tweets. In particular, the standard deviation of the lengths of tweets enables us to say whether the user tends to publish tweets of approximately the same length (e,g, identical tweets, which are common among certain types of bot accounts) or tends to vary a lot the size of the published posts. Figure 2 shows that bot profiles are more likely to exhibit a lower standard deviation of tweet lengths than humans, which means that bot profiles are more likely to publish tweets with approximately the same length and presumably the same type of content;
* _Frequency over the content of collected tweets and retweets_: a set of features that measure the frequency of appearances of retweets, URLs, mentioned users, and hashtags, over the sample of tweets and retweets retrieved via Twitter API for a given user account \(u\). Each feature of this category is computed using the following formula: \[\text{freq}_{c}(t_{u},r_{u})=\frac{\text{count}_{c}(t_{u}\cup r_{u})}{|t_{u} \cup r_{u}|}\] (1) where \(c\in\{\text{hashtags, mentioned\_users, URLs}\}\), \(t_{u}\) is the set of tokenized tweets for user \(u\) and \(r_{u}\) is the set of tokenized retweets for user \(u\) while \(\text{count}_{c}\) is a function that counts the number of times a word of category \(c\) appears in the input list of tokenized posts (tweets and retweets), including duplicates. Therefore, we are describing the mean number of hashtags, mentioned users, and URLs, for each retrieved tweet and retweet. Similarly, the frequency of retweets is calculated as follows. \[\text{freq}_{\text{retweets}}(t_{u},r_{u})=\frac{|r_{u}|}{|t_{u}\cup r_{u}|}\] (2) One intuitive reason that may justify the derivation of features such as, for example, the frequency of retweets and URLs, can be explained by observing the general trend of a significant percentage of bot profiles that avoid publishing self-made tweets and prefer to retweet specific content several times and post several URLs that usually redirect to spam, phishing, or, in general, undesirable web pages. Moreover, several bot profiles that, for example, promote political campaigns, are used to expose a high frequency of mentioned users, since they usually include the target politician's screen name in their tweets. Figure 3 confirms that bot profiles are more likely to exhibit a higher frequency of retweets than humans.