## 3 Methodology
### SEBD Architecture and Framework Phases
The researchers found that profile features have more weight in the performance of their model. #### 4.1.2 Evaluation Using Baseline

In this evaluation, we tested the model using group 1 hashtags, as shown in Table 1. We used a baseline tool to evaluate the resulting predictions. We extracted the account IDs and used Botometer API ([[https://rapidapi.com/OSoMe/api/botometer-pro](https://rapidapi.com/OSoMe/api/botometer-pro)]([https://rapidapi.com/OSoMe/api/botometer-pro](https://rapidapi.com/OSoMe/api/botometer-pro)), accessed on 30 September 2022) [50] to label these accounts using two different thresholds for the complete automation probability (CAP) scores, as shown in Table 4. CAP scores aim to recognize whether an account is run using automatic content software and the higher the score the higher the probability of this account being a bot. In [51], they suggest that the Botometer CAP threshold score is best at above 0.7. The results for the accuracy and F1 scores show good prediction of bot account when the CAP is 0.5, compared to when it is 0.75. Therefore, automatic behavior is not limited to bots, as they are restricted by Twitter policy and many human accounts (cyborg accounts) use automatic feeds in their account. The results of MCC indicate disagreement between the predictions of bots and humans using Botometer. This highlights that bots and cyborg accounts participate more in trending hashtags than human users [52]. Figure 3A,B represents the ROC curves for both thresholds using Botometer. The reason for the low area under the ROC curve (AUC) is the high false-positive rate. SEBD can even more accurately detect evolved bots than Botometer, as the second evaluation method shows. #### 4.1.3 Evaluation Using Benchmark Dataset

The third evaluation method was to use a ground truth dataset to train a supervised learning classifier and test it using our collected dataset (group 1) after predicting its labels using SEBD. We chose a random forest (RF) classifier to evaluate its performance, as it has been proven to effectively solve this problem [53]. For the ground truth dataset, we use the dataset available in GitHub ([[https://github.com/Niyaz94/Bot-Detction-On-Twitter/tree/main/Datasets](https://github.com/Niyaz94/Bot-Detction-On-Twitter/tree/main/Datasets)]([https://github.com/Niyaz94/Bot-Detction-On-Twitter/tree/main/Datasets](https://github.com/Niyaz94/Bot-Detction-On-Twitter/tree/main/Datasets)), accessed on 29 September 2022). The author of this dataset used Botometer to label the obtained accounts from the Twitter API [54]. The dataset consists of more than 100K samples. The performance results of this evaluation method using the \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Tool** & **CAP** & **Accuracy** & **F1** & **Recall** & **Precision** & **MCC** \\ \hline \multirow{2}{*}{Botometer} & 0.50 & 0.7698 & 0.8645 & 0.8668 & 0.8623 & 0.1002 \\ \cline{2-7}  & 0.75 & 0.6491 & 0.7665 & 0.8843 & 0.6765 & 0.1254 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Botometer results using group 1 dataset. Figure 3: (**A**) Receiver operating characteristic (ROC) curve using Botometer with CAP score of 0.75. (**B**) Receiver operating characteristic (ROC) curve using Botometer with CAP score of 0.5.

group 1 dataset are shown in Table 5. These results indicate that SEBD can effectively identify bots accounts. MCC is a measure of the quality of binary classifications, and takes into account true and false positives and negatives. A value of 1 indicates perfect performance, 0 indicates random performance, and \(-1\) indicates complete disagreement between the model and ground truth. In our case, an MCC of 0.38 indicates that the model is performing well but could still be improved. Nevertheless, SEBD succeeded in efficiently identifying bot accounts, as shown in Figure 4. Figure shows the ROC curve for the second evaluation scenario.