# Social Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection
## 5 Experimentation Results
### Ensemble Multi-Class Classifiers  
As presented earlier, we have built three ensemble classifiers (EMCREST, EMCS, EBBC) which combine the BBC classifier with three sets of different classifiers. We performed our experiments using the pruned dataset that was previously described and we used the pruned-feature set, which in the binary bot classification problem seemed to perform better than the full set of features. Following the same procedure as before, we split our data into train and test samples in a 75-25% ratio, and handled the imbalance of our training set using the ADASYN algorithm. Initially we trained the one-vs-rest algorithm, which actually creates six binary classification problems, where each class \"competes\" against all the others. The next step involved the training of a stacking Random Forest classifier which consists of five binary Random Forest classifiers, where the competing classes are always \"humans\" and each other type of bot. These binary classifiers were actually introduced in Section 3.2, where we used them to validate the labels we manually assigned. Having said that, we proceed with the results of our experiments, which are thoroughly presented in Table 7. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Best 1** & **Best 2** & **Best 3** & **Best 4** & **Best 5** \\ \hline _Full-Features_ & U: 0.821 & UT: 0.849 & UTN: 0.855 & UTHN: 0.855 & UCTHN: 0.851 \\ \hline _Pruned-Features_ & U: 0.851 & UT: 0.87 & **UTH: 0.874** & UCTH: 0.87 & UCTSH: 0.866 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Accuracy performance of the BBC classifiers in comparison to the different combinations of feature categories. Each column refers to the best combination for each number of combined feature categories. Apparently, the combination of **U**ser, **T**emporal and **H**ashtag feature categories provides the best results. However, the accuracy does not fluctuate significantly, regardless of the feature category combination that has been used. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Data \& \#Features** & **Precision** & **Recall** & **Accuracy** & **F1** & **G-Mean** \\ \hline _Full-features_[145] & 0.787 \(\pm\) 0.032 & 0.792 \(\pm\) 0.054 & 0.832 \(\pm\) 0.025 & 0.789 \(\pm\) 0.042 & 0.824 \(\pm\) 0.035 \\ \hline _Pruned-features_[100] & **0.893 \(\pm\) 0.02** & **0.849 \(\pm\) 0.03** & **0.861 \(\pm\) 0.025** & **0.87 \(\pm\) 0.026** & **0.862 \(\pm\) 0.025** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance of the BBC classifiers in both available datasets. As we can see, the EBBC classifier's performance is superior to that of all the other classifiers. Evidently, the EBBC classifier would be our choice even if we would just like to distinguish humans from bots in general, since it outperforms the binary bot or human classifier BBC. In the next figure, see Figure 7, we present the distribution of maximum prediction probabilities for the whole dataset. Apparently, apart from a few cases where the maximum probability lies below the 0.5 threshold, most instances have been correctly classified with a high probability. In this case, defining the most important features is not trivial, since we refer to a multi-class classification problem. For this reason, we examined the most important features per bot type, using the binary bot type classifiers. The results once more validated our hypothesis and manual assignment of labels to each bot type. More specifically, our experiments showed that the top 25 features are completely different for each bot type. This means, that there is not even one single feature among the twenty five top-ranking ones that is present in every class. In Figure 8, we observe that, surprisingly, the important features tend to become common for all bot classes after we reach the top 250, while the first common one makes its appearance after the first 25 ranked as most important.