# IoTDS: A One-Class Classification Approach to Detect Botnets in Internet of Things Devices
## 5 Results and Discussion
### Predictive Performance
#### 5.3.2 Classifiers' Performance
Figure 6 shows the mean results for all metrics (except F1-score) also considering a 1-s time-window, a training sample size corresponding to 10% of legitimate data, and the optmised hyperparameters. Both plots aim to ease the comparison of the results for the different classifiers. F1-scores were placed in an exclusive plot because they can represent the overall performance of each classifier. Since each dataset is composed of 10,800 legitimate and 3600 botnet instances, only 1080 (i.e., 10%) legitimate instances were used for training in each technique. Mean values for specificity and AUC were similarly good for all algorithms. On the other hand, precision and recall presented some variations for different algorithms, which demands more detailed analysis and can allow to point out which algorithms had better performances. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Classifier** & **Hyperparameter** & **Value** \\ \hline EE & contamination & 0.0650 \\ \hline IF & n\_estimators & 100 \\  & contamination & 0.0346 \\ \hline LOF & n\_neighbours & 12 \\  & contamination & 0.0091 \\ \hline \multirow{2}{*}{OSVM} & \(\gamma\) & 0.8941 \\  & \(\
u\) & 0.0166 \\ \end{tabular}
\end{table} 
Table 6: Optimised hyperparameters. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Classifier** & **Hyperparameter** & **Range** & **Number of Samples** \\ \hline EE & contamination & [0, 0.5] & 40 \\ \hline IF & n\_estimators & (10, 50, 100) & 3 \\  & contamination & [0, 1) & 100 \\ \hline LOF & n\_neighbours & [1, 20] & 20 \\  & contamination & [0, 0.5) & 20 \\ \hline OSVM (RBF, Linear) & \(\gamma\) & [0, 1) & 20 \\ \hline OSVM (Poly) & \(\gamma\) & [0, 1) & 20 \\  & degree & (2, 3, 4, 5) & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hyperparameters’ ranges evaluated to find the best performance. The first algorithm analysed is EE. This classifier presents a high performance on specificity and AUC, but is among the worst results for precision. The worst scenario of this algorithm was the Mirai botnet in the ST profile, and the best results were in the SC profile with Aidra. The high performance on the SC profile can be explained by the lack of additional legitimate interventions from users. Its behaviour is only based on its main function, which makes it easier to induce a proper model since the normal behaviour is more predictable. The ST profile, on the contrary, has more variations in its normal behaviour, meaning that it is more challenging to the classifier. IF presented high values of specificity and AUC, though the precision and recall values do not support the use of this method. A clear example of this was the Mirai infection (I2) in the profile ST. In this dataset, the IF algorithm considers all the instances as legitimate, having a high specificity performance, but both precision and recall presented values of 0%. The ST profile includes some legitimate interventions on top of its main function of transmitting video, which induced the algorithm  
Figure 5: F1-scores considering all datasets with optimised parameters. Figure 6: Mean results for accuracy, precision, recall, specificity and AUC considering all datasets with optimised hyperparameters. to classify the changes caused by Mirai as legitimate. The other datasets presented better results, being the profile SC infected by Mirai the one with the best outcomes. The LOF algorithm had the best average results in all datasets, outperforming the OSVM classifier. The scenario with the best results for this algorithm was the infection type 1 (Aidra) on the SC profile with almost 99% of AUC. Also, the precision and recall presented a better performance than OSVM. The worst results of this algorithm were for the infection type 1 (Bashlite) on the ST profile with a recall of 65%. This may be a consequence of the ST profile's peaks of usage, which made the algorithm classify legitimate variations as botnet behaviour. Nevertheless, the precision remained with a high score.