# Intelligent Detection of IoT Botnets Using Machine Learning and Deep Learning
## 4 Experimental Evaluation
### Multiclass Classification
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Botnet**} & \multirow{2}{*}{**Attack**} & \multirow{2}{*}{**Device**} & \multirow{2}{*}{**Benign**} & \multicolumn{6}{c}{**Bashlite**} & \multicolumn{6}{c}{**Mirai**} \\ \cline{6-13}  & & & & **COMBO** & **Junk** & **Scan** & **TCP** & **UDP** 1** & **ACK** & **Scan** & **SYN** & **UDP** 1** & **Plain UDP**2 \\ \hline \multirow{8}{*}{**BASHLITE**} & \multirow{4}{*}{**Junk**} & **Doorbell** & 0 & 7872 & 717 & 10 & 1 & 0 & 2 & 0 & 13 & 9 & 0 \\  & & **Baby Monitor** & 6 & 7874 & 1005 & 11 & 1 & 0 & 2 & 1 & 31 & 7 & 0 \\  & & **Security Camera** & 6 & 7259 & 1490 & 8 & 1 & 0 & 0 & 0 & 16 & 8 & 0 \\  & & **Webcam** & 4 & 7824 & 3535 & 10 & 0 & 0 & 2 & 0 & 49 & 5 & 1 \\ \cline{2-13}  & \multirow{4}{*}{**Scan**} & **Doorbell** & 122 & 0 & 0 & 6277 & 1 & 0 & 5 & 2508 & 0 & 35 & 12 \\  & & **Baby Monitor** & 179 & 0 & 0 & 5333 & 0 & 0 & 12 & 2747 & 0 & 15 & 0 \\  & & **Security Camera** & 144 & 0 & 0 & 5587 & 2 & 0 & 5 & 2572 & 0 & 19 & 0 \\  & & **Webcam** & 44 & 0 & 0 & 5875 & 6 & 0 & 16 & 2675 & 0 & 7 & 1 \\ \cline{2-13}  & \multirow{4}{*}{**TCP**} & **Doorbell** & 7 & 0 & 0 & 5 & 1 & 27,538 & 6 & 0 & 0 & 10 & 7 \\  & & **Baby Monitor** & 25 & 0 & 0 & 3 & 1 & 27,853 & 3 & 0 & 0 & 0 & 1 \\  & & **Security Camera** & 11 & 0 & 0 & 1 & 1 & 26,748 & 1 & 0 & 0 & 8 & 0 \\  & & **Webcam** & 17 & 0 & 0 & 7 & 0 & 29,187 & 1 & 0 & 0 & 0 & 5 \\ \hline \multirow{8}{*}{**Mirai**} & \multirow{4}{*}{**ACK**} & **Doorbell** & 0 & 0 & 1 & 0 & 0 & 2 & 0 & 0 & 29,856 & 895 \\  & & **Baby Monitor** & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 25,858 & 1727 \\  & & **Security Camera** & 0 & 0 & 0 & 0 & 0 & 4 & 4 & 0 & 0 & 17,441 & 0 \\  & & **Webcam** & 5 & 0 & 0 & 2 & 0 & 0 & 628 & 0 & 30,584 & 1286 \\ \cline{2-13}  & \multirow{4}{*}{**SYN**} & **Doorbell** & 0 & **25,415** & 0 & 4 & 0 & 1 & 0 & **11,198** & 170 & 0 & 0 \\  & & **Baby Monitor** & 1 & **23,909** & 0 & 5 & 0 & 0 & 0 & 10,823 & 787 & 0 & 0 \\  & & **Security Camera** & 0 & **9066** & 23 & 14 & 0 & 4 & 0 & 8155 & 1391 & 0 & 0 \\  & & **Webcam** & 0 & **20,428** & 0 & 20 & 0 & 0 & 0 & 8413 & 75646 & 0 & 0 \\ \cline{2-13}  & \multirow{4}{*}{**Plain UDP**2 & **Doorbell**} & 2 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 5344 & 19,001 \\  & & **Baby Monitor** & 2 & 0 & 0 & 4 & 0 & 0 & 0 & 0 & 0 & 4354 & 19,898 \\  & & **Security Camera** & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 8216 & 7834 \\  & & **Webcam** & 1 & 0 & 0 & 5 & 0 & 0 & 0 & 0 & 4250 & 20,827 \\ \hline \hline \end{tabular} The red bolded numbers are large numbers of misclassified samples; Gray shaded numbers indicate samples that are correctly classified; 1 UDP: UDP flooding; 2 Plain UDP: Less of an option of UDP flooding for higher packets per second.  
\end{table} 
Table 12: Detailed results of the naïve Bayes-based botnet detection. Figure 4: The F1-score of each attack type in the KNN-based model. Detection results by attack type, as shown in Table 13. The ACK of Mirai was mis-detected as UDP of Mirai, the SYN of Mirai was mis-detected as a COMBO of Bashlite and a scan of Mirai, and the UDP of Mirai was mis-detected as a Plain UDP of Mirai. Because of this, false positives occurred frequently. The average F1-score of each model as a result of training the dataset for each device using the three DL models and performing multiple classification is shown in Table 14. According to Table 14, CNN has the highest F1-score. Compared to CNN, RNN and LSTM have lower F1-score. For the CNN, although most of the detailed attacks were accurately detected, the detection F1-score for the TCP attack of Bashlite was 0%, as shown in Figure 5. According to the confusion matrix results (which is the table showing whether the class predicted by the model matches the original class of the target), the CNN model consistently detected the TCP attack of Bashlite as a UDP attack of Bashlite on all devices. In addition, for the security camera and webcam, the model detected the Plain UDP attack of Mirai as a UDP attack of Mirai.