# Intelligent Detection of IoT Botnets Using Machine Learning and Deep Learning
## 4 Experimental Evaluation
### Multiclass Classification  
The multiclass classification model considers 10 attacks injected into each device as individual attacks and classifies them into 11 groups, including benign. The F1-score of each model as a result of training each device dataset using the five ML models and performing multiple classifications is shown in Table 11. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Type of NB** & **Doorbell** & **Baby Monitor** & **Security Camera** & **Webcam** \\ \hline Gaussian NB & 0.95 & 0.84 & 0.88 & 0.98 \\ Bernoulli NB & 0.91 & 0.93 & 0.83 & 0.99 \\ Multinomial NB & 1 & 1 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Detection F1-score of each naïve Bayes on 4 different devices. Figure 2: Result of binary classification based on the three DL models. Compared to the binary classification in Table 8, DT and RF still provide F1-score closes to 1 for all devices, but NB and KNN show lower F1-scores. To determine why the F1-scores decreased in the NB model, the results of analyzing the F1-scores by attack type are shown in Figure 3.  
As shown in Figure 3, for NB based botnet detection under Bashlite attacks, junk, scan, and TCP detection have low F1-score, and for Mirai attacks, ACK, SYN, and Plain UDP detection show low F1-score. This occurs because, as shown in Table 12, the Junk of Bashlite was mis-detected as a COMBO (SYN+ UDP) of Bashlite, the scan of Bashlite was mis-detected as a scan of Mirai, and the TCP of Bashlite was mis-detected as the UDP of Bashlite. There were also several samples that were mis-detected as the UDP of Mirai from the ACK of Mirai and as a COMBO of Bashlite or a scan of Mirai from the SYN of Mirai and the UDP of Mirai from the Plain UDP of Mirai. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Device**} & \multirow{2}{*}{**ML**} & \multirow{2}{*}{**Benign**} & \multicolumn{8}{c}{**Bashlite**} & \multicolumn{8}{c}{**Mirai**} \\ \cline{3-11}  & & & **Combo** & **Junk** & **Scan** & **TCP** & **UDP** & **ACK** & **Scan** & **SYN** & **UDP** & **Plain UDP** \\ \hline \multirow{4}{*}{**Doorbell**} & NB & 0.99 & 0.97 & 0.08 & 0.7 & 0 & 1 & 0 & 1 & 0 & 0.99 & 0.78 \\  & KNN & 0.99 & 1 & 0.99 & 0.98 & 1 & 1 & 0.73 & 1 & 0.77 & 0.81 & 0.91 \\  & LR & 0.32 & 0 & 0 & 0.98 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\  & DT & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\  & RF & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \hline \multirow{4}{*}{**Baby Monitor**} & NB & 0.99 & 0.98 & 0.12 & 0.64 & 0 & 1 & 0 & 1 & 0.01 & 0.99 & 0.82 \\  & KNN & 0.98 & 1 & 1 & 0.97 & 1 & 1 & 0.75 & 1 & 0.77 & 0.79 & 0.91 \\  & LR & 0.36 & 0 & 0 & 0.63 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\  & DT & 1 & 1 & 1 & 1 & 0.97 & 1 & 1 & 1 & 1 & 1 & 1 \\  & RF & 1 & 1 & 1 & 1 & 0.98 & 0.99 & 1 & 1 & 1 & 1 & 1 \\ \hline \multirow{4}{*}{**Security Camera**} & NB & 0.99 & 0.93 & 0.17 & 0.67 & 0 & 1 & 0 & 0.99 & 0.07 & 1 & 0.49 \\  & KNN & 0.98 & 1 & 1 & 0.98 & 1 & 1 & 0.79 & 0.99 & 0.71 & 0.92 & 0.84 \\  & LR & 0.82 & 0 & 0 & 0.64 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\  & DT & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\  & RF & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \hline \multirow{4}{*}{**Webcam**} & NB & 0.99 & 0.99 & 0.04 & 0.68 & 0 & 1 & 0.02 & 0.99 & 0.21 & 0.97 & 0.83 \\  & KNN & 0.96 & 1 & 1 & 0.98 & 1 & 1 & 0.72 & 1 & 0.76 & 0.69 & 0.89 \\ \cline{1-1}  & LR & 0.95 & 0 & 0 & 0.64 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ \cline{1-1}  & DT & 1 & 1 & 1 & 1 & 0.94 & 1 & 1 & 1 & 1 & 1 & 1 \\ \cline{1-1}  & RF & 1 & 1 & 1 & 1 & 0.97 & 0.97 & 1 & 1 & 1 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: F1-score of each model based on the ML model. Figure 3: The F1-score of each attack type in naïve Bayes-based model. In KNN-based detection, the F1-score was universally high for Bashlite attacks, as shown in Figure 4, but in the case of the Mirai attack, the F1-score was always about 0.7 to 0.9, except for scan.