# Intelligent Detection of IoT Botnets Using Machine Learning and Deep Learning
## 4 Experimental Evaluation
### Binary Classification  
The binary classification model considers 10 different detailed Bashlite and Mirai attacks injected into IoT devices as one attack. It also distinguishes between attack or benign states, the latter of which means that the attack is not injected. We train our model using the dataset collected from each device based on the ML and DL models. We design these models using Keras, as well as Scikit-learn. Table 8 describes the design of our models. We then analyze the performance of the models through their F1-score measurements. The F-score is an index expressed as a single value considering both precision and recall, and the F1-score is the value that is given a weighted beta value of 1 for precision when calculating the F-score. The F1-score can be expressed by the following equation:  
\[\text{F1-score}\ =\frac{2\ \times\ \text{precision}\ \times\ \text{recall}}{\text{ precision}\ +\ \text{recall}}\]  
where  
\[\text{precision}\ =\frac{\text{TP}}{\text{TP}\ +\ \text{FP}}\ \text{and recall}\ = \frac{\text{TP}}{\text{FN}\ +\ \text{TP}}.\]  
True positive (TP) is the number of samples that are properly classified as benign. False negative (FN) is the number of samples that falsely detect benign data as a botnet. False Positive (FP) refers to a sample that incorrectly predicts an actual botnet as benign. A True Negative (TN) indicates the number of samples that are properly detected as a botnet. Table 9 shows the detailed detection results of precision, recall, and F1-score for each ML model. Figure 1: Our proposed framework for IoT botnet detection. In Table 9, all models except logistic regression (LR) are able to classify benign and botnet samples with very high performance. For LR, the precision, recall, and F1-score of benign samples are significantly lower than that of attack on all the devices. Thus, errors occur frequently in benign classification. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**ML Models**} & \multirow{2}{*}{**Index**} & \multicolumn{2}{c}{**Doorbell**} & \multicolumn{2}{c}{**Baby Monitor**} & \multicolumn{2}{c}{**Security Camera**} & \multicolumn{2}{c}{**Webcam**} \\ \cline{3-10}  & & \begin{tabular}{c} **Benign** \\ \end{tabular}  & \begin{tabular}{c} **Attack** \\ \end{tabular}  & \begin{tabular}{c} **Benign** \\ \end{tabular}  & \begin{tabular}{c} **Attack** \\ \end{tabular}  & \begin{tabular}{c} **Benign** \\ \end{tabular}  & \begin{tabular}{c} **Attack** \\ \end{tabular}  & \ \begin{tabular}{c} **Benign** \\ \end{tabular}  & \begin{tabular}{c} **Attack** \\ \end{tabular}  \\ \hline \multirow{3}{*}{NB} & Precision & 0.99 & 1 & 1 & 1 & 1 & 1 & 0.98 & 1 \\  & Recall & 1 & 1 & 1 & 1 & 0.99 & 1 & 0.99 & 1 \\  & F1-score & 0.99 & 1 & 1 & 1 & 1 & 1 & 0.98 & 1 \\ \hline \multirow{3}{*}{KNN} & Precision & 0.99 & 1 & 0.99 & 1 & 0.98 & 1 & 0.96 & 1 \\  & Recall & 0.99 & 1 & 0.97 & 1 & 0.97 & 1 & 0.96 & 1 \\  & F1-score & 0.99 & 1 & 0.98 & 1 & 0.98 & 1 & 0.96 & 1 \\ \hline \multirow{3}{*}{LR} & Precision & 0.03 & 0.94 & 0.17 & 0.85 & 0.21 & 0.9 In addition, the naive Bayes (NB) classification in Table 9 corresponds to multinomial NB, which has the best performance out of Gaussian NB, Bernoulli NB, and multinomial NB. As shown in Table 10, Gaussian and Bernoulli NB have a lower detection F1-score than multinomial NB. Therefore, we also used multinomial NB for multi-classification because multinomial NB provides high detection F1-score. Figure 2 shows the results of binary classification based on the three DL models. It can be seen that the CNN model has more than 0.99 F1-score on all devices and offers higher performance than the RNN and LSTM models. LSTM, which has the second highest performance, yields more than 0.99 F1-score for the baby monitor, security camera, and webcam, but only 80% for the doorbell. In the doorbell results, all botnets were accurately detected (except for 2) out of 290,000 botnet samples. However, for benign samples, 5000 samples (comprising about 61% of the 15,000 samples) were incorrectly detected as botnets.