# Towards Deep-Learning-Driven Intrusion Detection for the Internet of Things
## 5 Implementation and Results
### Evaluation Results  
We evaluated the detection system by measuring the performance metrics: recall, precision, and F1 score. Precision (P) and recall (R) are two important metrics used to evaluate detection performance when there is an imbalanced classification, and P-R curve refers to a curve composed of these two metrics. Precision also referred to as the positive predictive value, describes how good a model is at predicting the positive class. Precision can be defined as the ratio of the number of true positives divided by the sum of the true positives and false positives. \[Precision=\frac{TruePositives}{TruePositives+FalsePositives} \tag{8}\]  
Recall is calculated as the ratio of the number of true positives divided by the sum of the true positives and the false negatives. \[Recall(TPR)=\frac{TruePositives}{TruePositives+FalseNegatives} \tag{9}\]  
F1 score is the weighted harmonic mean of the precision and recall and reflects the balance between P and R.  
\[F_{1}=\frac{2*(precision*recall)}{(precision+recall)} \tag{10}\]  
To draw a comparison with existing IDS, we implemented current solution based on inverse weight clustering technique [34] and compared its performance with our proposed detection system. In the following evaluation, DL-Sim refers to the simulation results of the proposed deep-learning (DL)-based IDS, DL Testbed refers to the evaluation results obtained by running deep-learning algorithms on an experimental testbed of IoT devices and IWC refers to the existing IDS solution based on inverse weight clustering. We adopted the same attack data to test the intrusion-detection methods and plotted P-R curves under different attack scenarios. Based on the analysis of P-R curves, we obtain the optimum thresholds and calculate the performance evaluation metrics in different attack scenarios as shown in Tables 2-6. From analyzing the P-R curve for blackhole attacks in Figure 7, we observe that the proposed DL-based IDS has an average precision of 97% compared to 89% precision obtained in the related IWC-based IDS solution. The recall value of our system is however comparable to existing solution. We also note that the proposed IDS demonstrates a higher F1 score than other schemes consistently. From Figure 8, we observe that the proposed DL-based IDS demonstrates higher precision (96%) and recall (98.7%) rate for detecting DDoS attacks in comparison to related work with 91% precision and 95% recall rates. Our scheme also presents higher F1 score of 0.973. Figure 9 shows that the precision for opportunistic service attack normalizes to 95% from the beginning and is slightly lower than the precision value of compared scheme, whereas the recall rates for the systems are comparable. When methods share similar recall values (as presented through TPR), F1-score is an important evaluation metric in determining overall performance. In this case, with comparable TPR values, as F1 score is slightly higher in DL-based IDS, we can conclude that our method performs better. \begin{table}
\begin{tabular}{c c c c} \hline
**Method** & **Precision** & **TPR** & **F1 Score** \\ \hline DL-Sim & 96\% & 98\% & 0.97 \\ DL-Testbed & 93\% & 91\% & 0.92 \\ IWC & 98.37\% & 97\% & 0.97 \\ \hline \end{tabular}
\end{table} 
Table 6: Wormhole Attack Detection. \begin{table}
\begin{tabular}{c c c c} \hline
**Method** & **Precision** & **TPR** & **F1 Score** \\ \hline DL-Sim & 97.2\% & 96.4\% & 0.97 \\ IWC & 89\% & 95\% & 0.92 \\ \hline \end{tabular}
\end{table} 
Table 2: Blackhole Attack Detection. \begin{table}
\begin{tabular}{c c c c} \hline
**Method** & **Precision** & **TPR** & **F1 Score** \\ \hline DL-Sim & 96\% & 98\% & 0.97 \\ DL-Testbed & 98.47\% & 97\% & 0.97 \\ IWC & 98.37\% & 97\% & 0.97 \\ \hline \end{tabular}
\end{table} 
Table 4: DDoS Attack Detection. \begin{table}
\begin{tabular}{c c c c} \hline
**Method** & **Precision** & **TPR** & **F1 Score** \\ \hline DL-Sim & 97.2\% & 96.4\% & 0.97 \\ IWC & 89\% & 95\% & 0.92 \\ \hline \end{tabular}
\end{table} 
Table 3: Opportunistic Attack Detection.