# Challenges of Malware Detection in the IoT and a Review of Artificial Immune System Approaches
## 5 Quantitative Performance Analysis of Leading AIS Methods in IoT Malware Detection
### Detection Accuracy and F1-Score  
The NPS [50] uses both negative and positive detectors, and it overcomes two of the main challenges in securing the IoT applications. First, the method is lightweight, as it generates a smaller number of detectors compared to other AIS algorithms, such as the MNSA [51], with a higher detection rate accuracy, calculated using Equation (1). With 40 detectors in total (20 negative and 20 positive detectors), the NPS achieves up to a 91.92% detection rate, and a rate of up to 99.05% when using 60 detectors in total (30 negative and 30 positive detectors; see Figure 4). When reproducing the results of the MNSA, the detection rate accuracy increases to 80.51% when using 170 detectors in general (150 negative and 20 positive detectors). The mean detection accuracy rate for the NSNN [54] is 73.4%, which is lower than both NPS and MNSA algorithms. Second, it overcomes the false-negative detection challenge. As explained earlier, accuracy alone does not fully capture the detection performance as it does not highlight the false-negatives. In other words, an accuracy of detection of 75% may result from a 100% misclassification of malware (since 25% of the records are labeled as attacks--262.178/1,074,992, as shown in Table 2). To this end, we calculate the F1-Score (see Equation (4)), which is more representative of the performance when the data are not balanced.  
As shown in Figure 4, calculating the F1-Score for the NPS, we obtain a score of 96% when using 40 detectors in total. When using 60 detectors, the F1-Score for the NPS algorithm increases to 99%. The F1-Score for the MNSA increases to 87% when using 170 detectors, and the F1-Score for the NSNN is 73.5%. Overall, the NPS achieves almost a 14% improvement.  
\begin{table}
\begin{tabular}{l l} \hline \hline Total number of records used & 1,074,992 \\ \hline Number of attack files & 262,178 \\ \hline Number of benign files & 812,814 \\ \hline List of attacks & \
\begin{tabular}{l} Brute-force, Heartbleed attack, Botnet, Denial of service, \\ Distributed Denial-of-Service, Web attacks, \\ and infiltration of the network from inside \\ \end{tabular} \\ \hline Number of traffic features & 80 \\ \hline Some of the traffic features &
\begin{tabular}{l} Destination port, flow duration, average size of packet, \\ number of forward packets per second, \\ number of backward packets per second \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 2: NSL-KDD Dataset Used in the Experiment.  
Figure 4: Accuracy and F1-Score results of NPS, MNSA, and NSNN using NSL-KDD dataset.  
We present here a detailed explanation of the concepts used in the performance analysis:  
* True positive (TP): malware is detected as a malicious application;
* True negative (TN): benign software is detected as non-malicious application;
* False positive (FP): benign software is detected as a malicious application;
* False negative (FN): malware is detected as non-malicious application.  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{1}\]  
\[Precision=\frac{TP}{TP+FP} \tag{2}\]  
\[Recall=\frac{TP}{TP+FN} \tag{3}\]  
\[F1-Score=2\times\frac{Precision\times Recall}{Precision+Recall} \tag{4}\]