# AIS for Malware Detection in a Realistic IoT System: Challenges and Opportunities
## 3 NPS Simulations and Evaluation
### Performance Analysis of NPS  
In this section, we first summarize the performance of published malware detection algorithms using both UNWS datasets followed by the NPS simulations results. First, the authors in [30] presented the LSTM-based unsupervised deep learning model for malware detection in the IoT. The authors reported a 96% detection rate accuracy for running their model using both the Bot-IoT and UNSW-NB15 datasets combined. The main focus of this work is on detecting DoS and DDoS attacks. Second, the method presented in [31] achieved a 99% detection accuracy rate using the UNSW-NB15 dataset; however, the number of classes used for the classification was unclear. Next, the method presented in [32] achieved a 70% detection accuracy rate using the UNSW-NB15 dataset. The results were obtained using only 10% of the total number of records in the dataset. Similarly, the authors in [33] used only 10% of the total number of records in the UNSW-NB15 dataset and reported a 89% detection rate accuracy. Finally, the feed-forward ANN (FNN) and self-normalized neural network (SNN) presented in [34] used the Bot-IoT for their experiment. They reported a 95% detection accuracy rate for the FNN and 91% for the SNN. They only used 20% of the total number of records and 10 best features to run the proposed solutions. We ran the NPS algorithm with each of the UNSW datasets and reproduced the results with the NSL-KDD dataset under two scenarios: 20 detectors (see Figure 2) and 30 detectors (see Figure 3). Details of the implementation and a description of detectors are available in [9]. The detection accuracy rate was calculated as in Equation (1) presented in [9] where:  
* True positive (TP): malware is detected as a malicious application;
* True negative (TN): a benign application is detected as a nonmalicious application;
* False positive (FP): a benign application is detected as a malicious application;
* False negative (FN): malware is detected as a nonmalicious application. \[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{1}\]  
In addition, we calculated the precision (Equation (2)), recall (Equation (3)), and F1-score (Equation (4)) (as presented in [9]) to underline the performance of the algorithm regardless of the imbalance ratio in each dataset. \[Precision=\frac{TP}{TP+FP} \tag{2}\]  
\[Recall=\frac{TP}{TP+FN} \tag{3}\]  
\[F1\text{-}Score=2*\frac{Precision*Recall}{Precision+Recall} \tag{4}\] \begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{2}{c}{**UNSW-NB15 Dataset Features**} \\ \hline Total number of records & over 25,000,000 \\ \hline Attack files & 321,283 \\ \hline Number of benign files & 2,218,761 \\ \hline List of attacks & Fuzzers, analysis, backdoors, \\  & DoS, exploits, generic, \\  & reconnaissance, shellcode and worms \\ \hline Number of traffic features & 49 \\ \hline  & Destination port, flow duration, average size of \\  & packet, number of forward packets per second, \\  & number of backward packets per second \\ \hline \hline \end{tabular}
\end{table} 
Table 3: UNSW-NB15 Dataset. The figure below (Figure 2) shows the performance analysis of running the NPS algorithm using the NSL-KDD, Bot-IoT, and UNSW-NB15 datasets with 20 detectors in each set, \(C_{Pos}\) and \(C_{Neg}\). The NPS algorithm achieves high detection performance using all three datasets. The detection accuracy is up to 92%, the precision rate is up to 95%, and the recall rate is up to 97%. The results of creating 30 detectors in each set, negative and positive, using multiple datasets are shown in Figure 3. The detection accuracy goes up to 99%, the precision rate is 99%, and the recall rate is close to 100%. Figures 2 and 3 show a negligible gap when running the NPS using the three datasets. This difference is justifiable by the different number of records and types of attacks in each of the datasets. Therefore, this performance analysis shows the effectiveness of the NPS algorithm in detecting unknown malware attacks in IoT systems.