# Dynamic IoT Malware Detection in Android Systems Using Profile Hidden Markov Models
## 5 Evaluation and Results  
All the experiments were conducted using the following hardware specifications: (iMac) PC desktop, macOS operating system, 3.3 GHz processor speed, 1 TB capacity and 8 GB RAM. Herein, we present the details of how we evaluated our system and we discuss the experimental results based on each performance measure. The proposed solution, DIP, was evaluated using 24 families in the Drebin dataset which represents 30% of the dataset with 1435 malware and 500 benign apps. Table 3 shows the number of samples from each malware family that we used for testing. We chose malware applications randomly from the dataset and performed dynamic analysis to extract the API call sequence that was invoked at run time and then converted the API call sequence to Fasta files that contain the sequence of malware, as we did during  
Figure 5: Sample of ADRD family Fasta file and a sample of MSA ADRD family. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline \# & **Malware Family** & **Samples** & **\#** & **Malware Family** & **Samples** \\ \hline
1 & Adrd & 30 & 13 & Glodream & 19 \\
2 & BaseBridge & 112 & 14 & Hambo & 8 \\
3 & Boxer & 9 & 15 & Iconosys & 56 \\
4 & DroidDream & 21 & 16 & Imlog & 14 \\
5 & DroidKungFu & 205 & 17 & Ijfake & 3 \\
6 & ExploitLinuxLotoor & 23 & 18 & Kwin & 33 \\
7 & FakeDoc & 48 & 19 & MobileTx & 18 \\
8 & FakeInstaller & 276 & 20 & Opfake & 181 \\
9 & FakeRun & 16 & 21 & Plankton & 192 \\
10 & Gappusin & 18 & 22 & SendPav & 10 \\
11 & Geinimi & 7 & 23 & SMSreg & 19 \\
12 & Ginmaster & 112 & 24 & YZHC & 19 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Number of malware samples for testing. the implementation. Following this, we passed the file into PHMM to classify it. The tested malware gave a positive score for the family that has a likeness with the tested sequence and a negative score for other families. To classify malware families, we set the threshold to zero and we classified 1390 out of 1435 malware apps according to their families. For benign apps, we examined the 500 apps, and the system classified 475 apps as benign. The metrics that we used to evaluate the effectiveness of PHMM malware detection are:  
* Detection rate for each malware family according to Equation (1): \[\text{Detection Rate}=\frac{\text{Malware Samples Belonging to a Family }X\text{ and Correctly Classified in That Family}}{\text{Total Malware samples Belonging to Family }X}*100\] (1)
* FP rate which represents the ratio between the error number of unrelated retrieved cases and the total number of the ground truth of unrelated cases. FP rate is calculated according to Equation (2): \[\text{FP Rate}=\frac{\text{FP}}{\text{FP}+\text{TN}}\] (2) where TN is the number of unretrieved unrelated cases. * FN rate which represents the ratio between the unretrieved correct values and the total number of the ground truth of the correct cases FN rate is calculated according to Equation (3): \[\text{FN Rate}=\frac{\text{FN}}{\text{FN}+\text{TP}}\] (3) where TP is the number of the correct cases retrieved. * Precision which represents the ratio between the number of TP and the total number of retrieved cases. High precision means our system gives a more relevant result. Equation (4): \[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}\] (4)
* Recall is the ratio between the number of TP and the total number of correct cases. Recall is calculated according to Equation (5): \[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}\] (5)
* Accuracy is the number of correct cases examined divided by the number of all cases examined. High accuracy means the system can detect malware effectively. Accuracy is calculated according to Equation (6): \[\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{N}}\] (6) where N is the total number of all values. * F-measure, combines recall and precision. The best result of F-measures is the one closest to 1.