# KRDroid: Ransomware-Oriented Detector for Mobile Devices Based on Behaviors
## 5 Evaluation
### Experiments
#### 5.3.1 RQ1: What Is the Detection Effect of KRDroid?  
In this experiment, we took \(D_{1}\) and \(D_{3}\) as the dataset for testing. The test dataset contains 1862 ransomware and 1697 goodware applications from reference [24; 25; 26; 36; 37; 38]. In order to better evaluate the capability of KRDroid, we compared KRDroid with two ransomware-oriented detectors, HellDroid [36] and R-PackDroid [38]. HellDroid is a well-known ransomware-oriented detector, and we reproduced HellDroid from reference [36]. R-PackDroid is an on-device Android ransomware-oriented detector, and it can be download from reference [38]. The detailed result of this experiment is shown in Table 6.  
HelDroid correctly identified 1558 ransomware and 1397 goodware applications. The accuracy of HelDroid is 83.03%. R-PackDroid correctly identified 1692 ransomware and identified 1613 goodware applications. The accuracy of R-PackDroid is 92.86%. KRDroid correctly identified 1809 ransomware and 1655 goodware applications. The accuracy of KRDroid is 97.33%. The precision, recall, and F1-score of KRDroid are also higher than the two detectors.  
We randomly sampled 46 true negatives and further analyzed the result of HelDroid. After the real machine test and decompile analysis, we found that there were 28 samples in 46 true negatives cannot be detected because of the unseen languages. Nine ransomware applications in the rest of the true negatives cannot be detected because of the unsuccessful lock detection. All of these samples had already been detected as sensitive text. In addition, we found that there were four samples in these nine ransomware applications that belong to screen resource control ransomware. As we mentioned before, this kind of ransomware does not need some real _lock APIs_ such as _lockNow()_ to reach their goals. The last nine ransomware applications that are misjudged are true negatives, which had not been detected.  
The goal of R-PackDroid is to use a compact set of information more than enough to detect a wide variety of samples [22]. When building the detector, it uses the system API package list to represent the application rather than building multidimensional attack-pattern-based features. To some extent, it may cause some misjudgments because of the lack of effective information.  
In addition, as is aforementioned, ransomware in \(D_{1}\) is in the period of 2014-2021.6. KRDroid has good performance on identifying unseen ransomware in this experiment, which means that KRDroid is still valid when facing the latest samples in 2021.  
**Insight**. Due to the accurate characterization and comprehensive behavior-based features build of ransomware applications, KRDroid can detect ransomware by analyzing source code. It detects ransomware by means of detecting ransom behaviors. In this way, national languages requirements do not need to be taken into consideration during detection.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Positive(TP+TN)**} & \multirow{2}{*}{**Negative(FP+FN)**} & \multirow{2}{*}{**Accuracy**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**F1-Score**} \\ \cline{2-2} \cline{5-6}  & **Ransomware** & & & & & & \\ \hline HelDroid & 1558 & 1397 & 604 & 83.03\% & 83.67\% & 83.85\% & 83.76\% \\ R-PackDroid & 1692 & 1613 & 254 & 92.86\% & 90.87\% & 95.27\% & 93.02\% \\ KRDroid & 1809 & 1665 & 95 & 97.33\% & 97.15\% & 97.73\% & 97.44\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: The comparison results of KRDroid, R-PackDroid, and HELDORID.  
KRDoid has good generalization. It can identify unseen ransomware similar to the training samples and can also identify unseen ransomware applications after they have already evolved. To some extent, it can also show that our analysis and behavior-based feature extraction of ransomware applications is valuable.