# Machine-Learning-Based Android Malware Family Classification Using Built-In and Custom Permissions
## 5 Evaluation
### Effect of Classifiers and Custom Permissions  
We trained several machine learning classifiers and measured their metrics, as suggested in Section 4.3. The experiments were performed using the four feature sets (permission sets) listed as below. The features sets were formed by excluding 0-importance permissions and/or custom permissions from the 96 permissions. These experiments were planned to find out which permission set was most effective and which classifier performed best. * S96: 96 permissions;  
Figure 3: Feature importance of the top 10 permissions. S87: 87 permissions (excluding custom permissions);
* S64: 64 permissions (excluding 0-importance permissions);
* S56: 56 permissions (excluding custom permissions and 0-importance ones). We conducted _balanced_ 5-fold cross-validation in our experiments. As explained in Section 4, we split each malware family into 5 subsets and distributed them to the folds evenly. Therefore, the ratio of a malware family in any fold was nearly equal to the ratio of the malware family in the whole dataset. Figures 4-7 show the performance of the classifiers with permission sets S96, S87, S64, and S56, respectively (see Appendix A, Tables A1-A4 for the measured values). On the whole, boosting algorithms (AdaBoost, XGBoost, and LightGBM) performed better than other algorithms. AdaBoost achieved the highest precision, F1-score, ACC, and MCC for S96. LightGBM achieved the highest recall, F1-score, ACC, MCC, and BAC for S87. In many cases, XGBoost lied between AdaBoost and LightGBM. On the other hand, BAC is a metric suitable for imbalanced datasets as explained in Section 4.3. In all cases, i.e., for all classifiers and permission sets, BAC was lower than ACC. This implies that there were _small_ families that included many misclassified instances. We look into the family-level performance in Section 5.3. Table 3 summarizes the experiment results. It shows the most effective classifiers and their measured metrics for each permission set. AdaBoost achieved the highest precision, F1-score, ACC, and MCC, and LightGBM achieved the highest recall and BAC over all permission sets. LightGBM outperformed AdaBoost for permission sets S87, S64, and S56. For permission set S96, however, AdaBoost outperformed LightGBM. The achieved highest MCC was 0.9484 by AdaBoost, and the highest BAC was 0.9198 by LightGBM. Figure 4: Performance of classifiers with permission set S96. Figure 5: Performance of classifiers with permission set S87. Note that the highest values were achieved with S96 and S64, which were the permission sets containing custom permissions. To check whether custom permissions contributed to Android malware family classification, we wanted to statistically compare the MCC and BAC when only built-in permissions were used and when both built-in permissions were used and when both built-in permissions were used. Figure 6: Performance of classifiers with permission set S64. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **S96** & **S87** & **S64** & **S56** \\ \hline \multirow{2}{*}{Precision} & 0.9326 & 0.9190 & **0.9351** & 0.9198 \\  & (AdaBoost) & (AdaBoost) & (AdaBoost) & (AdaBoost) \\ \hline \multirow{2}{*}{Recall} & 0.9192 & 0.9066 & **0.9198** & 0.9056 \\  & (LightGBM) & (LightGBM) & (LightGBM) & (LightGBM) \\ \hline \multirow{2}{*}{F1-score} & **0.9212** & 0.9033 & 0.9173 & 0.9033 \\  & (AdaBoost) & (LightGBM) & (LightGBM) & (LightGBM) \\ \hline \multirow{2}{*}{ACC} & **0.9541** & 0.9421 & 0.9512 & 0.9424 \\  & (AdaBoost) & (LightGBM) & (LightGBM) & (LightGBM) \\ \hline \multirow{2}{*}{MCC} & **0.9484** & 0.9351 & 0.9453 & 0.9354 \\  & (AdaBoost) & (LightGBM) & (LightGBM) & (LightGBM) \\ \hline \multirow{2}{*}{BAC} & 0.9192 & 0.9066 & **0.9198** & 0.9056 \\  & (LightGBM) & (LightGBM) & (LightGBM) & (LightGBM) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Most effective classifiers and measured values. The highest value for each metric is emphasized as bold-face type. Figure 7: Performance of classifiers with permission set S56.