# Machine-Learning-Based Android Malware Family Classification Using Built-In and Custom Permissions
## 4 Experiment Setup
### Performance Metrics  
A confusion matrix shows how many instances in each actual class are classified into each (predicted) class. An element \(m_{ij}\) of a confusion matrix \(M\) is the number of instances in actual class \(i\) that is predicted as class \(j\). We can represent the confusion matrix as follows:  
\[M_{K\times K}=[m_{ij}]\quad(1\leq i\leq K,1\leq j\leq K), \tag{1}\]  
where \(K\) is the total number of classes. With respect to class \(k\), an instance is said to be positive if its predicted class is \(k\) and negative otherwise. Each instance belongs to one of the following categories with respect to class \(k\):  
* True positive: The actual class is \(k\), and the predicted class is also \(k\). The number of true positive instances is \(TP_{k}=m_{kk}\);
* False positive: The actual class is not \(k\), but the predicted class is \(k\). The number of false positive instances is \(FP_{k}=\left(\sum_{i=1}^{K}m_{ik}\right)-m_{kk}\);
* False negative: The actual class is \(k\), but the predicted class is not \(k\). The number of false negative instances is \(FN_{k}=\left(\sum_{j=1}^{K}m_{kj}\right)-m_{kk}\);
* True negative: The actual class is not \(k\), and the predicted class is also not \(k\). The number of true negative instances is \(TN_{k}=\left(\sum_{i=1}^{K}\sum_{j=1}^{K}m_{ij}\right)-\left(TP_{k}+FP_{k}+ FN_{k}\right)\).  
We can calculate the precision and recall of class \(k\) using Equation (2). Then, accuracy (\(ACC\)) is defined as the sum of true positives divided by the total number of instances. We also calculate balanced accuracy (\(BAC\)) using Equation (4) [20; 21; 49].  
\[Precision_{k}=\frac{TP_{k}}{TP_{k}+FP_{k}},\quad Recall_{k}=\frac{TP_{k}}{TP_{k }+FN_{k}}. \tag{2}\]  
\[Accuracy=\frac{\sum_{k=1}^{K}TP_{k}}{\sum_{i=1}^{K}\sum_{j=1}^{K}m_{ij}}. \tag{3}\]  
\[BAC=\frac{1}{K}\sum_{k=1}^{K}\frac{TP_{k}}{TP_{k}+FN_{k}}. \tag{4}\]  
Micro- and macro-averages are two ways of interpreting confusion matrices in multi-class classification. The micro-average represents the performance of a model by focusing on each sample. On the other hand, the macro-average shows the model's performance by focusing on each class. The macro-average is therefore more suitable for data with an imbalanced class distribution. The macro-averages of precision, recall, and F1-score are calculated as follows:  
\[Precision_{avg}=\frac{1}{K}\sum_{k=1}^{K}Precision_{k},\quad Recall_{avg}= \frac{1}{K}\sum_{k=1}^{K}Recall_{k}, \tag{5}\]  
\[F1\text{-}score=\frac{2\times Precision_{avg}\times Recall_{avg}}{Precision_{ avg}+Recall_{avg}}. \tag{6}\]  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **ANN** & **DNN** & & **Random Forest** & **Extra Trees** & **Ada Boost** & **XGBoost** & **Light GBM** \\ \hline
**Hidden layers** & 0 & 10 & **N estimators** & 100 & 90 & 90 & 100 & 100 \\
**Epochs** & 100 & 100 & **Depth** & & & 100 & & \\
**Batch size** & 128 & 128 & **Max depth** & 90 & 40 & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Parameters for machine learning algorithms.  
We used the following equation to calculate the _MCC_[21; 22; 43]:  
\[MCC=\frac{c\times s-\sum_{k}^{k}p_{k}\times t_{k}}{\sqrt{(s^{2}-\sum_{k}^{k}p_{k}^ {2})\times(s^{2}-\sum_{k}^{k}t_{k}^{2})}}, \tag{7}\]  
where:  
* \(t_{k}=\sum_{i}^{K}m_{ik}\): the number of samples belonging to class \(k\);
* \(p_{k}=\sum_{j}^{K}m_{kj}\): the number of samples predicted as class \(k\);
* \(c=\sum_{k}^{k}m_{ik}\): the total number of samples correctly predicted;
* \(s=\sum_{i}^{K}\sum_{j}^{K}m_{ij}\): the total number of samples.