# Tree-Based Classifier Ensembles for PE Malware Analysis: A Performance Revisit
## 4 Result and Discussion
### Comparison Analysis  
In the experiment, we employed a \(k\) cross-validation technique (\(k=10\)), where the final performance outcome for each tree-ensemble model is the mean of the ten folds. The performance of each model was measured based on six performance metrics, such as accuracy, MCC, precision, recall, AUC, and F1. These metrics are chosen to provide more accurate estimates of the behavior of the classifier ensembles under the experiment. Especially, [PERSON] et al. [50] have shown that MCC is more informative than accuracy and F1, which yield reliable estimates when used to balanced datasets, but misleading outcomes when applied to imbalanced data sets. For a binary classification problem, the outcome of a tree-based classifier ensemble is typically derived from a contigency matrix, \(\mathcal{T}= \begin{pmatrix}TP&FN\\ FP&TN\end{pmatrix} \), where \(TP\) is true positive, \(FN\) is false negative, \(FP\) is false positive, and \(TN\) is true negative. Let \(\xi_{+}=TP+FN\) and \(\xi_{-}=TN+FP\) be the number of samples labeled as malware and non-malware, respectively. Hence, the performance metrics used in this study can be calculated as follows. \[\text{Accuracy}=\frac{TP+TN}{\xi_{+}+\xi_{-}} \tag{3}\]  
\[\text{MCC}=\frac{TP\times TN-FP\times FN}{((TP+FP)\times\xi_{-}\times(TN+FN) \times\xi_{+})^{1/2}} \tag{4}\]  
\[\text{Precision}=\frac{TP}{TP+FP} \tag{5}\]  
\[\text{Recall}=\frac{TP}{\xi_{+}} \tag{6}\]  
Figure 3: Two-dimensional visualization of instance pairs using t-SNE technique of each malware dataset: (**a**) BODMAS, (**b**) Kaggle, and (**c**) CIC-MalMem-2022. \[\text{AUC}=\int_{0}^{1}\text{Recall}\times\frac{FP}{\xi_{-}}d\frac{FP}{\xi_{-}}= \int_{0}^{1}\text{Recall}\times\left(\frac{FP}{\xi_{-}}\right)^{-1}(x)dx \tag{7}\]  
\[\text{F1}=\frac{2 TP}{2 TP+FP+FN} \tag{8}\]  
Figure 4 presents the performance score of each algorithm on each dataset. Overall, considering MCC as a performance indicator, LightGBM is the worst-performing algorithm, while XGBoost (native) is the best-performing on BODMAS and Kaggle datasets, followed by GBM (H2O). Interestingly, random forest has also performed well on the remaining dataset. Using accuracy as a performance metric, it is also apparent that there are modest performance disparities amongst algorithms (e.g., all algorithms achieve 100% accuracy). Consequently, our results support the findings stated by [50]. In Table 7, we provide the performance average of each algorithm over various datasets and demonstrate that XGBoost (native) is superior to any competitors on the board in terms of accuracy, MCC, and precision metrics. On the other hand, when recall, AUC, and F1 metrics are utilized, GBM (H2O) shows a superior performance. This section includes a two-step statistical significance test using Quade omnibus test and Quade post-hoc test [51] to better comprehend the performance difference between tree-based ensemble models. Using a significant threshold \(\alpha=0.05\) and MCC as a performance indicator, the Quade omnibus test demonstrates that at least one classifier performs differently than others (\(p\)-value = 0.01725). Since we found significance in the previous test, we then applied the Quade post-hoc test to determine the pairwise performance difference between classifiers. Here, we considered XGBoost (native) as a control classifier for comparison with the remaining algorithms. Table 8 depicts the \(p\)-value of the pairwise \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Ensemble Algorithms** & **Accuracy** & **MCC** & **Precision** & **Recall** & **AUC** & **F1** \\ \hline CatBoost & 0.9940 & 0.9851 & 0.9943 & 0.9856 & 0.9988 & 0.9899 \\ XGBoost (native) & 0.9968 & 0.9922 & 0.9975 & 0.9923 & 0.9994 & 0.9949 \\ LightGBM & 0.9927 & 0.9823 & 0.9945 & 0.9828 & 0.9977 & 0.9885 \\ Random forest & 0.9961 & 0.9906 & 0.9959 & 0.9921 & 0.9994 & 0.9940 \\ GBM (H2O) & 0.9967 & 0.9920 & 0.9964 & 0.9978 & 0.9995 & 0.9971 \\ XGBoost (H2O) & 0.9960 & 0.9902 & 0.9956 & 0.9977 & 0.9994 & 0.9967 \\ \hline \hline \end{tabular}
\end{table}