# Tree-Based Classifier Ensembles for PE Malware Analysis: A Performance Revisit
## 3 Materials and Methods
### Tree-Based Ensemble Learning  
The tree-based ensemble is a non-ordinary learning paradigm that constructs and combines a set of base learners (e.g., decision trees or CART) as opposed to the common-place learning paradigm that attempts to construct individual learners from training data. Normally, an ensemble is formed in two processes, i.e., by first producing the base learners and then integrating them. For a decent ensemble, it is commonly considered that the base learners must be as accurate and diversified as possible [18]. This study considers four tree-based ensemble learning algorithms. It is worth mentioning that tuning the hyperparamters for each algorithm is carried out using _random search_ approach [45]. 1. Random forest [19] As its name implies, a random forest is a tree-based ensemble in which each tree is dependent on a set of random variables. The original formulation of random forest algorithm provided by [PERSON] [19] is as follows. A random forest employs trees \(h_{j}(\mathcal{X},\Omega)\) as its base learners. For training data \(\mathcal{D}=\{(\mathbf{x}_{1},y_{1}),\dots,(\mathbf{x}_{a},y_{a})\}\), where \(\mathbf{x}_{i}=(\mathbf{x}_{i,1},\dots,\mathbf{x}_{i,p})^{T}\) represents the \(p\) predictors and \(y_{i}\) represents the response, and a specific manifestation \(\omega_{j}\) of \(\Omega_{j}\), the fitted tree is given as \(\hat{h}_{j}(\mathbf{x},\omega_{j},\mathcal{D})\). More precisely, the steps involved in the random forest algorithm are described in Algorithm 1. We use a fast random forest implementation called _ranger_[46], available in \(R\), which is suitable for high-dimensional data such as ours. The list of random forest's hyperparameters for each malware dataset is provided in Table 2. We set the search space for each hyperparameter tuning is as follows. Number of trees = [50, 100, 250, 500, 750, 1000], split rule = ['gini','extratrees'], minimum node size = {1, 2,..., 10}, \(mtry\) = number of features \(\times\) {0.05, 0.15, 0.25, 0.333, 0.4}, sample fraction = {0.5, 0.63, 0.8}, and \(replace\) = {TRUE, FALSE}. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Hyperparameter** & **BODMAS** & **Kaggle** & **CIC-MalMem-2022** \\ \hline Number of trees & 100 & 1000 & 500 \\ Split rule & ’gini’ & ’gini’ & ’extratrees’ \\ Minimum node size & 4 & 8 & 6 \\ \(mtry\) & 119 & 30 & 18 \\ Sample fraction & 0.63 & 0.80 & 0.63 \\ \(replace\) & FALSE & FALSE & TRUE \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The final learning parameters of random forest used for each dataset after performing a _random search_. XGBoost [23]  
XGBoost is a scalable end-to-end tree-boosting strategy that generates a large number of sequentially trained trees. Each succeeding tree corrects the errors made by the preceding one, resulting in an efficient classification model. Through sparsity-aware metrics and multi-threading approaches, XGBoost not only addresses the algorithm's overfitting problem, but also boosts the speed of most real-world computational tasks. This study utilizes two different XGBoost implementations, such as native implementation in \(R\)[47] and \(H2O\)[48]. We set the search space of native XGBoost's hyperparameters are as follows. Maximum depth = {2, 3,..., 24}, eta = {0, 0.1, 0.2,..., 1.0}, subsample = {0.5, 0.6, 0.7, 0.8}, and column sample by tree = {0.5, 0.6, 0.7, 0.8, 0.9}. Moreover, we set the search space of XGBoost's hyperparameters implemented in \(H2O\) are as follows. Maximum depth = {1, 3, 5,..., 29}, sample rate = {0.2, 0.3,..., 1}, column sample rate = {0.2, 0.21, 0.22,..., 1}, column sample rate per tree = {0.2, 0.21, 0.22,...,1}, and minimum rows = {0, 1,..., log2\(\times\) number of rows-1}. The final learning parameters for both XGBoost implementations are presented in Table 3.  
(ii) CatBoost [24]  
CatBoost is built with symmetric decision trees. It is acknowledged as a classification algorithm that is capable of producing an excellent performance and ten times the prediction speed of methods that do not employ symmetric decision trees.