## 2 Background
### Tree-Based Ensembles  
Ensemble learning [7] is a machine learning technique that creates a final prediction model by combining many machine learning classifiers. Ensembles can learn complex data patterns for decision making and class prediction. In software defect prediction, there are two primary types of problem: regression in which the classifier predicts the number of defects, and classification in which the classifier predicts the label of an instance (i.e., defective or non-defective). Ensemble learning has two phases [7]: (1) training the base classifiers which consist of multiple individual classifiers and (2) construct a final prediction model using averaging or voting that are computed from the results of the base classifiers [8]. Ensembles are known to be homogeneous if only one type of algorithm is used in the base classifiers and are considered heterogeneous if various types of algorithm are used as the base classifiers. The main focus of this study is tree-based ensembles, which are boosting and bagging homogeneous ensembles where the base classifier is a decision tree [9]. Boosting [10] is a sequential ensemble method that combines multiple classification methods to create a low-bias model and thus promote better prediction performance. The key idea behind boosting is to construct classifiers iteratively and sequentially, with each model relying on the previous model, taking into account misclassified instances that were predicted by the previous classifiers. There are many boosting ensembles, the key variations being in how the base classifies are trained and combined. Bagging [11] or bootstrap aggregation is a parallel ensemble method that combines several models to produce a model with low variance and hence improved prediction performance. The essence behind bagging is to train multiple base classifiers independently and then combine the outputs by averaging or voting. Bagging's main advantage is the ability to parallelize the process of training the base classifiers, since they are fitted independently. The following are brief descriptions of the tree-based ensembles that were chosen:  
* **AdaBoost (Ada)**[10], or adaptive boosting, is a well-known boosting algorithm introduced by [PERSON] and [PERSON]. AdaBoost fits a series of base classifiers and then changes the weights of the instances by giving the misclassified ones a higher weight and then fitting the updated weights with a new base learner. The final prediction is calculated by integrating the results from all base classifiers using a weighted majority vote approach, in which each base classifier contributes based on its results (i.e., assigned a greater weight). * **Random forest (RF)**[12] is a bagging algorithm that employs a large number of small decision trees, each of which is developed from random dataset subsets. To increase the tree's diversity, a random subset of features is chosen at each node to produce the best split. Overfitting is less likely due to the randomness of the dataset and features. The majority vote is used in a classification problem to determine the final class label. * **Extra trees (ET)**[13], or extremely randomized trees, is similar to the RF algorithm but with additional randomness. ET differs from RF in two ways: (1) each decision tree is constructed using the entire dataset, and (2) it randomly selects the splits at each node (i.e., does not select the best splits). * **Gradient boosting (GB)**[14] is a generalization of the AdaBoost ensemble that can be constructed utilizing a variety of loss functions. GB, unlike AdaBoost, fits a new base classifier using gradients rather than the weight of misclassified instances. Using GB will boost the efficiency of fitting the base classifiers, but memory usage and processing time are inefficient. * **Histogram-based gradient boosting (HGB)**[15], or histogram-based gradient boosting, is a boosting ensemble that selects the best splits easily and reliably using feature histograms. It is more efficient than GB in terms of processing speed and memory utilization.