# A Review of Ensemble Learning Algorithms Used in Remote Sensing Applications
## 2 Principles of Ensemble Learning Algorithms  
### Bagging Algorithms  
Bagging is an ensemble technique that combines multiple learners trained on distinct subsamples of the original data. To build a bagging model, we generate multiple datasets by bootstrapping the training data and then develop models based on the individual datasets and make predictions using these models. All the predictions are then combined to produce a representative value such as the mean, median, or majority vote for classification and averaging, for regression, depending on the problem to be solved. Since an individual learner is often sensitive to noise in the training data, bagging, by aggregating multiple results in a single prediction, should provide stable and improved results with decreased variance [22].  
RF is a modified version of bagging, in which the classification and regression trees (CART) technique is often used as the individual learner. RF uses subsamples of the original data to build trees and randomly selects a subset of variables to determine each split in the tree [23]. RF excludes approximately 30% of training samples from the modeling due to bootstrapping and random subspace techniques and is often used to compute the out-of-bag (OOB) prediction error.  
Many studies have shown that RF outperforms other machine learning and statistical regression algorithms [13]. It has many advantages, some of which are the following. Both classification and regression issues can be resolved using RF. In addition, RF is insensitive to noise or outliers in the training data because it bins them. RF trains rapidly since it incorporates only part of the features for training. RF is easy to use since only one or two hyperparameters (i.e., the number of trees in the forest and the number of predictor variables to be randomly selected at each junction) are required to be tuned. In addition, an intrinsic estimate of the generalization error (the OOB error) is given. RF provides  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Algorithm**} & \multicolumn{2}{c}{**Implementation with R and Python**} \\ \cline{2-3}  & **R Package** & **Python Library** \\ \hline Bagging & ipred & scikit-learn \\ RF & randomForest/caret & scikit-learn \\ ERT & extraTrees & scikit-learn \\ AdaBoost & Adabag/fastAdaboost & scikit-learn \\ GBM & gbm & scikit-learn \\ XGBoost & xgboost/plyr & xgboost \\ LightGBM & lightgbm & lightgbm \\ CatBoost & catboost & catboost/scikit-learn \\ Stacking & stacks & vecstack/scikit-learn \\ \hline \hline \end{tabular}
\end{table}
Table 1: Implementation of ensemble learning algorithms using R and Python.  
a measure of variable importance as identified by the Gini index [24]. Despite these advantages, the spatial correlation of nearby observable data is disregarded by RF. Thus, RF kriging (RFK), which couples RF with residuals interpolated by conventional kriging, was proposed and successfully used for forest biomass mapping [25] and prediction of PM2.5 concentrations [26].  
Extremely randomized trees (ERT) is also a tree-based ensemble method. ERT differs from RF in the following ways: ERT uses all samples to train the base learner instead of using bootstrap resampling; and ERT randomly selects cutting points instead of calculating local optimal cutting points when splitting nodes, which injects more randomness than RF [22]. The results from different individual trees are therefore more diverse. Some studies have suggested that ERT produces more accurate predictions than RF [27]. ERT has been used in a wide variety of applications such as aboveground biomass estimation [28], prediction of ground level PM2.5 concentrations [29], modeling olive tree phenology [30], retrieval of downward longwave radiation [31], streamflow modeling [32], and estimation of terrestrial latent heat flux [33].