## 3 Ensemble-Based Methods
Subsequently, the target class \(\hat{y}\) of each unknown instance \(x\) in the test set is computed by Equation (3):  
\[\hat{y}=arg_{j}max\sum_{i=1}^{N}wi,j\chi A(C_{i}(x)=j \tag{3}\]  
where function \(argmax\) returns the value of the corresponding index to the largest value from the array, \(A=[1,2,3,\ldots\ldots M]\) is the set of unique class labels and \(\chi A\) is the characteristic function which takes into account the prediction \(j\in A\) of a classifier \(C_{i}\) on an instance \(x\) and creates a vector in which the \(j\) coordinate takes a value of 1 and the rest takes the value of 0. After several alternatives and improved versions for ensemble classifiers, ensemble methods are nevertheless categorized into three main types, namely boosting, bagging, and stacking. Through the boosting process, a strong learner is built from the collection of different weak learners (weak in the sense of accuracy when applied to the classification task). In boosting-based methods [44], an individual classifier iteratively learns to become specialized on a specific set of the training dataset. The weighted samples from a subset of the training dataset were used to train a component classifier in such a way which emphasizes previously misclassified samples. In boosting methods, experiments piloted on the training set using different learning models to prompt classifiers to produce output. The weight assigning concept is used in the boosting process to assign higher weights or higher costs for each classifier that misclassified the underlying example. Using the approach of the weighted average, the output of each classifier is updated to generate the final output [53]. The basic idea of a boosting-based method is to combine the weak learner to build a strong learner to improve the overall accuracy and performance. Among the boosting family, AdaBoost [54] and Gradient tree boosting [55] are well-known methods of boosting ensemble methods. In a boosting-based method, the resulting ensemble model is defined as the weighted sum of weak learners, as shown in the Equation (4). \[S_{L}(.)=\sum_{l=1}^{L}C_{l}*W_{l}(.) \tag{4}\]  
where \(c_{l^{\prime}s}\) are coefficients and \(w_{l^{\prime}s}\) are weak learners. Using Equation (4) and finding the best ensemble model is somehow a difficult optimization problem. Instead of using this model approach, we can use an iterative optimization process to find all the coefficients and weak learners that give the best overall additive model by adding the weak learners one by one, looking at each iteration for the best possible pair (coefficient, weak learner) to add to the current ensemble model. We recurrently define the \((s_{l})^{\prime}s\) as shown in Equation (5):  
\[S_{l}(.)=S_{l-1}(.)+C_{l}*W_{l}(.) \tag{5}\]  
where \(C_{l}\) and \(W_{l}\) are chosen such that \(S_{l}\) is the model that best fits the training data and therefore is the best possible improvement over \(S_{(l-1)}\). We can then denote this in Equation (6):  
\[(C_{l}*W_{l}(.))=arg_{c,w(.)}minE(S_{l-1(.)})+(C_{l}*W_{l}(.))=arg_{c,w(.)}min \sum_{n=1}^{N}e(y_{n},S_{l-1}(x_{n}))+c*w(x) \tag{6}\]  
where \(E(. )\) is the fitting error of the given model and \(e(.,. )\) is the loss/error function. Thus, instead of \"globally\" optimizing over all the \(L\) models in the sum, we approximate the optimum by optimizing \"locally\" building and adding the weak learners to the strong model one by one. Bagging methods based on bootstrap aggregation minimize the prediction variance by producing additional examples from the original data for the training set. The training of several base classifiers was carried out on the bootstrap instances of a refined subset of the training dataset, and by using the simple aggregation (majority voting), combines the output of these base classifiers into the final output, thus resulting in a more diverse ensemble, a key factor for an ensemble to work efficiently and effectively.