## 3 Methodology
### Ensemble Learning  
Ensemble learning has become a hot subject recently. Many studies have demonstrated that ensemble learning performance is superior to a single classifier [68, 69]. Ensemble machine learning algorithms combine several learning model predictions into a single \"ensemble\" model to maximize their efficiency [68]. Predictions that are good in various ways will result in a more stable and often better prediction than any individual member. Bagging, boosting, and stacking are traditional approaches to ensemble learning [70]. Since the classification method of ensemble learning often obtains better accuracy of classification than an individual classifier. However, the enhancement comes at the cost of computational time [71]. In essence, ensemble learning is proposed to decrease the randomness of the outcome of a one-time prediction. Multiple predictors can be obtained in two different primary ways, which are: (i) by using various algorithms to process the same dataset, and (ii) using various machine learning algorithms with a change of its hyperparameters. In general, changing algorithms' hyperparameters is crucial for optimization and is not related to integration. Therefore ensemble prediction models can be divided into two types; namely, homogeneous and heterogeneous ensemble models [72], which correspond to (i) and (ii), respectively. Training each model on various training data folds is one way of achieving differences between models. Models are naturally trained on various subsets of training data using resampling methods, such as cross-validation and bootstrap [73]. For a deep neural network model, learning the weights involves solving a high-dimensional problem of non-convex optimization [74]. The difficulty in solving this problem of optimization is that there are many \"good\" solutions, and the learning algorithm can bounce around and settle in one of them [75]. This is referred to as convergence with a stochastic optimization solution, where a collection of unique weight values defines a solution. A type of heterogeneous ensemble strategy is the stacking of ensemble learning. The heterogeneous ensemble combines many different base classifiers into a robust meta-classifier to increase the strong classifier's generalization potential. The ensemble approach takes advantage of both the base classifiers and the meta-classifier learning abilities, significantly improving the accuracy of the classification and the correctness rate, as shown in Algorithm 1. ```
1:Input: Training data \(D=\{x_{i},y_{i}\}_{i=1}^{m}\)
2:Output: ensemble meta-classifier \(H\)
3:Step 1: learn baseline classifiers
4:For\(c=1\) to \(n\)Do:
5:Learn \(h_{t}\) based on \(D\)
6:End For
7:Step 2: build up new dataset for prediction
8:For\(i=1\) to \(m\)Do:
9:\(D_{h}=\{x^{\prime}_{i},y_{i}\}\), where \(x^{\prime}_{i}=\{h_{1}(x_{i},\cdots,h_{T}(x_{i}))\}\)
10:End For
11:Step 3: learn meta-classifier
12:Return

### Proposed Architecture

Ensemble models can achieve lower generalization problems than individual models, but they are difficult to construct with deep neural networks because of the computational cost of training each model. Alternatively, multiple model snapshots should be trained during a single training, and predictions should be combined to create the ensemble classifier. However, this method has some drawbacks, such as the saved models being identical, leading to similar prediction errors and predictions that do not have many benefits from combining their predictions. The use of an aggressive learning rate forces significant changes in the model weights, and the essence of the model saved at each snapshot is one approach to promoting various models saved during a single training run. Several callbacks were used at some frequencies to monitor and track models during training. We have used checkpointing to save the network weights only when the validation dataset increases the classification accuracy. It enables us to monitor the model at the end of the epoch or batch during training.