# Evaluating Ensemble Learning Mechanisms for Predicting Advanced Cyber Attacks
## 4 Implementation and Pre-Dataset Processing
### Ensemble Learning Mechanism  
The design of the proposed model is illustrated in Figure 2, which begins with a data analysis phase. To ensure consistent data representation, numerical attributes are scaled to eliminate the mean and achieve unit variance. Categorical features are encoded to facilitate the sampling method, which addresses the challenge of imbalanced classification. Identifying a precise and accurate selection of features that directly affect the dependent variable is a critical stage in the process. In Ensemble learning, a number of ML algorithms are selected, therefore, we have selected the following algorithms: Random Forest (RF), Logistic Regression (LR), Decision Tree (DT), K Nearest Neighbours (KNN), Support Vector Machine (SVM), XGBoost (XG), CatBoost (CB), and Artificial Neural Networks (ANNs). Utility functions are used for model tuning by selecting model hyperparameters such as learning rate to obtain accurate prediction in an acceptable learning time. This can be performed automatically by trying different combinations of parameters and values. Grid search and random search methods are used for hyperparameter optimization. Essentially, ensemble learning design entails building a set of multiple machine learning (ML) algorithms that may be used in various ways depending on how the learning process is organized along with the dataset used. We have implemented the proposed  
Figure 2: The proposed Model Process. system using three approaches: bagging, boosting, and stacking to compare the obtained results from each learning procedure. As a result, we have to select which algorithms to use for base models and which ones are used to create the meta-model. Figure 3 illustrates the implementation and evaluation process. It's important to note that the ensemble model maintains multiple models, which might lead to increased complexity and higher processing requirements. Following the selection of ML algorithms, the model structure is designed by arranging these algorithms and how to obtain the final prediction. There are several effective methods for ensemble learning that can be utilized in different structures as shown in Figure 3. Begging (Bootstrapping) ensemble learning is performed by generating multiple samples (bags) of the main datasets. Learning algorithms, also known as base models, are applied independently to each subset of the data. The obtained outcomes are combined using an averaged activity, maximum voting, or any aggregation mechanisms. The goal is to decrease the prediction variance and to improve the predictor force. On the other hand, boosting ensemble learning, also referred to as forward additive modeling, uses a similar concept but applies a sequential procedure to each subset adaptively rather than conducting the training process in parallel. Each training model is based on the results of the previous one in order to boost the performance of every model. Subsets are not created randomly, yet it depends on the performance of the previous model. In the ensemble stacking approach, two levels of the training process are performed, in level 1. The main dataset is used to train the model using heterogeneous learning algorithms (base models). The outcome of level 1 training is used as an input to the level 2 training, also known as a prediction vector, which is obtained from base models. The new dataset with its features is considered as an input for the meta-model which is created to generate the final prediction. The stacking algorithm is shown in Algorithm 1. as an instance of an ensemble mechanism. To implement and evaluate the proposed model, several tools and methodologies were employed. The models were developed using Python and implemented with the Anaconda package and Jupyter Notebook as a development environment. The evaluation process involved several steps. Firstly, the dataset was pre-processed to ensure its compatibility with the models.