## 4 Experiments and Results
### Handling Data Imbalance  
In the domain of ML, classification problems are extensively investigated. The input data or predictor is assumed to predict the class label in the classification task, when the target or output is a categorical variable. One of the most common classification problems is to have an imbalanced class dataset where the number of observations for one of the target class labels differs significantly from other class labels. The current HVAC attack detection dataset suffers from imbalance. There are two class labels in the dataset, namely the class '0', which refers to 'not attack', and is extremely smaller than class '1', which refers to 'attack', since the difference between them exceeds 30,000. The dataset used in the current study has an extrinsic imbalance type that occurs when a dataset or data analysis is limited by storage, time, or other factors. To deal with the imbalanced dataset, various resampling techniques was experimented such as SMOTE oversampling and TomekLink undersampling. In oversampling, the synthetic data for the minority class is generated such that the number of samples in the minority class will be equal to the number of samples in majority class. However, undersampling reduces the number of samples for the majority class is reduced to balance the dataset. Table 8 contains the testing result of the proposed classifiers using SMOTE oversampling with selected features. Table 9 shows the test results of the proposed models with TomekLink undersampling. After applying over-sampling and under-sampling techniques, there was a noticeable improvement in the results for all the classifiers in general. They achieved results that outperformed the benchmark study. However, the best performing classifiers were the XGBoost and LGBM. The difference between them is that when oversampled with the SMOTE technique, the XGBoost results were AUC: 0.9999, ACC: 0.9998, recall: 0.9996, precision: 1.0000, and F1: 0.9998. while the LGBM results were AUC: 1.0000, ACC: 0.9997, recall: 0.9994, precision: 1.0000, and F1: 0.9997. In contrast, the XGBoost results when undersampled with TomekLink were AUC: 1.0000, ACC: 0.9991, recall: 0.9912, precision: 0.9975, and F1: 0.9943. While LGBM results were AUC: 1.0000, ACC: 0.9993, recall: 0.9924, precision: 0.9987, and F1: 0.9956. In conclusion, after the experiment it was found that the XGBoost classifier combined with the oversampling technique achieved the best results (see Figure 2). \begin{table}
\begin{tabular}{c c c c c c} \hline
**Classifier:** & **AUC** & **ACC** & **Recall** & **Precision** & **F1** \\ \hline DT & 0.9980 & 0.9980 & 0.9979 & 0.9982 & 0.9980 \\ RF & 1.0000 & 0.9994 & 0.9992 & 0.9997 & 0.9994 \\ GB & 0.9976 & 0.9769 & 0.9628 & 0.9909 & 0.9766 \\ AB & 0.9986 & 0.9835 & 0.9779 & 0.9891 & 0.9835 \\ XGBoost & 0.9999 & 0.9998 & 0.9996 & 1.0000 & 0.9998 \\ LGBM & 1.0000 & 0.9997 & 0.9994 & 1.0000 & 0.9997 \\ CB & 0.9986 & 0.9864 & 0.9855 & 0.9874 & 0.9864 \\ \hline \end{tabular}
\end{table} 
Table 8: Experiment III: Testing results of the classifiers after SMOTE algorithm using selected features. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **AUC** & **ACC** & **Recall** & **Precision** & **F1** \\ \hline DT & 0.9949 & 0.9981 & 0.9912 & 0.9849 & 0.9880 \\ RF & 0.9994 & 0.9928 & 0.9104 & 0.9986 & 0.9524 \\ GB & 0.9606 & 0.9830 & 0.7891 & 0.9952 & 0.8803 \\ AB & 0.9978 & 0.9912 & 0.8965 & 0.9916 & 0.9416 \\ XGBoost & 1.0000 & 0.9991 & 0.9912 & 0.9975 & 0.9943 \\ LGBM & 1.0000 & 0.9993 & 0.9924 & 0.9987 & 0.9956 \\ CB & 0.9963 & 0.9928 & 0.9306 & 0.9775 & 0.9534 \\ \hline \end{tabular}
\end{table} 
Table 9: Experiment IV: Testing results of the classifiers after TomekLinks algorithm using selected features. The XGBoost was applied with the default setting, except for one parameter, namely eval_metric ='mlogloss'.