## 4 Experiments and Results  
The models were implemented using the Python environment (ver. 3.9.7) and several libraries were used such as Sklearn library to implement the classification models. Additional libraries were used, such as NumPy (ver. 1.19.5) for processing numeric and array data, Pandas (ver. 1.3.4), it is a data frame and is used for data management, and Matplotlib (ver. 3.4.3) for data visualization. While XAI was implemented using SHAP (ver. 0.41.0) for global explanation and LIME (ver. 0.2.0.1) for local explanation. The dataset used in this study was split into training and testing by the hold out method in a ratio of 75:25. We utilized several classification models such as DT, RF, GB, AB, XGBoost, LGBM, and CB to detect attacks on HVAC systems based on log information. However, for the feature selection step forward feature selection algorithm was used. AUC, ACC, precision, recall, and F1 scores was used to evaluate the proposed model's performance. Furthermore, grid search method was used for model optimization. Several experiments were performed to investigate the effects of feature selection and data sampling using SMOTE oversampling and TomekLink undersampling. The following list of experiments was performed.  
* Experiment I: Training and testing the classifiers using all the features in the original dataset
* Experiment II: Training and testing the classifiers using the selected features and the original dataset
* Experiment III: Training and testing the classifiers using the selected features and the SMOTE oversampled dataset
* Experiment IV: Training and testing the classifiers using the selected features and the TomekLink undersampled dataset  
In general, the XGBoost classifier performed best compared to the other classifiers. In addition, most of the proposed classifiers have achieved results that have outperformed the benchmark study. Table 6 shows the test result of the classifiers using the complete dataset. While Table 7 contains the results of experiment II.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **AUC** & **ACC** & **Recall** & **Precision** & **F1** \\ \hline DT & 0.9880 & 0.9962 & 0.9781 & 0.9746 & 0.9763 \\ RF & 0.9990 & 0.9923 & 0.9052 & 0.9987 & 0.9496 \\ GB & 0.9583 & 0.9822 & 0.7813 & 0.9969 & 0.8760 \\ AB & 0.9986 & 0.9915 & 0.9137 & 0.9792 & 0.9453 \\ XGBoost & 0.9992 & 0.9991 & 0.9915 & 0.9976 & 0.9945 \\ LGBM & 0.9990 & 0.9995 & 0.9964 & 0.9976 & 0.9970 \\ CB & 0.9957 & 0.9914 & 0.9016 & 0.9907 & 0.9440 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Experiment I: Testing results of the proposed classifiers using all the features in the original dataset.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Feature Selection** & **No. of Features** & **AUC** & **ACC** & **Recall** & **Precision** & **F1** \\ \hline DT & & & 0.9045 & 0.9733 & 0.8226 & 0.8410 & 0.8317 \\ RF & & & 0.9714 & 0.9814 & 0.7679 & 1.0000 & 0.8687 \\ GB & & & 0.9429 & 0.9673 & 0.5966 & 0.9939 & 0.7456 \\ AB &
\begin{tabular}{c} Step Forward \\ feature selection \\ \end{tabular} & 24 & 0.9601 & 0.9690 & 0.6440 & 0.9550 & 0.7692 \\ XGBoost & & & 0.9891 & 0.9835 & 0.8153 & 0.9753 & 0.8882 \\ LGBM & & & 0.9865 & 0.9830 & 0.7983 & 0.9880 & 0.8831 \\ CB & & & 0.9426 & 0.9769 & 0.7339 & 0.9711 & 0.8360 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Experiment II: Testing results of the proposed classifiers after applying step forward feature selection.