## 5 Further Discussion  
The current research focuses on the attacks related to smart HVAC systems, which are considered to be a critical component of the building management system (BMS). We used HVAC system attack detection dataset published by [PERSON] et al. [8], which was acquired using the transient system simulation tool (TRNSYS). Furthermore, the performance of the proposed models was compared to the benchmark study, [8]. According to our findings, most of the classifiers, particularly the XGBoost classifier, outperformed the benchmark study across all the evaluation measures. Table 10 illustrates the comparison of the benchmark study with our proposed study. The results show that the current study outperformed the benchmark study. Furthermore, in the [PERSON] et al. [8] study the performance of the model was evaluated using only two evaluation measures, namely precision and recall. However, in the current study several classification evaluation measures are used to ensure the performance of the proposed models and to provide a comprehensive evaluation. [PERSON] et al. [8] achieved the highest results with IF classification and PCA for feature selection with a recall of 0.6049 and a precision of 0.9001. In the current study the recall was 0.9996 and the precision was increased from 0.9001 to 1.000. In addition, the number of features was reduced using set forward feature selection techniques. The number of reduced features used in the proposed model is reduced to 50% compared to the benchmark study [8]. After performing the step forward feature selection technique, it was found that the highest result was obtained with the XGBoost classifier where AUC: 0.9891, ACC: 0.9835, recall: 0.8153, precision: 0.9753, and F1: 0.8882. As previous shown in Table 1 the comparison of the related studies and in Table 10 the comparison of the proposed model and the benchmark study. [PERSON] et al. [9] used the fusion-based technique for the hidden voice attacks. The study has achieved the success rate of 0.941. The significance of the study was that extensive experiments were performed. However, the experiments were conducted over-the-air. Therefore, the experiments need to be performed in more controlled environment. Moreover, in the study, 40,000 samples \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Study** & **Data Partitioning** & **No. of Features** & **Model** & **AUC** & **ACC** & **Recall** & **Precision** & **F1** \\ \hline
[8] & 5-Fold cross Validation & 50 & IF & - & - & 0.6049 & 0.9001 & - \\ \hline Our Study & Hold out (75–25) & 24 & XGBoost & 0.9999 & 0.9998 & 0.9996 & 1.0000 & 0.9998 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Comparison of the proposed work with the benchmark study. Figure 4: Local Explanation of XGBoost using LIME. were used to train the model while, 1000 samples were used to test model performance. Furthermore, [PERSON] et al. [10] proposed the attack detection for HVAC systems. The nature of the dataset is an imbalance and the study has achieved a recall is 0.80, which is significantly lower than the recall of the proposed model. However, the study by [PERSON] and [PERSON] [11] has performed the unsupervised technique for the attack detection in HVAC system. The study also performed the visual evaluation using the ECG dataset. In addition, some studies have used NSL-KDD dataset, RF has outperformed with an accuracy of 0.9964 [15], while XGBoost has achieved the best results with an accuracy of 0.97. The NSL-KDD dataset contains the list of attacks on the network and is one of the widely used dataset for intrusion detection system. Additionally, UNSW-NB15 is another dataset for the intrusion detection. The dataset was used by [15; 16], the highest accuracy of 0.99 was achieved with KNN. Despite this, the study achieved the significant results, but the KNN model is a lazy learner and require huge testing time. However, the detection time is one of the essential factors in the attack detection systems.