## 4 Results
### Discussion  
In this study, the IDS is deployed on a smart thermostat built with the ESP32 microcontroller. The microcontroller boasts 440 KB of ROM and 520 KB of SRAM for program memory and instructions, respectively.  
We conducted an initial feasibility test by implementing the IDS, incorporating connection activity, statistical activity, and DNS activity features. Utilizing XGBoost, the IDS consumed 82.71% of program memory and required 3002.78 us to compute the output. Conversely, the IDS implemented with RF consumed 79.0% of program memory and took 1565.01 us for computation. Similarly, the IDS implemented with ANN utilized 76.15% of program memory and took 296.12 as for computation. Notably, DT outperformed XGBoost, ANN, and RF in terms of program memory and inference time, consuming just 66.73% of program memory and requiring only 11.37 us for computation.  
We also implemented the IDS using connection and statistical activity features. The most efficient XGBoost model utilized 76.38% of program storage and took 2056.2 us for computation. Following closely in accuracy, the DT model consumed 65.74% of program memory with an inference time of only 3.2 us. In contrast, RF consumed 75.68% of program memory with an inference time of 1312.4 us. The ANN model, ranking fourth in accuracy, consumed 75.87% of program memory with an inference time of 242.8 us.  
Furthermore, we implemented the IDS with the IDSH dataset, which combines sensor and network datasets. The highest accuracy achieved by XGBoost (estimator = 100, depth = 8) is 97.66% but with an inference time of 3515.9 us, whereas with XGBoost (estimator = 50, depth = 8), the IDS consumed 77.0% of program memory with an inference time of 1502.3 us. The RF model consumed 80.96% of program memory with an inference time of 3314.4 us, while the DT model consumed 66.78% of program memory with an inference time of 8.0 mi  
Figure 4: Comparison of XGBoost-, RF-, DT-, and ANN-based IDS implementation on smart thermostat using IDSH dataset.  
croseconds. The ANN model consumed 75.28% of program memory with an inference time of 2498.1 us. It is noteworthy that for the IDSH dataset, the XGBoost models with estimators of 150 and above encountered memory constraints preventing their implementation on the smart thermostat.  
DT operates on a simpler model, making decisions based on if-else rules at each node, which requires less computation time compared to the matrix multiplications and activity functions utilized in XGBoOST, ANN, and RF. As a result, the inference time of DT is less than that of XGBoOST, ANN, and RF. However, XGBoost captures complex relations in the data and handles non-linear patterns more effectively. Moreover, it possesses the capability to consider both past and future covariates present in the input data. The internal feature of hyperparameter tuning further contributes to its higher accuracy, making XGBoost's accuracy superior.  
Table 9 and Figure 4 present the results of the IDS deployed on the smart thermostat device using our IDSH dataset. The trained ML models were embedded in the ESP32 microcontroller-based smart thermostat, and the inference time was calculated after generating attacks from our Raspberry Pi-based attacker node. Based on these results, we determined that the inference time of XGBoost exceeds that of DT. However, given that the building dynamics are slow and a delay of a few milliseconds in HVAC control input will not significantly impact the indoor environment, we prioritize accuracy. Therefore, for the smart thermostat, the XGBoost (estimator = 50, depth = 8)-based IDS is embedded in the smart thermostat for intrusion detection.