## 4 Results
### Real-Time Testing of IDS on Smart Thermostat  
The smart thermostat uploads the data on the AWS cloud server ([[http://16.170.2](http://16.170.2)]([http://16.170.2](http://16.170.2)) 15.67/display). The smart home application provides users with remote access to turn ON/OFF the HVAC through the webpage. This bidirectional communication is vulnerable to DoS and MITM attacks. Using our smart thermostat dataset, we trained XGBoost, DT, RF, and ANN for IDS implementation on the smart thermostat. The testing results of the IDS implemented with XGBoost, RF, DT, and ANN are shown in Table 9. The IDS implemented with XGBoost (depth = 8, estimator = 100) achieved the highest accuracy of 97.66%. The best accuracy achieved with RF (depth is 8, estimators = 20) is 95.33%, with DT (with a depth of 15) it is 96.54%, and with ANN it is 93.41% as shown in Figure 4. The accuracy, F1-score, recall, and precision are summarized under the subfigure titled Accuracy (%), whereas the comparison of program memory requirements, inference time, FPR, and FNR are also shown in Figure 4. The results clearly show that although the inference time of XGBoost is high and it requires more program memory, it outperformed DT, ANN, and RF in terms of accuracy, F1-score, recall, precision, FPR, and FNR. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**ML Model** & \multirow{2}{*}{**Accuracy**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**F1-Score**} & \multirow{2}{*}{**FNR**} & \multirow{2}{*}{**FPR**} & \multicolumn{2}{c}{**Computation**} & \multicolumn{1}{c}{**Program**} \\
**(No. of Estimators)** & & & & & & & **Time \(\mu\)s (Average)** & **Storage** \\ \hline XGBoost (Estimator = 50, Depth = 8) & 99.92\% & 0.99 & 0.99 & 0.35\% & 0.05\% & 1223.4 & 75.0\% \\ \hline
** XGBoost (Estimator = 100, Depth = 8)** & **99.91\%** & **0.99** & **0.99** & **0.30\%** & **0.06\%** & **2056.2** & **76.38\%** \\ \hline XGBoost (Estimator = 150, Depth = 8) & 99.92\% & 0.99 & 0.99 & 0.30\% & 0.06\% & 2480 & 76.8\% \\ \hline XGBoost (Estimator = 200, Depth = 8) & 99.92\% & 0.99 & 0.99 & 0.30\% & 0.06\% & 2936.7 & 77.32\% \\ \hline RF (Estimator = 50, Depth = 8) & 99.52\% & 0.99 & 0.92 & 0.96 & 0.25\% & 0.49\% & 407.9 & 70.57\% \\ \hline RF (Estimator = 100, Depth = 8) & 99.52\% & 0.99 & 0.93 & 0.96 & 0.25\% & 0.48\% & 1312.4 & 75.68\% \\ \hline RF (Estimator = 150, Depth = 8) & 99.53\% & 0.99 & 0.93 & 0.96 & 0.25\% & 0.47\% & 2729.7 & 80.39\% \\ \hline RF (Estimator = 200, Depth = 8) & 99.52\% & 0.99 & 0.93 & 0.96 & 0.25\% & 0.48\% & 3584.2 & 85.08\% \\ \hline DT (Depth = 5) & 99.68\% & 0.99 & 0.96 & 0.97 & 0.60\% & 0.72\% & 3.2 & 65.74\% \\ \hline DT (Depth = 10) & 99.59\% & 0.99 & 0.94 & 0.96 & 0.79\% & 0.38\% & 8.4 & 65.96\% \\ \hline DT (Depth = 15) & 99.58\% & 0.99 & 0.94 & 0.96 & 0.93\% & 0.38\% & 12 & 66.23\% \\ \hline DT (Depth = 20) & 99.59\% & 0.99 & 0.94 & 0.96 & 0.84\% & 0.30\% & 11.4 & 66.34\% \\ \hline ANN (Layer 1 = 32, Layer 2 = 16 ) & 99.24\% & 0.99 & 0.88 & 0.93 & 11.28\% & 0.01\% & 102.3 & 74.67\% \\ \hline ANN (Layer 1 = 64, Layer 2 = 32) & 99.23\% & 0.99 & 0.88 & 0.97 & 11.09\% & 0.01\% & 242.8 & 75.87\% \\ \hline ANN (Layer 1 = 64, Layer 3 = 32) & 99.17\% & 0.98 & 0.89 & 0.93 & 10.78\% & 0.12\% & 478.2 & 76.01\% \\ \hline Layer 2 = 64, Layer 3 = 32) & 99.22\% & 0.99 & 0.89 & 0.93 & 10.83\% & 0.06\% & 3986.2 & 76.15\% \\ \hline \hline \end{tabular}
\end{table} 
Table 8: IDS implementation on edge using connection and statistical activity features of TON_IoT dataset.