# IoTBystander: A Non-Intrusive Dual-Channel-Based Smart Home Security Monitoring Framework
## 7 Experimental Results
### Testbed Setup and Evaluation Metrics  
We establish a real-world testbed that includes nine types of smart devices, as shown in Table 6 and Figure 7, and eleven types of dumb devices: microwave, range hood, clock, mouse, keyboard, vacuum cleaner, faucet, toilet, washing machine, doorbell, and gas stove.  
**Sensor selection.** To ensure the consistency and reliability of sensors, we chose standard off-the-shelf sensors that have been widely used in similar research contexts. Specifically, we used the Yue Changsheng USB Mini microphone and the built-in Raspberry Pi wireless network adapter to capture audio signals and wireless traffic, respectively. They were tested to be compatible with the Raspberry Pi, which serves as the main host device of IoTBystander. We manually tested all the sensors before using them and periodically observed the output during the run time to verify that the sensors were in good condition.  
**Data generation and collection.** By manually operating these devices to trigger physical events that make sounds, we create a dataset _HomeSound-13_, which contains  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Device Model (Abbreviation)** & **Wi-Fi** & **Bluetooth** & **Zigbee** \\ \hline Xiaomi Smart Plug (XP) & ✓ & & \\ Xiaomi Smart Camera (XC) & ✓ & & \\ Xiaomi Smart Kettle (XX) & ✓ & & \\ Gosund Smart Plug (GP) & ✓ & & \\ Huawei Chint Plug (HP) & ✓ & & \\ Xiaomi Smoke/Fire Alarm Detector (XD) & & ✓ & \\ Xiaomi Door and Window Sensor (XS) & & ✓ & \\ Xiaomi Gas Detector (XG) & ✓ & ✓ & \\ Xiaomi Smart Multi-Mode Gateway (XM) & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Smart devices in our testbed. Each smart device supports one or more communication protocols.  
Figure 7: Partial smart devices used in our testbed.  
WAV-format audio recordings of 13 distinct event types (see Table 7); each WAV file is labeled with the name of the corresponding event type. This dataset is used to assess the effectiveness of the proposed AER. In addition, we manually trigger physical events _by operating all the smart devices_ and simultaneously capture ambient traffic when these smart devices report the corresponding event notifications digitally. We record the timing for physically triggering the events and label the traffic packets whose timestamps are within the duration of these events. The collected traffic dataset _HomeTraffic-14_ is mainly used to learn the signatures of smart device events offline. To evaluate the performance of TER, we choose to physically trigger events and observe the output of TER in real time.  
To assess the effectiveness of IoTBystander, we use the following metrics:  
* \(Precision_{i}\)=\(\frac{TP_{i}}{TP_{i}+FP_{i}}\), which indicates the proportion of samples predicted as \(class_{i}\) that are indeed \(class_{i}\).
* \(Recall_{i}\)=\(\frac{TP_{i}}{TP_{i}+TN_{i}}\), which denotes the ratio of instances that are actually \(class_{i}\) and are accurately predicted as \(class_{i}\).
* \(F1\)-\(score_{i}\)=\(\frac{2 Precision_{i}\times Recall_{i}}{Precision_{i}+Recall_{i}}\), representing the harmonic mean of \(Precision_{i}\) and \(Recall_{i}\), and serving as a comprehensive metric for evaluating the performance of the multi-class model.
* \(Accuracy\)=\(\frac{\sum_{i}TP_{i}+\sum TN_{i}}{Total\ Samples}\), which reflects the ratio of correctly predicted instances across all classes to the total number of samples.