# An Adaptive Privacy Protection Method for Smart Home Environments Using Supervised Learning
## 5 Evaluation
### Experiment Setup  
We used the public dataset related to accurate activity recognition in a home setting [16] in the experiment. We also downloaded several datasets from WSU CASAS (Center for Advanced Studies in Adaptive Systems) as the sample datasets, to calculate the FDR for supervised learning in Algorithm 2.  
The experiment was carried out in a macOS environment and a PHP language was used to process the original datasets, that are in the formats of txt or dat. To use the datasets conveniently, we used PHP to convert the datasets into a uniform format and stored the results in MySQL. The details of the datasets are listed in Table 1, which includes the sensor number, trigger time, sensor status, and real event. For example, M17 was triggered at 12:49:52 and the event is wash_hands begin. M17 was triggered again at 12:50:42 and the event is wash_hands end.  
In evaluating the effectiveness of privacy protection, we propose using ACA, the average clustering accuracy. This is because, at Tier-1, the FATS attack would try to cluster the sensor data, which plays a critical role in the attack. Should a sensor node be clustered into the wrong group, the predicted ADL would be inaccurate [17], resulting in an ACA which falls within the range [0, 1]. The closer ACA gets to 1, the closer the clustering results will be to equaling the number of rooms and the distribution of the sensor nodes. Conversely, when the clustering result is inconsistent with the actual sensor distribution in the rooms, the ADL will be adequately protected.  
We also propose using FVR, which is the result of the total number of fake messages divided by the total number of real messages. Since it is hard to compare the three models under different conditions, we can use FVR to unify the main influencing factors. We compared the delay, energy consumption, and effectiveness of protection by the three models, through changing the FVR.  
To compare the energy consumption between the three models, we compared the FVR under the condition that the same level of privacy protection is provided. Because all of the three models take a noise-based approach, the more noise in the WSN, the higher the energy consumption [18]. When considering the energy consumed by running the algorithm, it is generally recognized that energy consumption by wireless communication is much higher than the energy consumption by computation. It has been found that 3,000 instructions are needed for a sensor node to transmit 1 bit of data over a distance of 100 meters [19]. Therefore, we can ignore the energy consumption of the algorithm in our experiment.  
In the evaluation of our model, the sample datasets were analyzed to derive the FDR and the time window. Then, through a number of iterations in supervised learning that would update \(\theta\), the transmission of real and fake data was collected by the sink node. Finally, we measured the FATS attack in Tier-1 to calculate the ACA.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**ID** & **Sensor Number** & **Trigger Time** & **Sensor Status** & **Real Event** \\ \hline
1 & M17 & 2008-02-27 12:49:52 & ON & Wash\_hands begin \\
2 & M16 & 2008-02-27 12:49:54 & ON & - \\
3 & AD1-B & 2008-02-27 12:50:01 & 0.302934 & - \\
4 & M17 & 2008-02-27 12:50:42 & OFF & Wash\_hands end \\
5 & M19 & 2008-02-27 12:51:01 & ON & Cook begin \\... &... &... &... &... \\
6 & T10 & 2008-02-27 13:07:14. & 26.5 \({}^{\circ}\)C & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of the datasets.