# Difficulties and Challenges of Anomaly Detection in Smart Cities: A Laboratory Analysis
## 5 Anomaly Detection
### Attack Detection with Machine Learning
#### 5.2.4 Test Phase  
In this section, we will show the final results of our study, which have been obtained using the test datasets. As we can see, different samples have been used for the training (i.e., training datasets), different samples to select the parameters of the algorithms (i.e., validation datasets), and different samples to evaluate the models (i.e., test datasets). This reduces the probability of overfitting. Additionally, as Figure 8 shows, the test datasets contain the same proportion of attack and attack-free samples. In this way, we have avoided having skewed datasets that could lead to biased conclusions. Figure 8: Dataset partition. To evaluate the predictive capabilities of the models, we have used the metrics summarized in Table 3, which have been widely used to assess anomaly detection techniques [53]. Each prediction of the model has been classified as a false positive (FP), if the prediction is 'anomalous' and the sample comes from the attack-free period; a true positive (TP), if the prediction is 'anomalous' and the sample comes from an attack period; a false negative (FN), if the prediction is 'normal' and the sample comes from an attack period; and a true negative (TN), if the prediction is 'normal' and the sample comes from the attack-free period. Below, we present the prediction results obtained in our experiments. The next section discusses these results. Figure 9a-c plot the prediction results for the three attacks comparing the two techniques using together all the variables from all the WSNs. The Figures 10-13 plot the prediction results for the three attacks comparing the two techniques creating a different model for each network. \begin{table}
\begin{tabular}{l l l} \hline \hline
**True Positive Rate (TPR)** &
\begin{tabular}{l} Also called detection rate or sensitivity, \\ measures the percentage of properly detected attacks \\ \end{tabular} & TP/(TP + FN) \\ \hline
**False Positive Rate (FPR)** &
\begin{tabular}{l} Also called fall-out, measures the percentage \\ of misclassified normal samples \\ \end{tabular} & FP/(FP + TN) \\ \hline
**F1-score** &
\begin{tabular}{l} Measures the number of true positives over the average \\ of predicted and real positives \\ \end{tabular} & 2 TP/(2 TP + FP + FN) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Metrics summary. Figure 9: **(a)** prediction results for the jamming attack period using the variables from all the wireless sensor networks (WSNs) (training with dataset _tr_all_ and test with dataset _te_all_jam_); **(b)** prediction results for the 50% selective forwarding attack period using the variables from all the WSNs (training with dataset _tr_all_ and test with dataset _te_all_sf50_); **(c)** prediction results for the 75% selective forwarding attack period using the variables from all the WSNs (training with dataset _tr_all_ and test with dataset _te_all_sf75_). Figure 11: (**a**) prediction results for the jamming attack period using the variables from WiFi network (training with dataset _tr_rpi_ and test with dataset _te_rpi_jam_); (**b**) prediction results for the 50% selective forwarding attack period using the variables from WiFi network (training with dataset _tr_rpi_ and test with dataset _te_rpi_jam_); (**c**) prediction results for the 75% selective forwarding attack period using the variables from WiFi network (training with dataset _tr_rpi_ and test with dataset _te_rpi_jf75_). Figure 12: (**a**) prediction results for the jamming attack period using the variables from the SCK network (training with dataset _tr_sck_ and test with dataset _te_sck_jam_); (**b**) prediction results for the 50% selective forwarding attack period using the variables from the SCK network (training with dataset _tr_sck_ and test with dataset _te_sck_g75_).