# Passive Indoor People Counting by Bluetooth Signal Deformation Analysis with Deep Learning
## 5 Experimental Results  
Experiments were conducted to evaluate the performance of the five models (DenseNN, CNN, LSTM, CNN+LSTM, Transformer) in both classification and regression scenarios. Moreover, we tested the models on the new Fine-Tuning dataset that was not involved in the pre-training phase, before and after a fine-tuning step. For each model, the fine-tuning step consists of performing a short training of the highest three layers of the model, that is, the layers related to the final response of the model. The inner and input layers are instead kept invariant (frozen). The short training and validation are carried out using the Fine-Tuning Training and Fine-Tuning Validation datasets described above, whereas the final test is carried out using the Fine-Tuning Test dataset. Figure 5: (**a**) An unbalanced dataset used in our experiment. (**b**) The effects of the data augmentation approach adopted in our experimental setup. We first tested the three models in the classification setup, both on the Pre-Training Test dataset without fine-tuning and on the Fine-Tuning Test dataset with fine-tuning and 350 samples for each class, and measured the accuracy and F1 performance for each model. We recall that the \(F1\) score is calculated according to Equation (1):  
\[F1=\frac{2*Precision*Recall}{Precision+Recall}=\frac{2*TP}{2*TP+FP+FN} \tag{1}\]  
As shown in Table 2, the hybrid CNN+LSTM model was the best performer, and performed slightly better than the more computationally demanding and memory-greedy Transformer architecture [48]. Notably, in our experiments, the F1 score always produced the same ranking as accuracy. This was expected because the datasets are balanced. Figure 5(a),b show the confusion matrix for the non-fine-tuned and the fine-tuned hybrid CNN+LSTM models, respectively, evidencing that their performance was nearly uniform for all classes. A test was also conducted on the Fine-Tuning Test dataset without fine-tuning, which produced accuracy values below 20%, not shown in the table. The reason for the poor performance of the classifiers when applied, without fine-tuning, to the Fine-Tuning Test dataset was explained in Section 3.2, and it also applies to the regression models. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{5}{c}{**Classification Models with and Without Fine-Tuning**} \\ \hline \hline
**Model** & **Accuracy, no FT** & **F1, no FT** & **Accuracy, with FT** & **F1, with FT** \\ \hline DenseNN & 77.36\% & 77\% & 79.41\% & 79\% \\ CNN & 84.79\% & 85\% & 93.77\% & 94\% \\ LSTM & 83.37\% & 83\% & 89.27\% & 89\% \\ CNN+LSTM & **85.53\%** & **86\%** & **96.83\%** & **97\%** \\ Transformer & 78.93\% & 79\% & 96.36\% & 96\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Classification accuracy results of the classification models, applied to the Pre-Training Test dataset (without fine-tuning) and to the Fine-Tuning Test dataset (with fine-tuning). Bold text indicates the best accuracy and F1 values. Figure 6: (**a**) Confusion matrix for the classification model CNN+LSTM before fine-tuning. (**b**) Confusion matrix for the classification model CNN+LSTM, fine-tuned with 350 samples per class. We then tested the three models in the regression setup on the Pre-Training Test dataset without fine-tuning. Table 3 shows that the CNN+LSTM hybrid model is again the best performer, both in terms of ACPR and mean absolute error (MAE). As stated in Section 3.2, the Almost-Correct Prediction Rate (ACPR) is a measure of the percentage of predictions with an Absolute Error smaller than \(Th\). We show the ACPR performance of the three models with the two thresholds \(Th=0.5\) and \(Th=1\) for the sake of comparability with other works in the literature that only measured the ACPR with \(Th=1\)[25; 44]. Note that the results in Table 3 are only shown for documentation purposes, as they do not represent the objective of this work, which is, instead, to illustrate the performance of the proposed approach with fine-tuning.