## 5 ML Setup and Results Analysis
### Tests and Results Analysis
This is because the classifier can detect more numbers of true positives and true negatives than false negatives and false positives. Therefore, a higher AUC value for a classifier has the better ability to distinguish between positive and negative classes. When AUC = 0.5, the classifier is not able to distinguish between positive and negative class points, meaning either the classifier is predicting random class or constant class for all the data points. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline
**Geofence** & **TSMF Data** & **ML Model** & **Accuracy** & **Precision** & **Recall** & **Kappa** & **F1** & **AUC** \\ \hline \multirow{6}{*}{
\begin{tabular}{c} Web LH \\ Building \\ \end{tabular} } & \multirow{3}{*}{WF-ML} & kNN & 1.0 & 1.0 & 1.0 & 1.0 & 1.0 & 1.0 \\  & & RF & 0.99 & 1.0 & 0.98 & 0.98 & 0.99 & 1.0 \\  & & SVM & 0.99 & 1.0 & 0.98 & 0.98 & 0.99 & 0.99 \\  & & ANN & 1.0 & 1.0 & 1.0 & 1.0 & 1.0 & 1.0 \\  & & CNN & 0.97 & 0.94 & 1.00 & 0.93 & 0.97 & 0.96 \\ \hline \end{tabular}
\end{table} 
Table 1: WRL Lab Indoor WF-ML results. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline
**Geofence** & **TSMF Data** & **ML Model** & **Accuracy** & **Precision** & **Recall** & **Kappa** & **F1** & **AUC** \\ \hline \multirow{6}{*}{ \begin{tabular}{c} Web LH \\ Building \\ \end{tabular}  } & \multirow{3}{*}{MF-ML} & kNN & 0.90 & 0.89 & 0.91 & 0.80 & 0.90 & 0.95 \\  & & RF & 0.93 & 0.94 & 0.92 & 0.86 & 0.93 & 0.97 \\  & & SVM & 0.92 & 0.89 & 0.96 & 0.85 & 0.93 & 0.96 \\  & & ANN & 0.91 & 0.91 & 0.90 & 0.82 & 0.91 & 0.96 \\  & & CNN & 0.92 & 0.90 & 0.96 & 0.85 & 0.93 &Figures 24 and 25 show a confusion matrix that summarizes the prediction results of the whole system when it classifies the user's predicted geofence across all their associated geofences in the building layer and the indoor layer. The chosen ML deployed model was the ANN one due to the TensorFlow lite [38] support on the android platform in general and the lower complexity and size comparing to the CNN life mode. Furthermore, the classical models are not supported yet by TensorFlow lite. Most of the selected algorithms' accuracies fluctuate within a range of 2% to 4%, which is not that large in the overall prediction performance of the deployed model on the mobile application. The problems were with the inference time and the optimization of the mobile application. During testing, the ANN and CNN models were easily tested with the TensorFlow lite library,  
Figure 21: ROC curves for WF-ML models in Maraa’shly LH. Figure 20: ROC curves for WF-ML models in WRL Lab. which is supported by the Flutter framework [39] that was used in the mobile application development. Flutter is an open-source UI software development kit created by Google. It is used to develop cross-platform applications for Android, iOS, Linux, macOS, and Windows. Therefore, the developed application can be deployed to work on an iOS device with no problems with the current implementation. Figure 23: ROC curves for MF-ML models in Neo LH building. Figure 22: ROC curves for MF-ML models in Communication building. The inference time was nearly 0.0016 s, with all of the ANN models and CNN models achieving 0.12 s. However, the classical models were not supported by the TensorFlow like library. The solution was to use another library called ml_algo [40] to test the models. The inference time of kNN was 10.18 s as an example of the classical ML models. This number was expected as the TensorFlow like library leverages specialist mobile accelerators and it provides on-device machine learning inference with low latency and a small binary size by decreasing the precision of the ML models. Therefore, the choice was to use the ANN model as the difference in accuracy is pretty low and it has the best inference time so far among the other algorithms. Each geofence has taken as input 50 scans of the Wi-Fi and MF metrics to indicate the performance of the geofences classification. The buildings layer accuracy of the system is 76%.