# Indoor Localization System Using Fingerprinting and Novelty Detection for Evaluation of Confidence
## 4 Performance Evaluation
### Evaluation of Confidence Using Prediction Probability  
As mentioned in Section 3.1, the evaluation of confidence using prediction probability uses the threshold \(\alpha\) to decide whether to accept or discard an estimated class/position based on the probability of the sample belonging to that class. Only samples with prediction probability higher than \(\alpha\) are accepted.  
Figure 8a shows the relationship among the value of the threshold \(\alpha\), and the final accuracy of the accepted classifications (in red) as well as the percentage of accepted estimations (in blue). Similarly, Figure 8b shows the behavior of the average localization error (in red) and, again, the percentage of accepted estimations (in blue) as we increase the threshold \(\alpha\). Finally, Figure 8c shows the behavior of the room estimation accuracy, i.e., we are using the room name as classes. These results show several characteristics of our proposed LocFiND solution that we will discuss in the next paragraphs.  
First, as expected, as we increase the threshold \(\alpha\), the number of accepted estimations decreases (blue curve in Figure 8a-c). However, since we are accepting higher quality predictions, the accuracy increases (Figure 8a), and thus the average localization error decreases (Figure 8b) and the room accuracy also increases (Figure 8c).  
For instance, compared to a traditional kNN, in the case of Figure 8a,b, a threshold \(\alpha=40\%\) accept 85% of the predictions while increasing the accuracy from 36% to 41% and decreasing the average localization  
increasing only slightly the accuracy to 96.4%. This is mostly because it is easier to estimate rooms and the classifier is able to do it with higher probabilities.  
On the other hand, when we use a threshold \(\alpha=80\%\), only 20% of the position estimations were forwarded to the application, as depicted in Figure 8a,b. However, these estimations were very reliable, having almost 70% accuracy and a localization error lower than 0.9 m. A better result came from the room accuracy, in which 92% of the predictions were accepted with an accuracy of almost 99%.  
To have a better understanding of how the localization errors are distributed, Figure 9 shows the cumulative distribution function for the thresholds \(\alpha=25\%\), 45%, and 75%. This graph shows the percentage of estimations (\(y\)-axis) with an error lower than the \(x\)-axis. In this curve, the smaller the \(x\)-axis value and the larger the \(y\)-axis value, the better since this means that the majority of the estimations have lower errors. As we can see, for instance, when using a threshold \(\alpha=25\%\), only 68% of the estimations resulted in an error lower than or equal to 2 m. On the other hand, for a threshold \(\alpha=75\%\), almost 85% of the estimations resulted in errors lower than or equal to 2 m.  
Thus, as we can see, the choice of which threshold to use really depends on the scenario and type of application. The main challenge is how to decide what is the best case for our current problem: a higher number of position estimations with a lower accuracy or fewer estimations with higher accuracy. However, we can make this decision easier if we can improve this solution by using better mechanisms for the evaluation of confidence. Thus, in our LocFiND solution, we used Novelty Detection to improve this evaluation of confidence.