## 5 Evaluation
### The Algorithm
Figure 5 reports the confusion matrices of the _location_ classifier (the root of the hierarchy in Figure 2) for the last time instant of the different algorithms shown in Figure 4. Results are normalized over the sum of all of the elements in the matrix. In each matrix, rows are ground truth labels and columns are predicted labels. Matrix entries are colored, with darker shades corresponding to larger entries. Each row represents the results for a different group of users who have, at most 10%, from 10% to 25%, and more than 25% labeling errors, respectively. The matrices in the first column report results of the SkeL algorithm, the second column report results of the NonSkeL one, the third column report results of the Bother one, and the last column report results of the Ignore one. Clearly, Bother has the best performance, as can be seen by the dark coloring of the main diagonal (corresponding to correct predictions), and shows the upper limit one can reach by always having access to the ground truth during training. While not as accurate, SkeL has a similar qualitative behavior, with on-diagonal entries typically larger than off-diagonal ones. The bad performance of Ignore and NonSkeL are due to the fact that they tend to overpredict certain classes (as shown by the dark columns): _University_ and _On the Move_ for Ignore, _University_ and _Home_ for NonSkeL. The first row represents the results of the users with fewer labeling errors. Given that these users tend to provide high-quality labels, SkeL is only slightly better than NonSkeL. This is consistent with the \(F_{1}\) score in the first row of Figure 4, where the red curve and blue curve are close to each other. However, SkeL manages to substantially improve the recognition ability for _On the move_ and _Other location_, the classes for which the user feedback is least frequent, at the cost of a (smaller) decrease in accuracy for the most popular ones, _University_ and _Home_. This behavior is even more evident when considering users with a higher fraction of errors (middle row, from 10% to 25% labeling errors), where the fraction of times _On the move_ and _Other location_ get predicted and the prediction is correct increases from 2.64% to 10.96% and from 1.52% to 5.69%, respectively. The third row reports results for the least reliable users, with more than 25% labeling errors. While the distance to the \"ideal\" (for the machine) setting represented by Bother increases, the advantage of SkeL over NonSkeL (and Ignore) also widens. The tendency of the user to always reply with _Home_ and _University_ does affect SkeL, which starts to show a similar behavior (dark columns in the matrix). On the other hand, despite the substantial unreliability of the user feedback, it still manages to recover a large fraction of the _On the Move_ (from 6.63% to 14.70%) and the _Other location_ (from 0.62% to 8.28%) classes, both virtually lost by NonSkeL. Figure 4: Results averaged over users with at most 10% (**top row**), from 10% to 25% (**middle row**), and more than 25% (**bottom row**) labeling errors. **Left column**: \(F_{1}\) scores on left-out data. **Right column**: number of times user is contradicted.