## 5 Evaluation
### The Algorithm  
The next component to be evaluated is the SkeL algorithm. As discussed in Section 2, the Pred procedure of SkeL consists of a hierarchy of multiclass classifiers, one for each internal node in the hierarchy. Each classifier implemented a random forest [20], which is known to be robust to labeling noise [21], to evaluate the ability of SkeL to improve over an already noise-robust baseline. The confidence parameter \(\theta\) of SkeL has been set to 0.2, which has resulted in a reasonable trade-off between accurate training and cognitive effort for the user. For simplicity, we have used an infinite queue (\(d=\infty\)) for the confidence update due to the relatively short duration of the experiment. This being an experiment built on previously collected data, we could not query users in real-time in case of conflicts. To simulate a collaborative, non-adversarial user, we assumed that the user returns the ground truth label when his/her initial label is contested. The evaluation of SkeL is performed by comparing it with three alternative algorithms:  
* NonSkeL, which never contradicts the user (obtained by replacing solveConflict with a train and update step, as occurs in the training phase);
* Ignore, which simply ignores any example for which a conflict arises (obtained by removing everything from the else onwards in Algorithm 3);
* Bother, which always contradicts the user (obtained by calling challengeUser after all askUser calls, and removing solveConflict). As presented in the previous section, a surprisingly high proportion of labels present inconsistencies. To estimate the effect of this large and very diverse proportion of labeling errors on the performance of the system, we divided the set of users in the three groups reported in the table. Figure 4 reports the results of SkeL and of the three alternatives for an increasing number of iterations. Each row represents the results for a different group of users: at most 10% labeling errors (top), 10% to 25% (middle), more than 25% (bottom). The left column reports \(f_{1}\) scores averaged over all users in the corresponding group with a \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Label Noise Level** & \(\leq\)10\% & 10–25\% & \(\geq\)25\% \\ \hline
**Users** & 21.6\% & 51.4\% & 27.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Percentage of users with each label noise level. number of training samples greater than 200. The score for each user is computed on a fixed test set, namely the latest 15% of all of the data available for that user that were not used for training. This score provides an estimate of the performance of the algorithms when making predictions on future data. Note that we consider a label as correctly predicted if it is compatible with the ground-truth label, because this is the only type of reliable supervision we have access to. Results clearly indicate that our skeptical algorithm (red curve) consistently outperforms a non-skeptical alternative (blue curve). As expected, the advantage is moderate for users with a relatively small fraction of labeling errors (top figure), and grows with the unreliability of the users, reaching a gap of 0.20 for users with more than 25% labeling errors (bottom figure). Ignoring conflicting cases (brown curve) is clearly not an option, as it achieves the worst performance in all cases. On the other hand, when always having access to correct supervision, Bother (green curve) clearly achieves the highest performance. However, SkeL is capable of getting reasonably close to this upper bound when enough iterations are provided, at a fraction of the cost in terms of user effort. The right column reports the number of times the user is contradicted for SkeL (red curve) and for Bother (green curve), for which, they are simply the number of iterations. SkeL clearly contradicts more when facing increasingly unreliable users. However, the cost remains substantially lower than that of Bother, going from 13% (top figure) to 23% (bottom figure).