# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 5 Results and Discussion
### Semi-Synthetic Datasets Results  
Table 4 shows the results concerning the \"Non-homogeneous labels\", where it is supposed that the labelers' performance depends on the input space \(\mathcal{X}\). We show the non-parametric Friedman test results to establish their statistical significance. The null hypothesis settles that all algorithms perform equal [39]. Additionally, we fix the significance threshold as \(p<0.05\). The GPC-GOLD standard is not included within the test to compare only multiple annotators approaches. First, we notice that most of the classification schemes present a considerably high performance for both AUC and Acc; in fact, the average AUC and Acc for all methods (except MA-DGRL and MA-MAE) are similar compared to the upper bound GPC-GOLD. The above behavior demonstrates high-quality labels, which is confirmed considering the performance of the most naive approach GPC-MV. Furthermore, we highlight that our RCDNN presents the best average ranking and the second AUC and Acc scores. Then, from non-linear-based approaches, we notice that a naive approach, as GPC-MV, obtains similar performance compared with sophisticated ones, such as KAAR, LKAAR-SVM, and LKAAR-GPC. Nevertheless, as we already comment, such an outcome is a consequence of simulating annotators with suitable quality, which favors the majority voting method. Besides, MA-GPC presents the lowest average ranking compared with its other non-linear methods, resulting from a lack of generalization (over-fitting). Regarding the results for the linear models, they achieve lower performance than non-linear ones. As seen, there is no statistical evidence to establish that our RCDNN outperforms its competitors (\(p\)-value = 0.2). We explain such an outcome because, for this experiment, the quality of the labels is significantly high; thus, similar performances are obtained. On the other hand, Table 5 shows the results concerning the simulation method \"Biased coin (Non-homogeneous)\". At first sight, there exists a generalized lower performance compared with previous results in Table 4. To explain such an outcome, we recall the stimulation parameters \(\mathbf{A}\) in Equation (10), where the element \(1-a_{c,r}\) (column \(r\), row \(c\)) indicates the \(r\)-th annotator's performance in region \(c\). Accordingly, taking the average by column to the matrix \(1-\mathbf{A}\), we obtain the annotators accuracy \([0.57,\ 0.53,\ 0.66,\ 0.61,\ 0.51]\). We remark that the labelers' accuracy is considerably low for this experiment, which impacts the algorithms' performance. RCDNN achieves the best predictive performance in both the overall accuracy and the AUC score; RCDNN also obtains the best average ranking. Moreover, the non-linear competitors KAAR, LKAAR-GPC, and LKAAR-SVM achieve competitive results. However, GPC-MV and MA-GPC offer the lowest classification scores. Regarding GPC-MV, the result is explained because GPC-MV corresponds to the most naive approach. After all, it considers that the whole annotators achieve similar performance. On the other hand, the MA-GPC achieves a similar performance compared with GPC-MV; such a behavior proves that MA-GPC is more prone to over-fitting [9]. Figure 5: Target and estimated annotators dependencies for the synthetic 2D PCA Iris dataset. In the left, the Pearson correlation coefficients (absolute value) from simulated accuracies (experts reliability) in matrix \(\mathbf{A}\) of Equation (10) are shown. In the right, the dependencies among the annotators estimated from the RCDNN \(\epsilon_{4}\) layer’ weights are displayed. Remarkably, simple classifiers as MA-LFC, MA-DGRL, and LKAAR-LR obtain competitive outcomes compared to the non-linear competitors; in fact, all the linear models excepting MA-MAE outperform GPC-MV and MA-GPC. An additional experiment is conducted: an LR-based classifier using the ground truth (following a similar scheme for GPC-GOLD) is trained overall datasets, obtaining an average AUC equal to 87.21 (close enough to the MA-DGRL and LKAAR-LR performances).