# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 5 Results and Discussion  
### Synthetic Dataset Results  
A controlled experiment is performed to estimate the performance of inconsistent labelers as a function of the input space while highlighting their dependencies. For this first experiment, the 2D PCA Iris dataset is employed (see Section 4.1). Besides, the data are divided into five clusters using the K-means technique to emulate five annotators using the approach \"Biased coin (Non-homogeneous)\". A matrix \(\mathbf{A}\in[0,1]^{R\times R}\) is used to set a different score (annotator reliability) for each pair annotator-cluster, as follows:  
\[\mathbf{A}=\left(\begin{array}{cccccc}0&0.9&0.5&0.15&0.6\\ 0.9&0&0.3&0.4&0.75\\ 0.5&0.3&0&0.6&0.3\\ 0.15&0.4&0.6&0&0.8\\ 0.6&0.75&0.3&0.8&0\\ \end{array}\right). \tag{10}\]  
Note that the value \(a_{c,r}\) refers to the probability that the annotator \(r\) fails labeling a sample that belongs to the cluster \(c\); thus, a zero-value means a perfect annotator for the correspondent cluster. The \(r\)-th annotator is an expert (its labels correspond to the ground truth) in the region \(c=r\).  
Figure 3 shows the decision boundaries generated by our approach for the first experiment. As shown, RCDNN offers a suitable representation for the multi-class classification problem; an AUC score of 0.9837 is achieved, which demonstrates its generalization capability, even in cases where the ground truth is unknown. Indeed, RCDNN codes both the relationship between the input space and the annotator's behavior and the dependencies among their labels, which improves the quality of the expert codification [1; 16; 29]. To empirically support the above statement, Figure 4 shows each annotator's simulated accuracy and the reliability estimated by our RCDNN. The latter elucidates how our method performs a satisfactory identification of the zones where the labelers have the best accuracy. The above is not unexpected because the annotators' accuracy (simulated) is compared with their reliability (estimated); hence, the regions where a specific labeler obtains the higher accuracy should match the regions where the estimated reliability is closer to 1.  
In addition, Figure 5 shows a comparison between the Pearson correlation coefficients (absolute value) from the labelers' performance in Equation (10), configuring the simulated dependencies among the annotators, and the Pearson correlation coefficients (absolute value) from the weight matrix \(\mathbf{\Phi}_{\mathbf{\epsilon}_{4}}\in\mathbb{R}^{(K+R)\times R}\) of the layer \(e_{4}(\cdot)\) (RCDNN annotators'  
Figure 2: RCDNN architecture details. \(\epsilon_{s}\) stands for dense layer. \(\epsilon_{1}\) holds a linear activation, \(e_{2}\) includes a tanh-based activation, and \(\epsilon_{3}\) and \(\epsilon_{4}\) output the hidden ground truth label and the annotator’s reliability fixing a softmax and a sigmoid activation, respectively.  
dependencies estimation). Comparing the real and the estimated dependencies, it is noticeable that, even though the exact matrix is not recovered, our approach efficiently finds tendencies between annotators' performances. Thereby, the learned representation from hidden layers (see Figure 2) allows coding both linear and non-linear patterns that recover the expert dependencies from data. Then, our deep model estimates the unknown ground truth and the relationships between annotators.  
Figure 4: RCDNN-based annotators performance (reliability) estimation for the synthetic experiments (2D PCA Iris data). In the first column (from **top** to **bottom**), the simulated accuracy for each annotator is presented based on Equation (10). The second column shows (from **top** to **bottom**) the estimated annotators’ reliability (\(\lambda_{\tau}\)).  
Figure 3: RCDNN’s decision boundaries for the 2D-PCA Iris dataset (synthetic scenario). AUC \(=~{}0.9837\). The point’s color stands for the Iris dataset classes. \(PCA_{1}\) and \(PCA_{2}\) stand for the first and second PCA-based projections.