# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 5 Results and Discussion
### Introducing Spammers and Malicious Annotators  
As a final experiment, we wish to analyze the impact of spammers and malicious annotators on the performance of our multi-labeler classifier. For concrete testing, we use the pima dataset, which holds 768 instances; from this dataset, we use 538 samples for training and the remaining 230 for testing. We create synthetic labels from 5 annotators generated from the biased coin (Non-homogeneous) procedure (see Section 4.2 and Equation (10)). According to Figure 6 (blue dots), we notice that from the 5 labelers, two are categorized as suitable labelers, one as Spammers and the remaining as Malicious. Then, we add \(R_{e}\) additional annotators aiming to test our approach in extreme scenarios, where the number of malicious or spammers annotators increases. The labels are simulated as follows: a random number \(\alpha_{n}^{r}\) is sampled from a Bernoulli distribution with parameter \(p_{r}\); then if \(\alpha_{n}^{r}=0\), \(y_{n}^{r}=y_{n}\), and \(y_{n}^{r}=\tilde{y}_{n}\) otherwise. For Spammers, we use \(R_{e}=65\), and \(p_{r}=0.5\) (see red dots in Figure 6); alike, for malicious labelers, we fix \(R_{e}=20\), and \(p_{r}=0.6\) (see green dots in Figure 6). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**AUC(\%)**} \\ \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Voice Dataset**} & \multirow{2}{*}{**Polarity Dataset**} & \multirow{2}{*}{**Music**} & \multirow{2}{*}{**Average AUC**} & \multirow{2}{*}{**Average Ranking**} \\  & **G** & **R** & **B** & & & & \\ \hline GPC-GOLD & 93.66 & 93.66 & 93.66 & 80.26 & 92.84 & 90.81 & - \\ GPC-MV & 90.17 & 84.73 & 84.04 & 71.14 & 88.79 & 83.77 & 6.8 \\ MA-LFC & 89.99 & 90.59 & 87.27 & 72.06 & 85.99 & 85.18 & 6.4 \\ MA-DGRL & 85.45 & 90.14 & 79.33 & 56.13 & 88.32 & 79.86 & 8.4 \\ MA-MAE & 91.08 & 89.12 & 80.74 & 48.73 & 81.92 & 78.31 & 8.4 \\ MA-GPC & 91.50 & 91.16 & 80.81 & 61.18 & 82.53 & 81.43 & 6.8 \\ KAAR & 89.85 & 93.50 & 89.20 & **77.46** & 88.96 & 87.79 & 3.8 \\ LKAAR-LR & 90.39 & 92.92 & 88.94 & 68.28 & 84.43 & 84.99 & 6.0 \\ LKAAR-SVM & 92.06 & 93.02 & 86,98 & 72.70 & 89.98 & 87.70 & 3.6 \\ LKAAR-GPC & 90.78 & 93.60 & 89.79 & 76.50 & 86.44 & 87.42 & 3.4 \\ RCDNN (ours) & **92.24** & **94.19** & **92.57** & 76.04 & **93.29** & **89.66** & **1.4** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Fully real-world datasets results. Bold: the method with the highest performance excluding the upper bound (target) classifier GPC-GOLD. The last column presents the average ranking for AUC score, in bold the best average ranking. The Friedman test returns a Chi-square value of 26.71 (\(p\)-value = 0.0015). Figure 7 presents the classifiers' performance as a function of the number of spammers (left in Figure 7) and malicious annotators (in right Figure 7). First, we analyze the effect of Spammers annotators on the RCDNN's performance. From the results in Figure 7 (left), we remark that when the number of Spammers is less than 40, the performance of our approach is not affected. However, when the number of Spammers exceeds 40, the RCDNN' AUC becomes unstable, oscillating between 0.6 and 0.8. Accordingly, we highlight that the critical point is presented when the percentages of good, spammers and malicious labelers are, respectively, 4.65%, 90.70%, and 4.65%; which shows that our RCDNN is robust in the presence of a high number of Spammers. Now, we compare our RCDNN with two state-of-the-art models, MA-LFC (linear model with the more competitive performance according to Table 6) and LKAAR-GPC (Non-linear model with the more competitive AUC in Table 6). We notice that the LKAAR-GPC behavior is similar to our approach when the number of spammers is greater than 35, the AUC starts to descend gradually. Conversely, we note that the MA-LFC's performance is drastically affected by the spammers; in fact, for more than 8 spammers, the AUC is close to 0.2. Second, we inspect the consequences when malicious labelers are added.