# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 4 Experimental Set-Up
### RCDNN Detailed Architecture and Training  
The proposed RCDNN architecture for multiple annotators comprises:  
* IN: An input layer fed by the input samples \(\mathbf{X}\in\mathbb{R}^{N\times P}\);
* \(e_{1}(\cdot)\): A dense layer coding relevant patterns from input features to perform. The number of neurons is set as \(h=\lfloor\rho P\rceil\), where \(\rho\in\{0.5,1,1.5\}\) is chosen empirically; a linear-based activation function is used to code input data linear dependencies;
* \(e_{2}(\cdot)\): A dense layer fixing a _tanh_-based activation function with \(J=K+R\) neurons to reveal non-linear relationships;
* \(e_{3}(\cdot)\): A fully-connected layer with \(K\) neurons and a _softmax_-based activation function, which is employed to estimate the hidden ground truth \(\zeta_{k,n}\);
* \(e_{4}(\cdot)\): A dense layer with \(R\) neurons and a _sigmoid_-based activation function, which is used to compute the annotators' reliability in \(\lambda^{r}_{n}\);
* For all provided \(e_{s}\) layers l1 plus l2-based regularization strategy is used, searching the regularization weights within the range {1e-3,1e-2,1e-1};
* Batch Normalization and Dropout layers are included between layers to avoid vanishing and exploding gradient issues. Additionally, it favors the RCDNN's generalization capability as exposed in Section 3.2. See Figure 2 for details;
* The optimization problem in Equation (7) is solved by using a Back-propagation algorithm as usual. Moreover, to favor scalability, we utilize a mini-batch-based gradient descent approach with automatic differentiation (RMSprop-based optimizer is fixed).  
We clarify that our RCDNN is flexible, and it admits different deep structures such as recurrent or convolutional layers aiming to deal with complex tasks (e.g., computer vision or natural language processing). Moreover, our approach can build from different activation functions (RELU, ELU, sigmoid, softmax). However, the last layers (in this particular case \(e_{3}\) and \(e_{4}\)) need to be designed to code each annotator's behavior and the hidden ground truth. For example, the parameter \(\lambda^{r}_{n}\) represents an estimation for the annotators' reliability; accordingly, we need to use an activation function whose output belongs to the range \([0,1]\).  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Approach** & **Brief Description** \\ \hline GPC-GOLD & A GPC using the real labels (upper bound). \\ GPC-MV & A GPC using the majority voting of the labels as the ground truth. \\ MA-LFC [2] & A LRC with constant parameters across the input space. \\ MA-DGRL [32] & A multi-labeler approach that considers as latent variables the annotator performance. \\ MA-MAE [37] & A LRC where the sources parameters depend on the input space. \\ MA-GPC [14] & A multi-labeler GPC, which is an extension of MA-LFC by using a non-linear approach. \\ KAAR [16] & A kernel-based approach that employs a convex combination of GPC, it codes the labelers dependencies. \\ LKAAR-(LR,SVM,GPC) [16] & A localized kernel alignment-based annotator relevance analysis using a combination of LRC, SVM, GPC \\  & respectively. It models both the annotators dependencies and the relationship between the labelers’ behavior and the input features. \\ \hline \hline \end{tabular}
\end{table}
Table 3: A short overview of the tested state-of-the-art approaches. GPC: Gaussian processes classifier, LRC: logistic regression classifier, MV: majority voting, MA: multiple annotators, MAE: Modelling annotators expertise, LFC: Learning from crowds, DGRL: distinguishing good from random labelers, KAAR: kernel alignment-based annotator relevance analysis, LKAAR: localized kernel alignment-based annotator relevance analysis.