# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 3 Methods
### Regularized Chained Deep Neural Network for Multiple Annotators  
Let \(g:\mathcal{X}\rightarrow\mathcal{Y}\) be a classification function trained on the input-output set \(\{\mathbf{x}_{n},y_{n}\}_{n=1}^{N}\), where \(\mathbf{x}_{n}\in\mathcal{X}\subseteq\mathbb{R}^{P}\) is a \(P-\)dimensional input feature vector corresponding to the \(n\)-th instance with label \(y_{n}\in\mathcal{Y}\subseteq\{1,2,\ldots,K\}\), being \(K\) the number of classes. \(y_{n}\) is assumed to be the absolute ground truth. However, in many real-world classification problems, instead of the ground truth, multiple labels are provided by \(R\in\mathbb{Z}^{+}\) experts with different levels of ability [12], where the \(r\)-th expert annotates \(|\Omega_{r}|\leq N\) instances, being \(|\Omega_{r}|\) the cardinality of the set \(\Omega_{r}\) containing the indexes of samples labeled by expert \(r\). Further, let \(\Psi_{n}\) be the index set of annotators who labeled the \(n\)-th instance. Next, it is possible to build a dataset from multiple annotators \(\mathcal{D}=\left\{\mathbf{X},\mathbf{Y}=\{y_{n}^{r}\}_{n=1}^{N};r\in\Psi_{n}\right\}\), where \(y_{n}^{r}\) is the decision of annotator \(r\) for instance \(n\). Following the work proposed by authors in [32]; here, a regularized chained deep neural network (RCDNN) is introduced for classification tasks from multiple annotations. Concerning this, let \(\lambda_{n}^{r}\in\{0,1\}\) be a binary variable representing the \(r\)-th annotator reliability: \(\lambda_{n}^{r}=1\) if \(y_{n}^{r}=y_{n}\), and \(\lambda_{n}^{r}=0\) in other case. If \(\lambda_{n}^{r}=1\), the label \(y_{n}^{r}\) is modeled by means of a categorical distribution; otherwise, if \(\lambda_{n}^{r}=0\), \(y_{n}^{r}\) is supposed to follow an uniform distribution. In consequence, the likelihood function in Equation (3) is rewritten as:  
\[p(\mathbf{Y}|\mathbf{\theta})=\prod_{n=1}^{N}\prod_{r\in\Psi_{n}}\left(\prod_{k=1}^{K} \zeta_{n,k}^{g(y_{n}^{r}-k)}\right)^{\lambda_{n}^{r}}\left(\frac{1}{K}\right)^{ 1-\lambda_{n}^{r}}, \tag{4}\]where \(\delta(y_{n}^{\prime}-k)=1\), if \(y_{n}^{\prime}=k\), and \(\delta(y_{n}^{\prime}-k)=0\), otherwise. Moreover, \(\zeta_{n,k}=p(y_{n}^{\prime}=k|\lambda_{n}^{r}=1)\) is the estimation of the hidden ground truth for the \(n\)-th instance in class \(k\). Accordingly, an architecture holding \(J=K+R\) outputs is required within our RCDNN for modeling the likelihood parameters \(\theta\) in Equation (4). In particular, \(K\) output layers \(\{\theta_{k}(\cdot)\}_{k=1}^{K}\) are fixed to estimate the hidden ground truth \(\zeta_{n,k}\) based on a softmax function as follows:  
\[\zeta_{n,k}=\theta_{k}(f_{1}(\mathbf{x}_{n}),\dots,f_{K}(\mathbf{x}_{n}))=\frac{\exp( f_{k}(\mathbf{x}_{n}))}{\sum_{i=1}^{K}\exp(f_{i}(\mathbf{x}_{n}))}. \tag{5}\]  
Furthermore, a step function can be used to compute the annotator reliability. Yet, the step function is approximated through \(R\) output layers \(\{t_{r}(\cdot)\}_{r=1}^{R}\), fixing the well-known sigmoid activation to avoid discontinuities and favor the RCDNN implementation:  
\[\lambda_{n}^{r}=\iota(f_{m}(\mathbf{x}_{n}))=\frac{1}{1+\exp(f_{m}(\mathbf{x}_{n}))}, \tag{6}\]  
where \(m\in\{K+1,\dots J\}\) is the index of the output linked to the estimation of the reliability of \(r\)-th expert. Afterward, the log likelihood of Equation (4) is used to compute the RCDNN weights and bias in \(\mathbf{\phi}\), as follows:  
\[\mathbf{\phi}^{*}=\arg\min_{\mathbf{\phi}}-\sum_{n=1}^{N}\sum_{r\in R_{n}}\Bigg{[} \lambda_{n}^{r}(\mathbf{\phi})\Bigg{(}\sum_{k=1}^{K}\delta(y_{n}^{\prime}-k)\log( \zeta_{n,k}(\mathbf{\phi}))\Bigg{)}-(1-\lambda_{n}^{r}(\mathbf{\phi}))\log(K)\Bigg{]}, \tag{7}\]  
where \(\lambda_{n}^{r}(\mathbf{\phi})\) and \(\zeta_{n,k}(\mathbf{\phi})\) highlight the dependency between the annotator reliability/ground truth estimation and the RCDNN weights and bias. Figure 1 summarizes the RCDNN pipeline as a classifier for a dataset holding multiple annotators.