# Regularized Chained Deep Neural Network Classifier for Multiple Annotators
## 4 Experimental Set-Up  
### Tested Datasets  
The introduced RCDNN classifier for multiple annotators scenarios is tested in three different kind of datasets. The first category, termed _2D-PCA iris dataset_, is intended to show graphically how our method works. The principal component analysis (PCA) algorithm is applied to reduce the well-known Iris dataset dimension from four to two [33], aiming to easily observe some preliminary results in a cartesian plane and illustrate how multiple annotations can be simulated.  
The second category comprises datasets where the input data come from real-world applications. Still, the labels from multiple annotators are obtained synthetically (_Semi-synthetic datasets_). The latter is carried out to control the labeling process. In particular, nine datasets of binary and multi-class-classification tasks are studied from the well-known UCI repository. ([[http://archive.ics.uci.edu/ml](http://archive.ics.uci.edu/ml)]([http://archive.ics.uci.edu/ml](http://archive.ics.uci.edu/ml)), (accessed on 19 April 2021)) The chosen datasets include: Wisconsin Breast Cancer database-(breast); BUPA liver disorders-(bupa); Johns Hopkins University Ionosphere database-(ionosphere); Pima Indians Diabetes database-(pima); Tic-Tac-Toe Endgame database-(tic-tac-toe); Iris Plants database-(iris); Wine data set-(wine); and Image Segmentation dataset--(segmentation). Besides, the publicly available bearing data collected by the Case Western Reserve University-(Western) is tested. The aim is to build a system to diagnose an electric motor's status based on two accelerometers. The feature extraction was performed as in [34].  
The third category includes _Real-world datasets_, where both the input features and the labels come from real-world applications. The Massachusetts Eye and Ear Infirmary Disordered Voice Database from the Kay Elementics company is proved. A subset holding \(N=218\) voice records is considered from both healthy and different voice issues. Each voice record is parametrized using the Mel-frequency cepstral coefficients (MFCC) to obtain an input space with a dimensionality of \(P=13\). A set of physicians label each voice record by assessing its quality through the GRBAS protocol, comprising the evaluation of five qualitative scales: Grade of dysphonia (G); Roughness (R); Breathiness (B); Aesthenia (A); and Strain (S). For each scale, the specialist provides a tag ranging from 0 (healthy voice) to 3 (severe disease) [35]. A five multi-class task is built (one per each qualitative scale). However, five binary classification problems are provided to access the ground truth, which is useful for validation purposes [36]. The second dataset is named music genre, which corresponds to a collection of song records with 30 s of length and labeled from one to ten, depending on their music genre: classical, country, disco, hip-hop, jazz, rock, blues, reggae, pop, and metal. A total of 700 samples are selected randomly and published in the AMT platform to obtain labels from multiples sources. The feature extraction is performed by following the work in [32], to obtain an input space with \(P=124\) features. The third one is a sentiment polarity dataset, which corresponds to a collection of more than 10,000 sentences labeled as positive or negative. The AMT platform published 4999 sentences to gather answers about their polarity. From this process, 27,747 answers were obtained from 203 labelers. The remaining were kept for testing purposes. Each sample is parametrized following the procedure in [32] to get an input space in \(P=1200\) dimensions. Table 2 summarizes the tested semi-synthetic and real-world datasets.