# High-Quality Data from Crowdsourcing towards the Creation of a Mexican Anti-Immigrant Speech Corpus
## 4 Results and Discussion
### Result 1: Assessing the Performance of the Crowdsourced Annotation Protocol  
To address RQ1, we conducted two experiments to evaluate the performance of the crowdsourced annotation protocol. The first experiment focused on obtaining high-quality data annotations by following the annotation protocol outlined by experts in the field, mentioned in Section 3.2, and involving as many non-expert annotators as possible. The second experiment aimed to analyze the time that non-expert annotators took compared to expert annotators, and the role of the crowdsourced annotation platform in facilitating the process.  
For the first experiment, 47 contributors participated in the crowdsourced annotation process, as described in Section 3.4. Not all contributors were able to pass the control mechanism described in 3.3.2. Upon presenting the contributors with control data (from Pool C), it was found that 15 of them did not meet the established threshold of the coefficient of agreement (0.627), as shown in Figure 7, which illustrates the distribution of kappa coefficients obtained from the non-expert users. The average kappa coefficient for these users was 0.444.  
Figure 8 displays the kappa coefficient distribution for non-expert users who did pass the threshold. These users achieved an average kappa value of 0.796 with a standard deviation of 0.105, indicating good-quality annotations for the \"real\" data.  
The contrast in the kappa coefficients distribution between the non-expert users who passed the threshold and those who did not pass highlights an aspect of crowdsourced annotation: not all individuals possess the qualifications to adhere to established annotation protocols guided by expert knowledge. Furthermore, upon exami  
Figure 7: Kappa coefficient distribution for users that did not pass the set threshold, obtained from \(m\) non-experts (where \(m=15\)) through the platform with data quality control protocols.  
the kappa coefficients between experts and non-experts, a noticeable disparity emerged, highlighting the significant difference in data quality that would have resulted if we had allowed all users to annotate \"real\" data without quality control. Therefore, we strongly recommend that other research teams implement annotation protocols, including a control mechanism, such as the minimum acceptable agreement coefficient discussed in this paper.  
An interesting pattern emerged during the analysis of the second experiment, which involved comparing the time that the non-expert annotators spent annotating the data to the time taken by expert annotators. We observed that the expert annotators completed the annotations in less time than the non-experts. Ideally, all unlabeled data should be annotated by experts, as this would likely result in faster annotation overall.  
However, it became clear that annotating a substantial amount of documents solely with the help of experts would require a significant amount of time. In light of this, we decided to leverage the annotations provided by non-expert contributors that adhered to the expert criteria that we were looking for. Although it was evident that these non-expert annotators took nearly three times longer than the experts, shown in Figure 9, their contributions were valuable.  
Here, the crowdsourced annotation platform proved to be advantageous. By allowing multiple non-experts, who had passed the threshold, to annotate different parts of the dataset simultaneously or independently, we optimized the annotation process and significantly reduced the overall time that it would have taken otherwise. This approach proved to be a convenient and efficient way to handle the annotation of a large volume of data, responding to RQ1.  
Figure 8: Kappa coefficient distribution for users that passed the set threshold, obtained from \(n\) non-experts (where \(n=32\)) through the platform with data quality control protocols.