# High-Quality Data from Crowdsourcing towards the Creation of a Mexican Anti-Immigrant Speech Corpus
## 3 Methodology
### Stage III: Data Pools
#### 3.3.2 Annotation Batches with Trade-Off in Time and Quality  
To consider the time and quality trade-off, we asked 32 users to annotate the control data. Our objective was to observe the time required by the users to complete the task. They had to determine whether a tweet contained anti-immigrant speech or not. The average time to classify a tweet was 12.583 s, as shown in Figure 4.  
Based on the average time obtained, we conducted a simulation experiment to determine the appropriate batch size for annotation. We wanted to ensure that the non-expert  
Figure 4: Average time that the 32 users took to categorize tweets as anti-immigrant or not, where the x-axis represents the users and the y-axis represents the average time that each user took to choose the category.  
users did not spend more than 20 min annotating. Considering that the average time spent annotating the control data was 6.29 min, we selected a real annotation batch of 50 tweets, since this would require approximately 10.49 min for the user to annotate, resulting in a total of 17.18 min. This fell within the desired maximum timeframe of 20 min per annotation batch.  
For users to annotate the 50 \"real\" unlabeled tweets, we first needed to evaluate the user agreement coefficient. However, before proceeding, we conducted an experiment to establish the kappa coefficient for comparison with the non-expert annotators. Our experiment aimed to assess the impact of incorrect categorizations on control data. We allowed the annotators to make up to 9 errors in categorizing the tweets. However, the resulting kappa coefficient was 0.448, indicating only moderate agreement according to the interpretation of kappa. This agreement level is not highly recommended.  
Considering this, we turned to the kappa coefficient among the experts in the field. Even the experts did not achieve a perfect kappa coefficient, with the highest agreement value being 0.821. Therefore, we decided to adjust our kappa coefficient base. Instead of using 1 as the perfect score, we used 0.821. Mathematically, we subtracted 0.821 from 1, resulting in a difference of 0.179. We added this difference to the kappa coefficient obtained by the users, creating what we now refer to as the adjusted kappa coefficient.  
Allowing users to make 9 errors produced an adjusted kappa coefficient of 0.627. In terms of kappa interpretation, this indicated substantial agreement.  
By adjusting the kappa coefficient threshold, we acknowledged the unrealistic expectation of achieving perfect agreement, even among experts. This adjustment allowed for a more practical and feasible assessment of agreement while still maintaining a reasonable level of quality in the annotations. It recognized that a high level of agreement, although not perfect, could still provide valuable and meaningful insights into the task at hand.  
Lowering the kappa coefficient threshold in this context ensured that we did not exclude annotations from competent and knowledgeable non-experts who could provide valuable perspectives and insights, even if their judgments may differ slightly from the experts. It allowed us to leverage the collective wisdom of a diverse group of annotators while maintaining a level of agreement that was substantial and reliable for the specific task or domain.  
In conclusion, our annotation process involved trade-offs regarding time efficiency and agreement among annotators. We determined an appropriate batch size to ensure that users spent no more than 20 min annotating based on observations and simulation experiments. While initial agreement levels were moderate, we adjusted the kappa coefficient to account for realistic limitations and achieve substantial agreement. This approach allowed us to incorporate valuable perspectives from competent non-expert users while maintaining a practical assessment of annotation quality.