# High-Quality Data from Crowdsourcing towards the Creation of a Mexican Anti-Immigrant Speech Corpus
## 4 Results and Discussion
### Result 2: Evaluating Non-Experts' Annotations  
To answer RQ2, we used Pool C annotation statistics, manually annotated by experts and non-experts, as a proxy to estimate the quality of the 1650 new tweets annotated by non-expert contributors. A reduced number of six tweets accounted for 81.63% of a total of 98 erroneous annotations made by non-experts who reached the kappa coefficient threshold. Errors were equally distributed between the positive (52.04%) and the negative (47.96%) classes. Specifically for the anti-immigrant class, three tweets accounted for 82.35% of errors. Responding to RQ2, while non-experts recognized almost all anti-immigrant tweets in Pool C, providing them with a guideline was insufficient to replicate expert annotation. In some instances, the guideline and the overall task framework reached some limits, which will be discussed here. 1. _\"la caravana de migrantes se dettwo aki en Infonavit este es un hondureho\"_. Translation: \"The migrant caravan stopped here at Infonavit, this guy is Honduran\". Fourteen contributors mislabeled tweet number 1 as anti-immigrant, and the average annotation time was high (21 s). Although the tweet did not contain offensive or disrespectful comments about the individual identified as Honduran, the last words of the sentence clearly objectified the person mentioned. Following the criteria established in Section 3.2.1, it was not suffcient to classify it as anti-immigrant, as the tweet fell within a \"border zone\" between the two classes. When applied to ambiguous tweets or low-intensity anti-immigrant speech, binary classification sometimes is not sufficient to meet the required level of complexity of HS. Other works have attempted to overcome this issue by introducing more classes or sub-classes--for example, based on the intensity of HS [20]--although it seems to add higher complexity to the annotation process and results in lower agreement between contributors. 2. _\"En Tijuana la caravana migrante LGBT llego a una zona residencial rentando un AirPhB. Los vecinos se quejaron, pero en lugar de cuestionar la logistica gubernamental, mandaron mesajes de odio hacia los migrantes\"_.  
Figure 9: Annotation time distribution per text between experts and non-experts. Translation: \"In Tijuana, the LGBT caravan arrived in a residential neighborhood and rented an Airbnb. The neighbors complained, but instead of questioning the government's logistics, hate messages were sent to migrants\". Tweet number 2 was mislabeled as anti-immigrant by 11 contributors. Although the tweet mentions migrants, hate, and a specific vulnerable group, the author is denouncing the attitudes of the neighbors. In this specific case, contributors either misread the tweet or did not understand the guideline. To keep the annotation process as fast as possible for non-experts, we did not specify in the guideline that a tweet containing anti-immigrant speech or action was not necessarily anti-immigrant. Ideally, a very detailed guideline should be available for contributors to avoid such errors, but it would make the reading process longer, and \"time is of the essence\" when working with non-experts. For future works, an alternative solution would be to include an optional panel for contributors who need additional assistance in the annotation platform. It is quite interesting that the three positive instances of anti-immigrant speech with the highest number of erroneous classifications, presented below, all fall under criterion 3 described in Section 3.2.1. _\"Dennucian a migrante centroamericano por abusar sexualmente de una nitta en Tijuana\"_. Translation: \"Central American migrant is reported for sexually assaulting a girl in Tijuana\". _\"Detectan primer caso de variecia en Caravana Migrante en Mexico. Mexico, dede declarar alerta sanitaria, ante alto riesgo de contagio a la poblacion. Autoridades omiten casos de enfermedades infectocontagiosas y degenerativas de migrantes\"_. Translation: \"First chicken pox case detected in the migrant caravan in Mexico.