# High-Quality Data from Crowdsourcing towards the Creation of a Mexican Anti-Immigrant Speech Corpus
## 1 Introduction
### Related Work  
There is growing interest in the scientific community regarding the automatic detection of online hate speech (HS), online content that spreads hate against a particular group or individual based on characteristics such as gender, origin, culture, or religious belief. [PERSON] et al. recently published an overview of existing research and resources on such topics [14]. Until recently, most research has focused on HS as a generic phenomenon and has been published in English, the most spoken language. However, HS is a complex object that varies in intensity, directness, targeted groups, and cultural and geographical context. In other words, HS is related to a particular temporal and territorial context. Quoting [PERSON] et al. [15], undertaking a generic approach \"could be a limitation because the resulting models may not be as effective, reliable, and, paradoxically, generalizable as those trained with real examples of a specific context, a specific type of hate, and a specific discriminatory category, separating and differentiating concepts, characteristics, and linguistic nuances\". Several recent analyses have been conducted on anti-immigrant or xenophobic content detection in specific cultural, temporal, and geographical areas. In English, [PERSON] et al. collected anti-immigrant tweets published in the UK, the U.S., and Canada following specific events related to immigration in these three nations [16]. [PERSON] et al. studied white nationalist rhetoric on Twitter during the 2016 United States presidential election and relied on a mixed approach combining a dictionary and supervised machine learning to detect racist and xenophobic speech in a dataset of more than one billion tweets [17]. Another research study was conducted in European countries at the forefront of the unfolding migratory crisis in 2015. In Italy, researchers created a corpus in Italian of offensive tweets against immigrants, Muslims, and Rom [18; 19]. They used a mixed annotation process that included both experts and paid crowd workers to follow the same novel multilayered classification scheme that considered the presence of HS but also five other categories: aggressiveness, offensiveness, irony, stereotypes, and intensity [20]. Anti-immigrant speech detection in Spanish was the object of a collective event at SemEval 2019. In Task 5, participants were provided with a publicly available dataset, HatEval, that contained 1991 labeled tweets regarding immigrants [21]. To build the dataset, they relied on expert annotation using a combination of techniques to collect data from Twitter, including hateful keywords related to Spanish and Latin American contexts. [PERSON] et al. and [PERSON] et al. used HatEval to train and compare several machine learning models to detect HS against immigrants [22; 23]. [PERSON] et al. manually created an ad hoc dataset to train deep and shallow learning models to detect anti-immigrant speech in European Spanish [15]. Based on a broad definition of HS against migrants, considering xenophobia and racism, they downloaded and filtered Twitter data with keywords assembled during an exploratory qualitative stage conducted in the Spanish Twittersphere. Regarding specific aggressiveness detection for Latin American Spanish, the most relevant works are the MEX-A3T track at IberLEF 2019 [24], where the organizers considered two tasks focused on authorship and aggressiveness in Mexican tweets, and the Language Model for Misogyny Detection in Latin American Spanish [25]. Inspired by the investigations described in this section and building on previous work [26], this paper presents new methodological elements to build a corpus for HS detection under certain constraints. We worked with a group of experts for the manual annotation process, had no financial resources to use crowdsourcing services such as those provided by Crowdflower, and had to build our own annotation platform.