dataesr/scientific-paragraphs-categorization huggingface dataset:
We acknowledge financial support from the  US National Science Foundation   SBR 9024683  ( 1991-1993 );  BCS 9615291  ( 1996-1998 );  BCS 0106664 (2002-2004 );  BCS 0352681 (2003-2004 );  BCS 0648762  (2007-2009;  BCS 1030081  ( 2010-2012 ), the  US National Endowment for the Humanities  ( 2012-2014 ), the  US IIE  J. W. Fulbright Sr Scholars Program 2002-2003; 2012), and the  National Geographic Society  ( 1996-1997 ). The research of S. W. -J. and J. F. at  Songo Mnara  was supported by the  Antiquities Division of the Ministry of Natural Resources and Tourism, Tanzania , and also by the  National Science Foundation  ( BCS 1123091 ) and the  Arts and Humanities Research Council  ( AH/J502716/1 ). The Madagascar modern sample collection was supported by the MAGE consortium. D. R. is an Investigator of the  Howard Hughes Medical Institute , and the ancient DNA laboratory work and analysis were also supported by  National Institutes of Health  grant  HG012287 , by  John Templeton Foundation  grant  61220 , by the  Allen Discovery Center program , which is a  Paul G.
First, the email addresses of 171 experts were retrieved from Google Scholar by using the keywords "misinformation" and "fake news" (first twelve pages since 2018). Second, 93 experts were recruited from the program committee of MISDOOM, an international conference on mis/disinformation. For this, we collected their email address and emailed them directly. Third, 95 experts were contacted via the De Facto mailing list, a network of French experts on misinformation, and 36 experts were contacted because they attended an ICA pre-conference on the future of mis/disinformation studies. Fourth, 190 experts were contacted via snowball sampling: they were recommended by other experts at the end of the survey. The experts were also allowed to share the survey with their lab or For the future of the misinformation field, it's important to.
Currently, a significant portion of published research on online hate speech relies on existing textual corpora. However, when examining a specific context, there is a lack of preexisting datasets that include the particularities associated with various conditions (e. g. , geographic and cultural). This issue is evident in the case of online anti-immigrant speech in Mexico, where available data to study this emergent and often overlooked phenomenon are scarce. In light of this situation, we propose a novel methodology wherein three domain experts annotate a certain number of texts related to the subject. We establish a precise control mechanism based on these annotations to evaluate non-expert annotators. The evaluation of the contributors is implemented in a custom annotation platform, enabling us to conduct a controlled crowdsourcing campaign and assess the reliability of the obtained data.
Our results demonstrate that a combination of crowdsourced and expert data leads to iterative improvements, not only in the accuracy achieved by various machine learning classification models (reaching 0. 8828) but also in the model's adaptation to the specific characteristics of hate speech in the Mexican Twittersphere context. In addition to these methodological innovations, the most significant contribution of our work is the creation of the first online Mexican anti-immigrant training corpus for machine-learning-based detection tasks.
Introduction Nowadays, high-quality data are a valuable resource because modern artificial intelligence (AI) depends entirely on them. Numerous AI applications use machine learning methods that must be adjusted by processing large amounts of good-quality data. However, accessing high-quality data is one of the greatest challenges in using machine learning algorithms, in cases such as the study of online anti-immigrant speech in Mexico, a growing issue repeatedly pointed out by the Mexican government and international organizations  [1, 2]. Despite having a vast amount of HS resources available from public data repositories, there are no Mexican anti-immigrant speech resources for natural language processing. The closest works that we have found are not related to Mexican Spanish anti-immigrant speech (see Section 1.
2). In this scenario, crowdsourcing is an alternative when there are no data to use in specific machine learning tasks. However, although crowdsourcing is an interesting solution, it presents serious challenges. It is difficult to control the quality of the data collected and annotated by a large group of non-experts. The algorithm's accuracy can be significantly affected by human errors and spammers, which can lead to poor data quality. The latter can ultimately cause useless models for in-production technology. According to the Harvard Business Review, "poor data quality is enemy number one to the widespread, profitable use of machine learning"  [3]. In this article, we construct a high-quality training corpus using a mixed approach combining expert annotation and non-expert annotation obtained with crowdsourcing to study Mexican anti-immigrant speech on Twitter.
Moreover, anyone with access to Web 2. 0 can now generate content, express their opinions, and react to content produced by others. The inherent distance in online interactions and the varying degrees of the echo-chamber effect and competition tend to create an online debate that is more polarized than its traditional counterparts, especially on controversial and widely discussed topics such as migration  [10]. Recently, several scholars have studied the emergence of anti-immigrant speech in Mexican social media  [2, [11] [12] [13]. As relevant as they are, these works only consider a small portion of the anti-immigrant discourse published on Mexican social networks, both in terms of the amount of data considered and the temporal coverage of the analysis. We still have very little visibility on this emerging phenomenon.
Several recent analyses have been conducted on anti-immigrant or xenophobic content detection in specific cultural, temporal, and geographical areas. In English, Pitropakis et al. collected anti-immigrant tweets published in the UK, the U. S. , and Canada following specific events related to immigration in these three nations  [16]. Siegel et al. studied white nationalist rhetoric on Twitter during the 2016 United States presidential election and relied on a mixed approach combining a dictionary and supervised machine learning to detect racist and xenophobic speech in a dataset of more than one billion tweets  [17]. Another research study was conducted in European countries at the forefront of the unfolding migratory crisis in 2015. In Italy, researchers created a corpus in Italian of offensive tweets against immigrants, Muslims, and Rom  [18, 19].
They used a mixed annotation process that included both experts and paid crowd workers to follow the same novel multilayered classification scheme that considered the presence of HS but also five other categories: aggressiveness, offensiveness, irony, stereotypes, and intensity  [20]. Anti-immigrant speech detection in Spanish was the object of a collective event at SemEval 2019. In Task 5, participants were provided with a publicly available dataset, HatEval, that contained 1991 labeled tweets regarding immigrants  [21]. To build the dataset, they relied on expert annotation using a combination of techniques to collect data from Twitter, including hateful keywords related to Spanish and Latin American contexts. Plaza-Del-Arco et al. and  Hasan et al. used  HatEval to train and compare several machine learning models to detect HS against immigrants  [22, 23].
Arcila-Calderón et al. manually created an ad hoc dataset to train deep and shallow learning models to detect anti-immigrant speech in European Spanish  [15]. Based on a broad definition of HS against migrants, considering xenophobia and racism, they downloaded and filtered Twitter data with keywords assembled during an exploratory qualitative stage conducted in the Spanish Twittersphere. Regarding specific aggressiveness detection for Latin American Spanish, the most relevant works are the MEX-A3T track at IberLEF 2019  [24] , where the organizers considered two tasks focused on authorship and aggressiveness in Mexican tweets, and the Language Model for Misogyny Detection in Latin American Spanish  [25]. Inspired by the investigations described in this section and building on previous work  [26] , this paper presents new methodological elements to build a corpus for HS detection under certain constraints.
We worked with a group of experts for the manual annotation process, had no financial resources to use crowdsourcing services such as those provided by Crowdflower, and had to build our own annotation platform. Lastly, our work stands out because we built an anti-immigrant speech corpus in a very specific temporal and geographical context not yet considered in Mexican Spanish. Moreover, our work expands the horizon of online anti-immigrant speech detection in specific linguistic, geographical, and temporal contexts by building a Mexican Spanish corpus, which is particularly relevant as migration is becoming an increasingly polarizing and politically instrumentalized topic. All these contributions together make this a novel methodology to obtain quality data for machine learning.
Research Purpose and Research Questions The general objective of this work is to propose a sophisticated methodology to create a dataset of human-annotated tweets that can then be used to train machine learning models to detect online anti-immigrant speech in Mexican Spanish. This can be seen as the first stepping stone in the research project briefly presented in Section 1. 1 and described in a previous communication  [26]. Thus, the dataset must not only be adequate for machine learning purposes to achieve internal statistical accuracy, but it also must represent "real" data that can be used to conduct an extensive analysis of Mexican anti-immigrant online speech produced on Twitter between 2018 and 2022. More specifically, this work aims to overcome one of the limitations of a human-labeled corpus: the significant amount of time required by expert annotators to label a large corpus of examples by hand.
Assembling the First Training Corpus The qualitative exploratory process in Stage I was an opportunity to build the first instance of an anti-immigrant dataset. While exploring the Mexican online conversation on Twitter, one expert collected positive and negative instances of anti-immigrant speech based on the definition described in Section 3. 2. 1. We included a corpus of misogynistic posts used in previous work  [28]  to bolster negative instances of anti-immigrant speech. After this, we retrieved the first instance of the corpus, which was composed of 1073 anti-immigrant tweets and 4548 negative instances. Using this first dataset, we trained a convolutional neural network (CNN) for binary classification and achieved an initial accuracy rate of 0. 7675 (see Table  2 ), as detailed in  [26].
3. Anti-immigrant speech uses negative stereotypes to qualify migrants. In our dataset, the "qualities" attributed most often to migrants were "illegals", "lazy", "demanding", "poor", "dirty", "irresponsible", and "criminals", as illustrated in the tweet below: "La redacción completa es: refugio para migrantes maras salva truchas y todos los delincuentes que uyen de otros países. unos genios!" Translation: "The entire story is to give refuge to Mara Salvatrucha migrants and all criminals fleeing from other countries. Such geniuses!". 4. Anti-immigrant speech often presents migrants as invaders. This idea is built on the representation of a geographical space being violently disrupted: "su ayuda solo favorece y promueve que nuevas caravanas sigan llegando trastornando enormemente la vida cotidiana de las ciudades fronterizas de México" Translation: "Your help only encourages and fosters new caravans to arrive while causing havoc in the daily life of Mexican border towns".
Agreement Metrics To evaluate the quality and operability of the five criteria, we asked three experts to classify a small dataset of tweets retrieved during the previous stage. The group of experts was integrated with two Ph. D. candidates who specialized in migration issues and one professor who specialized in border studies and migration in the Mexican southern region. Since our goal was to test the limits of the guideline's usefulness to classify potentially problematic and ambiguous tweets, we did not build the dataset randomly. We manually selected a small amount of clearly pro-migrant and neutral tweets, a large amount of antiimmigrant tweets from the sample at hand, and some "problematic" examples. The latter were difficult to classify because of a lack of context (irony, sarcasm), a mixture of forms and ideas characteristic of both pro-and anti-immigrant speech, or the group or individual targeted was not clearly identifiable.
Lastly, we selected 76 tweets. The experts had to independently classify the dataset using the criteria. We then obtained the experts' feedback regarding the criteria and specific tweets where no agreement was reached. As expected, some anti-immigrant tweets were not easily identifiable. We agreed on 73 tweets, three unclassified. Inter-rater agreement was needed to assess the reliability of the classifications made by the aforementioned experts. This was important to ensure that we were working with high-quality control data. We established a baseline rate to evaluate the agreement between expert annotators with a statistical metric named the kappa coefficient value. Its formula is presented in Equation  (1). κ = P o -P e 1 -P e (1) In Equation (  1 ), we see the kappa coefficient, where P o is the observed proportion of agreement between annotators and P e is the expected proportion of agreement between annotators due to luck.
Splitting into Annotation Data Pools The main process to compose Pool A was data filtering. In this pool, we tried to concentrate on relevant tweets for annotation. The raw data obtained from AGEI (Section 3. 1. 1) were filtered using the spatiotemporal criteria defined by dates of interest (Table  1 ). Note that experts selected the days of interest. They recognized special events related to migrants traversing Mexico. Thus, the spatiotemporal filtering retained only tweets published on such days in the Mexican territory and part of its northern and southern borders. A second filter split the result of the first filter based on a set of keywords related to the subject. The split created the first two pools: Pool A was composed of alleged anti-immigrant data, and Pool B was composed of alleged non-anti-immigrant data.
Annotation Batches with Trade-Off in Time and Quality To consider the time and quality trade-off, we asked 32 users to annotate the control data. Our objective was to observe the time required by the users to complete the task. They had to determine whether a tweet contained anti-immigrant speech or not. The average time to classify a tweet was 12. 583 s, as shown in Figure  4. Based on the average time obtained, we conducted a simulation experiment to determine the appropriate batch size for annotation. We wanted to ensure that the non-expert users did not spend more than 20 min annotating. Considering that the average time spent annotating the control data was 6. 29 min, we selected a real annotation batch of 50 tweets, since this would require approximately 10. 49 min for the user to annotate, resulting in a total of 17.
Designing the Crowdsourcing Platform Traditionally, data annotation is one of the most time-consuming tasks to obtain a language model due to its manual process. A set of annotators must observe texts and classify them manually, usually in a spreadsheet  [26, 33]. Moreover, as the number of documents grows, this traditional method becomes impractical and inefficient. An annotation platform becomes necessary to address these challenges. It provides a user-friendly interface to annotate data by following a well-defined methodology with quality control mechanisms to improve the overall performance of the language model. Moreover, it allows many annotators to work simultaneously on different parts of the dataset, regardless of the location of the annotators, translating into greater productivity and a significant reduction in the time required.
At the outset, we explored an annotation platform called Doccano, which allows users to annotate data at no cost, as the authors in  [15]  did. However, our specific requirements necessitated certain features not readily available on the platform. For instance, we aimed to ensure the quality of the annotated data by evaluating the users' performance in the control pool and comparing it to the inter-annotator agreement obtained in Section 3. 3. 2 (0. 627). Additionally, we needed a mechanism where each user could annotate at the same time a different portion of the dataset by establishing the number of documents allocated for annotation and the inter-annotator coefficient in the platform, ensuring that only if the user passed the set kappa coefficient would they be able to annotate the following 50 unlabeled documents from the dataset.
As a result of these requirements and considerations, we decided to develop a custom annotation platform from scratch. This approach allowed us to tailor the platform to meet the needs of our research project. Furthermore, we recognized the value of contributing to the research community by making the platform open source-it can be found in the GitHub repository-enabling other researchers to benefit from and customize it according to their specific annotation requirements, as well as obtaining the data (on demand): https://github. com/Kgazcah/Annotation-platform, accessed on 19 June 2023.
User Interface The platform offers users the possibility to upload the dataset that is to be annotated. Additionally, the control data, previously annotated by experts, can be uploaded. It is also possible to configure the inter-annotator kappa agreement coefficient, which is a measure to quantify the annotated data's quality. The platform allows the configuration of more parameters, such as the size of the data chunks that users will annotate after passing the predefined threshold. The admin panel displays all the set configurations, including the total number of documents in the dataset and the progress of annotations completed. Our platform also offers the flexibility to edit these parameters, delete datasets if necessary, and obtain a shareable link to invite multiple users to participate.
We used the identified confusing words (e. g. , "delincuente", "negr") to search and include examples of texts using these words but having the negative class. As a result, after retraining a classification model with better data, the produced model increased its accuracy and achieved the better sorting of the remaining data and a better understanding of the pragmatics in anti-immigrant speech. It is worth mentioning that the bias detected by words such as "delincuente" (criminal) or "negro" (black) is not a bias associated with the quality of the data but rather a problem inherent to the classifier itself. This is important because we discovered that the final models could conveniently generalize the differences in the anti-immigrant speech, which had a good impression on the experts.
Result 1: Assessing the Performance of the Crowdsourced Annotation Protocol To address RQ1, we conducted two experiments to evaluate the performance of the crowdsourced annotation protocol. The first experiment focused on obtaining highquality data annotations by following the annotation protocol outlined by experts in the field, mentioned in Section 3. 2, and involving as many non-expert annotators as possible. The second experiment aimed to analyze the time that non-expert annotators took compared to expert annotators, and the role of the crowdsourced annotation platform in facilitating the process. For the first experiment, 47 contributors participated in the crowdsourced annotation process, as described in Section 3. 4. Not all contributors were able to pass the control mechanism described in 3.
Ideally, all unlabeled data should be annotated by experts, as this would likely result in faster annotation overall. However, it became clear that annotating a substantial amount of documents solely with the help of experts would require a significant amount of time. In light of this, we decided to leverage the annotations provided by non-expert contributors that adhered to the expert criteria that we were looking for. Although it was evident that these nonexpert annotators took nearly three times longer than the experts, shown in Figure  9 , their contributions were valuable. Here, the crowdsourced annotation platform proved to be advantageous. By allowing multiple non-experts, who had passed the threshold, to annotate different parts of the dataset simultaneously or independently, we optimized the annotation process and significantly reduced the overall time that it would have taken otherwise.
Result 3: Any Model Will Learn To answer RQ3, we designed an experiment to prove that after using the proposed methodology, we obtained good-quality labeled data ready for machine learning algorithms. For the experiment, we used a set of 5272 manually labeled tweets, through the platform described in Section 3. 4, half of which were positive (anti-immigrant) and half were negative (non-anti-immigrant). The texts were transformed into vectors so that they could function as input for three different types of models, so that we could answer RQ3. The vectorization consisted of using a previously trained convolutional neural network (described in a previous work  [26] ) as if it were an encoder. In other words, we use the text of the tweets as input and obtained their 250-dimensional vector representation from the Keras API Embedding layer.
Result 4: Interpretability Analysis To complete the analysis of our corpus, we attempted to establish more effective terms associated with anti-immigrant speech in this context. We used a representative sample of the collected data to obtain a classification model via a white box learner that gives a classification result and also allows us to infer the relationships between the results and the terms or words given as explanatory variables. We used a bag-of-words vectorizer and a linear learner (logistic regression) to map directly between individual words and the model coefficients, and also to describe the mathematical relationship between each word and the dependent variable associated with the class to be predicted (positive and negative anti-immigrant speech). The coefficient values indicate how much the class likelihood changes if there is a one-unit shift in an independent word while holding the remaining words in the model constant.
Remarks and Limitations The research project described in this paper contributes to the field of HS detection by providing several resources for other machine-learning-focused works. Firstly, we created the first corpus of anti-immigrant tweets written in Mexican Spanish and, to our knowledge, the first dataset in Latin American Spanish on this specific type of HS. At the time of writing, 3326 positive and 8256 negative instances compose the "Mexican Anti-Immigrant Speech Training Corpus" (MAISTC). Although the number of negative and positive examples might seem low, we focused on the quality of the data over quantity, while still retaining sufficient instances to train an automatic detection model with acceptable performance. Our dataset has "real" and concrete examples of HS and non-HS produced in the Mexican Twittersphere and collected through a rigorous annotation protocol based on empirical knowledge and solid definitions.