SlowGuess/ACL_2020_OCR huggingface dataset:
Serra Sinem Tekiroglu<sup>1</sup>, Yi-Ling Chung<sup>1,2</sup>, and Marco Guerini<sup>1</sup>
$^{1}$ Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy  $^{2}$ University of Trento, Italy
tekiroglu@fbk.eu, ychung@fbk.eu, guerini@fbk.eu
Recently research has started focusing on avoiding undesired effects that come with content moderation, such as censorship and overblocking, when dealing with hatred online. The core idea is to directly intervene in the discussion with textual responses that are meant to counter the hate content and prevent it from further spreading. Accordingly, automation strategies, such as natural language generation, are beginning to be investigated. Still, they suffer from the lack of sufficient amount of quality data and tend to produce generic/repetitive responses. Being aware of the aforementioned limitations, we present a study on how to collect responses to hate effectively, employing large scale unsupervised language models such as GPT-2 for the generation of silver data, and the best annotation strategies/neural architectures that can be used for data filtering before expert validation/post-editing.
Owing to the upsurge in the use of social media platforms over the past decade, Hate Speech (HS) has become a pervasive issue by spreading quickly and widely. Meanwhile, it is difficult to track and control its diffusion, since nuances in cultures and languages make it difficult to provide a clear-cut distinction between hate and dangerous speeches (Schmidt and Wiegand, 2017). The standard approaches to prevent online hate spreading include the suspension of user accounts or deletion of hate comments from the social media platforms (SMPs), paving the way for the accusation of censorship and overblocking. Alternatively, to weigh the right to freedom of speech, shadow-banning has been put into use where the content/account is not deleted but hidden from SMP search results. Still, we believe that we must overstep reactive identify-and-delete strategies to responsively intervene in
the conversations (Bielefeldt et al., 2011; Jurgens et al., 2019). In this line of action, some Non-Governmental Organizations (NGOs) train operators to intervene in online hateful conversations by writing counter-narratives. A Counter-Narrative (CN) is a non-aggressive response that offers feedback through fact-bound arguments and is considered as the most effective approach to withstand hate messages (Benesch, 2014; Schieb and Preuss, 2016). To be effective, a CN should follow guidelines similar to those provided in the 'Get the Trolls Out' project<sup>1</sup>, in order to avoid escalating the hatred in the discussion.
Still, manual intervention against hate speech is not scalable. Therefore, data-driven NLG approaches are beginning to be investigated to assist NGO operators in writing CNs. As a necessary first step, diverse CN collection strategies have been proposed, each of which has its advantages and shortcomings (Mathew et al., 2018; Qian et al., 2019; Chung et al., 2019).
In this study, we aim to investigate methods to obtain high quality CNs while reducing efforts from experts. We first compare data collection strategies depending on the two main requirements that CN datasets must meet: (i) data quantity and (ii) data quality. Finding the right trade-off between the two is in fact a key element for an effective automatic CN generation. To our understanding none of the collection strategies presented so far is able to fulfill this requirement. Thus, we test several hybrid strategies to collect data, by mixing niche-sourcing, crowd-sourcing, and synthetic data generation obtained by fine-tuning deep neural architectures specifically developed for NLG tasks, such as GPT-2 (Radford et al., 2019). We propose using an author-reviewer framework in which an author is tasked with text generation and a reviewer
<sup>1</sup> http://stoppinghate.getthemrollsout.org/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1177-1190 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
can be a human or a classifier model that filters the produced output. Finally, a validation/post-editing phase is conducted with NGO operators over the filtered data. Our findings show that this framework is scalable allowing to obtain datasets that are suitable in terms of diversity, novelty, and quantity.
We briefly focus on three research aspects related to hate online, i.e. available datasets, methodologies for detection, and studies on the effectiveness of the textual intervention. In the following section instead, we will focus on a few methodologies specifically devoted to HS-CN pairs collection.
Hate datasets. Several datasets have been collected from SMPs including Twitter (Waseem and Hovy, 2016; Waseem, 2016; Ross et al., 2017), Facebook (Kumar et al., 2018), WhatsApp (Sprugnoli et al., 2018), and forums (de Gibert et al., 2018), in order to perform hate speech classification (Xiang et al., 2012; Silva et al., 2016; Del Vigna et al., 2017; Mathew et al., 2018).
Hate detection. Most of the research on hatred online focuses on hate speech detection (Warner and Hirschberg, 2012; Silva et al., 2016; Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018) employing features such as lexical resources (Gitari et al., 2015; Burnap and Williams, 2016), sentiment polarity (Burnap and Williams, 2015) and multimodal information (Hosseinmardi et al., 2015) to build a classifier.
Hate countering. Recent work has proved that counter-narratives are effective in hate countering (Benesch, 2014; Silverman et al., 2016; Schieb and Preuss, 2016; Stroud and Cox, 2018; Mathew et al., 2019). Several CN methods to counter hatred are outlined and tested by Benesch (2014), Munger (2017), and Mathew et al. (2019).
3 CN Collection Approaches
Three prototypical strategies to collect HS-CN pairs have been presented recently.
Crawling (CRAWL). Mathew et al. (2018) focus on the intuition that CNs can be found on SMPs as responses to hateful expressions. The proposed approach is a mix of automatic HS collection via linguistic patterns, and a manual annotation of replies to check if they are responses that counter the original hate content. Thus, all the material collected is made of natural/real occurrences of HS-CN pairs.
Crowdsourcing (CROWD). Qian et al. (2019) propose that once a list of HS is collected from SMPs and manually annotated, we can briefly instruct crowd-workers (non-expert) to write possible responses to such hate content. In this case, the content is obtained in controlled settings as opposed to crawling approaches.
Nichesourcing (NICHE). The study by Chung et al. (2019) still relies on the idea of outsourcing and collecting CNs in controlled settings. However, in the nichesourcing the CNs are written by NGO operators, i.e. persons specifically trained to fight online hatred via textual responses that can be considered as experts in CN production.
4 Characteristics of the Datasets
Regardless of the HS-CN collection strategy, datasets must meet two criteria: quality and quantity. While quantity has a straightforward interpretation, we propose that data quality should be decomposed into conformity (to NGOs guidelines) and diversity (lexical & semantic). Additionally, HS-CN datasets should not be ephemeral, which is a structural problem with crawled data since, due to copyright limitations, datasets are usually distributed as a list of tweet IDs (Klubička and Fernández, 2018). By generating the data through crowdsourcing or nichesourcing, the problem is avoided.
Quantity. While the CRAWL dataset is very small and ephemeral, representing more a proof of concept than an actual dataset, the CROWD dataset involved more than 900 workers to produce  $\approx 41K$  CNs. On the other hand, the NICHE dataset is constructed by the participation of  $\approx 100$  expertoperators to obtain  $\approx 4K$  pairs (in three languages) and resorted to HS paraphrasing and pair translation to obtain the final  $\approx 14K$  HS-CN pairs. Evidently, employing non-experts, e.g., crowdworkers or annotators, is preferable in terms of data quantity.
Quality. In terms of quality, we consider that diversity is of paramount importance, since verbatim repetition of arguments can become detrimental for operator credibility and for the CN intervention itself. Following Li et al. (2016a), we distinguish between (i) lexical diversity and (ii) semantic diversity. While lexical diversity focuses on the diversity in surface realization of CNs and can be captured by word overlapping metrics, semantic diversity focuses on the meaning and is harder to be cap
tured, as in the case of CNs with similar meaning but different wordings (e.g., "Any source?" vs. "Do you have a link?").
(i) Semantic Diversity & Conformity. To model semantic diversity and conformity, we focus on the CN 'argument' types that are present in various datasets. Argument types are useful in assessing content richness (Hua et al., 2019). In a preliminary analysis, CROWD CNs are observed to be simpler and mainly focus on 'denouncing' the use of profanity while NICHE CNs are found richer with a higher variety of arguments. On the other hand, CRAWL CNs can cover diverse arguments to a certain extent while being highly prone to contain profanities. To perform a quantitative comparison, we randomly sampled 100 pairs from each dataset and annotated them according to the CN types presented by Benesch et al. (2016), which is the most comprehensive CN schema to our knowledge.
CRAWL	CROWD	NICHE
Hostile	50	0	0
Denouncing	16	76	10
Den. + Oth.	0	10	9
Other	34	14	81
RR	3.16	4.83	2.72
Table 1: Diversity analysis of the three datasets. Semantic diversity is reported in terms of CN type percentages, Lexical diversity in terms of Repetition Rate (RR - average over 5 shuffles).
The results are reported in Table 1. For the sake of conciseness we focus on the hostile, denouncing, and consequences classes, giving other to all remaining types (including the fact class). Clearly, CRAWL does not meet the conformity standards of CNs considering the vast amount of hostile responses  $(50\%)$ , still granting a certain amount of type variety (other:  $34\%$ ). Contrarily, CROWD conforms to the CN standards (hostile:  $0\%$ ), yet mostly focuses on pure denouncing  $(76\%)$  or denouncing with simple arguments  $(10\%)$ . The class other  $(14\%)$  consists of almost only simple arguments, such as "All religions deserve tolerance". In NICHE instead, arguments are generally and expectedly more complex and articulated, and represent the vast majority of cases  $(81\%)$ . Few examples of CN types are given in Table 2.
(ii) Lexical Diversity. The Repetition Rate (RR) is used to measure the repetitiveness of a collection of texts, by considering the rate of non-singleton n-gram types it contains (Cettolo et al., 2014; Bertoldi
et al., 2013). We utilize RR instead of the simple count of distinct ngrams (Xu et al., 2018; Li et al., 2016b) or the standard type-token ratio (Richards, 1987) since it allows us to compare corpora of diverse sizes by averaging the statistics collected on a sliding window of 1000 words. Since CROWD and NICHE contain repeated CNs for different  $\mathrm{HSs}^2$ , we first removed repeated CNs and then applied a shuffling procedure to avoid that CNs that are answering to the same HS (so more likely to contain repetitions) appear close together. Results in Table 1 show that NICHE is the dataset with more lexical diversity (lower RR), followed by CRAWL and CROWD.
Discussion. We can reasonably conclude that: (i) crawling, as presented in (Mathew et al., 2018), is not a mature procedure yet for CN collection, even if it is promising, (ii) nichesourcing is the one producing the best and most diverse material by far, however it is also the most challenging to implement considering the difficulty of making agreements with NGOs specialized in CN creation and it does not provide sufficient amount of data. (iii) On the contrary, CROWD seems to be the only one that can grant the amount of data that is needed for deep learning approaches, but contains more simple and stereotyped arguments. A summary of the pros and cons of each collection approach is presented in Table 3.
5 CN Generation through Author-Reviewer Architecture
Since none of the aforementioned approaches alone can be decisive for creating proper CN datasets, we propose a novel framework that combines crowdsourcing and nichesourcing to obtain new quality data while reducing collection cost/effort. The key elements of this combination are: (i) there must be an external element in the framework that produces HS-CN candidates, (ii) non-experts should prefilter the material to be presented/validated by experts. Thus, we settle on the author-reviewer modular architecture (Oberlander and Brew, 2000; Manurung et al., 2008). In this architecture the author has the task of generating a text that conveys the correct propositional content (a CN), whereas the reviewer must ensure that the author's output satisfies certain properties. The reviewer finally evaluates the text
2While this is an explicit data augmentation choice in NICHE, for CROWD it seems to derive from writing the same CNs for similar HSs by crowd-workers.
Hostile	“Hell is where u belong! Stupid f***t... go hang yourself!!”
Denouncing	“The N word is unacceptable. Please refrain from future use.”
Fact	“The majority of sexual assaults are committed by a family member, friend, or partner of the victim, and only 12% of convicted rapists are Muslim. It is not the religion, its the individuals, whether they’re Muslim or not.”
Table 2: Some examples of the categories relevant to our analysis. Hostile from CRAWL dataset, Denouncing from CROWD, Fact (other) from NICHE.
Quantity	Quality	Non-eph.
Conf.	Diver.
Crawl	✓	-	✓	-
Crowd.	✓	✓	-	✓
Niche.	-	✓	✓	✓
Table 3: Comparison of different approaches proposed in the literature according to the main characteristics required for the CN dataset.
Figure 1: The author-reviewer configuration. The author module produces HS-CN candidates and the reviewer(s) filter them. Finally, an NGO operator validates and eventually post-edits the filtered candidates.
viability and picks the ones to present to the NGO operators for final validation/post-editing.
The author-reviewer architecture that we propose differs from the previous studies in two respects: (i) it is used for data collection rather than for NLG, (ii) we modified the original configuration by adding a human reviewer and a final post-editing step.
We first tested four different author configurations, then three reviewer configurations keeping the best author configuration constant. A representation of the architecture is shown in Figure 1.
6 The Author: Generation Approaches
In order to obtain competent models that can provide automatic counter-narrative hints and suggestions to NGO operators, we have to overcome the data bottleneck/limitations, i.e. either the limited amount of training data in NICHE or its repetitiveness in CROWD, especially for using neural NLP approaches. Since pre-trained Language Models (LMs) have achieved promising results when fine-tuned on challenging generation tasks such as chit-chat dialog (Wolf et al., 2019; Golovanov et al., 2019), we propose using a recent large-scale language model GPT-2 (Radford et al., 2019).
GPT-2 is an unsupervised transformer-based (Vaswani et al., 2017) LM trained on a dataset of 8 million web pages, capable of generating coherent text and can be fine-tuned and/or conditioned on various NLG tasks. We used the medium model, which was the largest available during our experimentation and contains 345 million parameters, with 24 layers, 16 attention heads, and hidden state size of 1024. We fine-tuned two models with GPT-2, one on NICHE and one on CROWD datasets for counter-narrative generation.
NICHE - Training and test data. We have split 5366 pairs of HS-CN for training and the rest (1288 pairs) for testing. In particular, the original HS-CN pairs, one HS paraphrase, and the pairs translated from FR and IT were kept for training while the other HS paraphrases were used for testing. See Chung et al. (2019) for further details.
CROWD - Training and test data. Although the CROWD dataset was created for dialogue level HS-CN, we could extract HS-CN pairs by selecting the dialogues in which only 1 utterance was labeled as HS. Therefore, we could guarantee that the crowd-produced CNs are exactly for the labeled utterance. We then applied a 80/20 training and test split, obtaining 26320 and 6337 pairs.
Author	RR	Novel.	BLEU	BertS.
TRFcrowd	8.93	0.04	0.305	0.485
GPTcrowd	5.89	0.46	0.270	0.482
TRFniche	4.89	0.10	0.569	0.457
GPTniche	3.23	0.70	0.316	0.445
Table 4: Evaluation results of the best author configurations with different datasets. Novelty is computed w.r.t. to the corresponding training set, RR in the produced test output.
Generation Models. We fine-tuned GPT-2 $^3$ , with a batch size of 1024 tokens and a learning rate of 2e-5. The training pairs are represented as [HS_start_token] HS [HS_end_token] [CN_start_token] CN [CN_end_token]. While we empirically selected model checkpoint at the  $3600^{th}$  step of fine-tuning with NICHE dataset, with CROWD dataset we selected the checkpoint at the  $5000^{th}$  step. After fine-tuning the models the generation of CNs for the test HSs has been performed using Nucleus Sampling (Holtzman et al., 2019) with a p value of 0.9, which provides an enhanced diversity on the generation in comparison to the likelihood maximization decoding methods while preserving the coherency by truncating the less reliable tail of the distribution. At the test time, the input HSs are fed into models as conditions, which are used as the initial contexts while sampling the next tokens. Given an input HS, the models produce a chunk of text which is a list of HS-CN pairs of which the first sequence marked with [CN_start_token] CN [CN_end_token] is the generated output.
Baselines. In addition to the fine-tuned GPT-2 models, we also evaluate two baseline models. Considering the benefits of the transformer architectures on parallelization and learning long-term dependencies over recurrent models (Vaswani et al., 2017), we have implemented the baseline models using transformer architecture. The models have been trained similar to the base model described by Vaswani et al. (2017) with 6 transformer layers, batch size of 64, 100 epochs, 4000 warmup steps, input/output dimension of 512, 8 attention heads, inner-layer dimension of 2048, and drop-out rate of 0.1. We used Nucleus Sampling also for the baselines with a p value of 0.9 during decoding.
In brief, we have trained four different configu
rations/models as authors:
1.  $\mathbf{TRF}_{\text {crowd }}$ : baseline on CROWD dataset
2.  $\mathbf{GPT}_{\text {crowd }}$ : fine-tuned GPT-2 on CROWD dataset
3.  $\mathbf{TRF}_{\text {niche }}$ : baseline on NICHE dataset
4.  $\mathbf{GPT}_{\text {niche }}$ : fine-tuned GPT-2 on NICHE dataset
Metrics. We report both standard metrics (BLEU (Papineni et al., 2002), BertScore (Zhang et al., 2019)) concerning the lexical and semantic generation performances and a specific Diversity metric (RR) regarding the generation quality. As a second quality metric, we report Novelty (Wang and Wan, 2018) based on Jaccard similarity function (a variant of the same metric is used also by Dziri et al. (2019)). While diversity is used to measure the ability of the model to produce diverse/ varied responses with respect to the given input HS, novelty is used to measure how different the generated sequences are with regard to the training corpus (Wang and Wan, 2018).
Results. Results of the author model experiments are shown in Table 4. In terms of BLEU and BertScore, baseline models yield a better performance. However, a few peculiarities of CN generation task and the experiment settings hinder the direct and objective comparison of the presented scores among the models. First, gathering a finite set of all possible counter-narratives for a given hate speech is a highly unrealistic target. Therefore, we have only a sample of proper CNs for each HS, which is a possible explanation of very low scores using the standard metrics. Second, the train-test splits of NICHE dataset contain same CNs since the splitting has been done using one paraphrase for each HS and its all original CNs, while CROWD train-test splits have a similar property since an exact same CN can be found for many different HSs. Consequently, the non-pretrained transformer models, which are more prone to generating an exact sequence of text from the training set, show a relatively better performance with the standard metrics in comparison to the advanced pre-trained models. Some randomly sampled CNs, generated by the various author configurations, are provided in Appendix.
Regarding the generation quality, we observe that baseline models cannot achieve the diversity achieved by GPT-2 models in terms of RR - both for NICHE and CROWD (4.89 vs 3.23, and 8.93
3We adopted the fine-tuning implementation from https: //github.com/nshepperd/gpt-2
vs. 5.89). Moreover, GPT-2 provides an impressive boost in novelty (0.04 vs 0.46 and 0.10 vs 0.70). Among the GPT-2 models, the quality scores (in terms of RR and novelty) of the CNs generated by  $\mathrm{GPT}_{\text {niche }}$  are more than double in comparison to those generated with  $\mathrm{GPT}_{\text {crowd }}$ .
With regard to the overall results,  $\mathrm{GPT}_{\text {niche }}$  is the most promising configuration to be employed as author. In fact, we observed that, after the output CN, the over-generated chunk of text consists of semantically coherent brand-new HS-CN pairs, marked with proper HS/CN start and end tokens consistent with the training data representation. Therefore, on top of CN generation for a given HS, we can also take advantage of the over-generation capabilities of GPT-2, so that the author module can continuously output plausible HS-CN pairs without the need to provide the HS to generate the CN response. This expedient allows us to avoid the ephemerality problem for HS collection as well.
To generate HS-CN pairs with the author module, we basically exploited the model test setting and conditioned the fine-tuned model with each HS in the NICHE test-set. After removing the CN output for the test HS, we could obtain new pairs of HS-CN. In this way, we generate 2700 HS-CN pairs that we used for our reviewer-configuration experiments.
7 The Reviewer
The task of the reviewer is a sentence-level Confidence Estimation (CE) similar to the one of Machine Translation (Blatz et al., 2004). In this task, the reviewer must decide whether the author output is correct/suitable for a given source text, i.e. a hate speech. Consistently with the MT scenario, one application of CE is filtering candidates for possible human post-editing, which is conducted by the NGO operator by validating the CN. We tested three reviewer configurations:
1. expert-reviewer: Author output is directly presented to NGO operators.
2. non-expert-reviewer: Author output is filtered by human reviewers, then validated by operators.
3. machine-reviewer: Filtering is done by a classifier neural-architecture before operator validation.
7.1 Human Reviewer Experiment
In this section we describe the annotation procedure for the non-expert reviewer configuration.
Setup. We administered the generated 2700 HS-CN pairs to three non-expert annotators, and instructed them to evaluate each pair in terms of CN 'suitableness' with regard to the corresponding hate speech.
Instructions. We briefly described what an appropriate and suitable CN is, then we instructed them not to overthink during the evaluation, but to give a score based on their intuition. We also provided a list of 20 HS-CN pairs exemplifying the proper evaluation.