SlowGuess/ACL_2020_OCR huggingface dataset:
Measurement. We opted for a scale of 0-3, rather than a CE binary response, since it allows us to study various thresholds for better data selection. In particular, the meanings of the scores are as follows: 0 is not suitable; 1 is suitable with small modifications, such as grammar or semantic; 2 is suitable; and 3 is extremely good as a CN. We also ask to discard the pairs in which the hate speech was not well formed. For each pair we gathered two annotator scores.
Filtered Data. After the non-expert evaluation, we applied two different thresholds to obtain the pairs to be presented to the expert operators: (i) at least a score of 2 by both annotators  $(\mathrm{Reviewer}_{\geq 2})$  yielding high quality data where no post editing is necessary, (ii) at least a score 1 by both annotators  $(\mathrm{Reviewer}_{\geq 1})$  providing reasonable quality with a possible need for post-editing.
The statistics reported in Table 5 show that high quality pairs (Reviewer $_{\geq 2}$ ) account for only a small fraction (10%) of the produced data and only one third was of reasonable quality (Reviewer $_{\geq 1}$ ), while the vast majority was discarded. Some randomly selected filtered pairs are provided in Appendix.
Threshold	count	Percentage
Reviewer≥2	276	10.0%
Reviewer≥1	902	32.6%
at least one 0	1723	62.2%
bad HS	145	5.2%
Reviewermachine	-	40.2%
Table 5: Percentage of filtered pairs according to various filtering conditions.
7.2 Machine Reviewer Experiment
As the machine reviewer we implemented 2 neural classifiers tasked with assessing whether the given HS-CN is a proper data pair. The two models are based on BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2019) architectures.
Training data. We created a balanced dataset with 1373 positive and 1373 negative examples for training purposes. The positive pairs come both from NICHE dataset and from the examples annotated in the human reviewer setting (Reviewer $_{\geq 2}$ ). The negative pairs consist of the examples annotated in the human reviewer setting, in the 'at least one 0' bin. In addition, 50 random HSs from NICHE-training are utilized with verbatim repetition as HS-HS to discourage the same text for both HS and CN in a pair, and 50 random HSs are paired with other random HSs simulating the condition of inappropriate CNs with hateful text.
Test data. We collected a balanced test set, with 101 positive and 101 negative pairs. Both positive and negative examples are created replicating the non-expert reviewer annotation described in Section 7.1 for new CN generation with NICHE test set by using the author model  $\mathrm{GPT}_{\text {niche }}$ .
Models. For the first model, we follow the standard sentence-pair classification finetuning schema of the original BERT study. First, the input HS-CN is represented as [CLS] HS_tokens [SEP] CN_tokens [SEP] and fed into BERT. By using the final hidden state of the first token [CLS] as the input, originally denoted as  $C \in R^H$ , we obtain a fixed-dimensional pooled representation of the input sequence. Then, a classification layer is added with the parameter matrix  $W \in R^{KH}$ , where K denotes the number of labels, i.e. 2 for HS-CN classification. The cross-entropy loss has been used during the fine-tuning.
We have conducted a hyperparameter tuning phase with a grid-search over the batch sizes 16 and 32, the learning rates [4,3,2,1]e-5 and the number of epochs in the range of 3 to 8. We obtained the best model by fine-tuning uncased BERT-large, with a learning rate of 1e-5, batch size of 16, and after 6 epochs at the  $1029^{th}$  step on a single GPU.
The second model is built by fine-tuning ALBERT, which shows better performance than BERT on inter-sentence coherence prediction by using a sentence-order prediction loss instead of next-sentence prediction. In sentence-order prediction loss, while the positive examples are created sim
ilar to BERT by using the consecutive sentences within the same document, the negative examples are created by swapping sentences, which leads the model to capture the discourse-level coherence properties better (Lan et al., 2019). This objective is particularly suitable for HS-CN pair classification task, since HS and CN order and their coherence are crucial for our task. We fine-tuned ALBERT similarly to BERT model, by adding a classification layer on top of it. We applied the same grid-search that we used for BERT model to fine-tune ALBERT-xxlarge which contains 235M parameters. We saved a checkpoint at every 200 steps and finally, obtained the best model by using the learning rate of 1e-5, the batch size of 16, and at the  $1200^{th}$  step. $^4$
Metrics. To find the best model for machine reviewer, we compared BERT and ALBERT models over the test set. Although it seems more intuitive to focus on precision since we search for an effective filtering over many possible solutions, we observed that a model with a very high precision tends to overfit on generic responses, such as "Evidence please?". Therefore, we aim to keep the balance between the precision and recall and we opted for F1 score for model selection. We report the best configurations for each model in Table 6, and the percentage of filtered pairs in Table 5. ALBERT classifier outperformed BERT model in all three metrics; F1, Precision, and Recall. Considering  $6\%$  of absolute F1 score improvement with respect to BERT model, we employed ALBERT model as the Machine Reviewer.
Reviewermachine	F1	Precision	Recall
ALBERT	0.73	0.74	0.73
BERT	0.67	0.69	0.65
Table 6: F1, Precision and Recall results for the two main classifier configurations we tested.
8 NGO Operators Experiments
To verify that the author-reviewer approach can boost HS-CN data collection, we run an experiment with 5 expert operators from an NGO. We compared the filtering strategies to reveal the best depending on several metrics.
4All the experiments have been conducted on a single GeForce RTX 2080 Ti GPU. Only the ALBERT classifier model has been trained with 8 TPU cores on Google Cloud.
Approach	NGO time	Crowd time	RR	Novelty	Pairs selec	Pairs final
no suggestion	480	-	2.72	-	-	-
Reviewer expert	76	-	3.56	0.73	100%	45%
Reviewer ≥ 1	72	215	4.31	0.70	33%	54%
Reviewer machine	68	-	4.48	0.68	40%	63%
Reviewer ≥ 2	49	703	5.70	0.65	10%	72%
Table 7: Results for CN collection under various configurations. RR for 'no suggestion' is computed on NICHE dataset and the time needed is the one reported in (Chung et al., 2019). Time is expressed in seconds. Pairs<sub>select</sub> indicates the percentage of original author pairs that have been passed to the expert for reviewing, Pairs<sup>final</sup> indicates the percentage of selected pairs that have been accepted or modified by the expert themselves. Crowd<sub>time</sub> is computed considering that annotators gave a score every 35 seconds, and we required two judgments per pair.
Within Subject Design. We administered lists of HS-CN pairs to 5 operators from each filtering condition, and instructed them to evaluate/modify each pair in terms of 'suitableness' of the CN to the corresponding HS.
Instructions. For each HS-CN pair, we asked the operators: a) if the CN is a perfect answer, to validate it without any modification, b) if the CN is not perfect, but a good answer can be obtained with some editing, to modify it, c) if the CN is completely irrelevant and/or needs to be completely rewritten to fit the given HS, to discard it.
Measurement. The main goal of our effort is to reduce the time needed by experts to produce training data for automatic CN generation. Therefore the primary evaluation measure is the average time needed to obtain a proper pair. The other measurements of interest are Diversity and Novelty, to understand how the reviewing procedure can affect the variability of the obtained pairs.
Procedure and material. We gave the instructions along with a list of 20 HS-CN exemplar pairs for each condition (i.e.  $\text{Reviewer}_{\geq 1}, \geq 2, \text{machine\_expert}$ ). The condition order was randomized to avoid primacy effect. In total, each NGO operator evaluated 80 pairs. Pairs were sampled from the pool of 2700 pairs described before (apart from the automatic filtering condition). To guarantee that the sample was representative of the corresponding condition, we performed a stratified sampling and avoided repeating pairs across subjects.
Results and Discussion. As it is shown in Table 7, there is a substantial decrease in data collection time (NGO  $time$ ) when automatic generation mechanisms are introduced (no suggestion vs. Reviewer  $expert$ ). If crowd filtering is applied (Reviewer  $\geq 1$ ,  $\geq 2$ ), the amount of time can be further reduced, and the more stringent the filtering criterion, the higher the time saved. Conversely,
the more stringent the filtering criterion, the higher the time to obtain a filtered pair from non-expert annotators  $(\mathrm{CROWD}_{time})$ . For instance to obtain a single pair with at least a score of 2 by both annotators, 700 sec (around  $12\mathrm{min}$ ) are needed on average (only  $10\%$  of examples are in  $\geq 2$  condition). Results indicate that providing an automatic generation tool meets the first goal of increasing efficiency of the operators in data collection.
Regarding diversity and novelty metrics, prefiltering author's output (Reviewer $_{\geq 1}$ ,  $\geq 2$  and machine) has a negative impact: the more stringent the filtering condition the higher the RR and the lower the novelty of the filtered CNs. We performed some manual analysis of the selected CNs and we observed that especially for the Reviewer $_{\geq 2}$  case (which was the most problematic in terms of RR and novelty) there was a significantly higher ratio of "generic" responses, such as "This is not true." or "How can you say this about an entire faith?", for which reviewers agreement is easier to attain. Therefore, the higher agreement on the generic CNs reveals itself as a negative impact in the diversity and novelty metrics. Conversely, the percentage of pre-filtered pairs that are accepted by the expert increases with the filtering condition becoming more stringent, the baseline being  $45\%$  for the Reviewer $_{\text{expert}}$  condition.
As for the amount of operators' effort, we observed a slight decrease in HTER with the increase of pre-filtering conditions, indicating an improvement in the quality of candidates. However, HTER scores were all between 0.1 and 0.2, much below the 0.4 acceptability threshold defined by Turchi et al. (2013), indicating that operators modified CNs only if "easily" amendable.
<sup>5</sup>Human-targeted Translation Edit Rate is a measure of post-editing effort at sentence level translations (Specia and Farzindar, 2010).
Finally, we observe that despite reducing the output diversity and novelty, the reduction of expert effort by  $\mathrm{Reviewer}_{\geq 2}$  in terms of the percentage of the obtained pairs is not attainable by a machine yet. On the other hand, automatic filtering (Reviewer_machine) is a viable solution since (i) it helps the NGO operators save time better than human filter  $\geq 1$ , (ii) it preserves diversity and novelty better than  $\mathrm{Reviewer}_{\geq 2}$  and in line with  $\mathrm{Reviewer}_{\geq 1}$ .
To counter hatred online and avoid the undesired effects that come with content moderation, intervening in the discussion directly with textual responses is considered as a viable solution. In this scenario, automation strategies, such as natural language generation, are necessary to help NGO operators in their countering effort. However, these automation approaches are not mature yet, since they suffer from the lack of sufficient amount of quality data and tend to produce generic/repetitive responses. Considering the aforementioned limitations, we presented a study on how to reduce data collection effort, using a mix of several strategies. To effectively and efficiently obtain varied and novel data, we first propose the generation of silver counter-narratives – using large scale unsupervised language models – then a filtering stage by crowd-workers and finally an expert validation/post-editing. We also show promising results obtained by replacing crowd-filtering with an automatic classifier. As a final remark, we believe that the proposed framework can be useful for other NLG tasks such as paraphrase generation or text simplification.
This work was partly supported by the HATEMETER project within the EU Rights, Equality and Citizenship Programme 2014-2020. We are grateful to Stop Hate UK that provided us with the experts for the evaluation. Finally, there are also many people we would like to thank for their help and useful suggestions: Eneko Agirre, Simone Magnolini, Marco Turchi, Sara Tonelli and the anonymous reviewers among others.
Susan Benesch. 2014. Countering dangerous speech: New ideas for genocide prevention. Washington, DC: United States Holocaust Memorial Museum.
Susan Benesch, Derek Ruths, Kelly P Dillon, Haji Mohammad Saleem, and Lucas Wright. 2016. Counterspeech on twitter: A field study. Dangerous Speech Project. Available at: https://dangerousspeech.org/counterspeech-ontwitter-a-field-study/.
Nicola Bertoldi, Mauro Cettolo, and Marcello Federico. 2013. Cache-based online adaptation for machine translation enhanced computer assisted translation. In MT-Summit, pages 35-42.
Heiner Bielefeldt, Frank La Rue, and Githu Muigai. 2011. Ohchr expert workshops on the prohibition of incitement to national, racial or religious hatred. In Expert workshop on the Americas.
John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In *Coling* 2004: Proceedings of the 20th international conference on computational linguistics, pages 315-321.
Pete Burnap and Matthew L Williams. 2015. Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making. *Policy & Internet*, 7(2):223-242.
Pete Burnap and Matthew L Williams. 2016. Us and them: identifying cyber hate on twitter across multiple protected characteristics. *EPJ Data Science*, 5(1):11.
Mauro Cettolo, Nicola Bertoldi, and Marcello Federico. 2014. The repetition rate of text as a predictor of the effectiveness of machine translation adaptation. In Proceedings of the 11th Biennial Conference of the Association for Machine Translation in the Americas (AMTA 2014), pages 166-179.
Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN - CCounter NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2819-2829, Florence, Italy. Association for Computational Linguistics.
Fabio Del Vigna, Andrea Cimino, Felice DellOrletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. 2019. Augmenting neural response generation with context-aware topical attention. In Proceedings of the First Workshop on NLP for Conversational AI, pages 18-31.
Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):85.
Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. EMNLP 2018, page 11.
Njagi Dennis Gitari, Zhang Zuping, Hanyurwimfura Damien, and Jun Long. 2015. A lexicon-based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4):215-230.
Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf. 2019. Large-scale transfer learning for natural language generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6053-6058.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Homa Hosseinmardi, Sabrina Arredondo Mattson, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. 2015. Detection of cyberbullying incidents on the instagram social network. arXiv preprint arXiv:1503.03909.
Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument generation with retrieval, planning, and realization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661-2672.
David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019. A just and comprehensive strategy for using NLP to address online abuse. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3658-3666.
Filip Klubicka and Raquel Fernandez. 2018. Examining a hate speech corpus for hate speech detection and popularity prediction. In LREC.
Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1-11.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016b. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Ruli Manurung, Graeme Ritchie, and Henry Thompson. 2008. An implementation of a flexible author-reviewer model of generation using genetic algorithms. In Proceedings of the 22nd Pacific Asia Conference on Language, Information and Computation, pages 272-281, The University of the Philippines Visayas Cebu College, Cebu City, Philippines. De La Salle University, Manila, Philippines.
Binny Mathew, Navish Kumar, Pawan Goyal, Animesh Mukherjee, et al. 2018. Analyzing the hate and counter speech accounts on twitter. arXiv preprint arXiv:1812.02712.
Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou shalt not hate: Countering online hate speech. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 369-380.
Kevin Munger. 2017. Tweetment effects on the tweeted: Experimentally reducing racist harassment. Political Behavior, 39(3):629-649.
Jon Oberlander and Chris Brew. 2000. Stochastic text generation. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 358(1769):1373-1387.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate
speech. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4757-4766, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Brian Richards. 1987. Type-token ratios: What do they really tell us? Journal of child language, 14(2):201-209.
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv preprint arXiv:1701.08118.
Carla Schieb and Mike Preuss. 2016. Governing hate speech by means of counterspeech on facebook. In 66th ica annual conference, at fukuoka, japan, pages 1-23.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10.
Leandro Araújo Silva, Mainack Mondal, Denzil Correa, Fabricio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In ICWSM, pages 687-690.
Tanya Silverman, Christopher J Stewart, Jonathan Birdwell, and Zahed Amanullah. 2016. The impact of counter-narratives. Institute for Strategic Dialogue, London. https://www.strategicdialogue.org/wp-content/uploads/2016/08/Impact-of-Counter-Narratives_ONLINE.pdf-73.
Lucia Specia and Atefeh Farzindar. 2010. Estimating machine translation post-editing effort with hter. In Proceedings of the Second Joint EM+/CNGL Workshop Bringing MT to the User: Research on Integrating MT in the Translation Industry (JEC 10), pages 33-41.
Rachele Sprugnoli, Stefano Menini, Sara Tonelli, Filippo Oncini, and Enrico Piras. 2018. Creating aWhatsapp dataset to study pre-teen cyberbullying. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 51-59.
Scott R Stroud and William Cox. 2018. The varieties of feminist counterspeech in the misogynistic online world. In Mediating Misogyny, pages 293-310. Springer.
Marco Turchi, Matteo Negri, and Marcello Federico. 2013. Coping with the subjectivity of human judgements in mt quality estimation. In Proceedings of