High-quality argumentative information in low resources approaches improve counter-narrative generation
Authors: Damián Furman
Pablo Torres
José Rodríguez
Diego Letzen
Maria Vanina Martinez
Laura Alonso Alemany
Venue: EMNLP 2023 Findings (2023)
Keywords: counter-narratives
argument mining
finetuning
TL;DR: Counter-narrative generation can be significantly improved with very few, high-quality examples and argumentative information

Abstract: It has been shown that high quality fine-tuning boosts the performance of language models, even if the size of the fine-tuning is small.
In this work we show how highly targeted fine-tuning improves the task of hate speech counter-narrative generation in user-generated text, even for very small sizes of training (1722 counter-narratives for English and 355 for Spanish). Providing a small subset of examples focusing on  single argumentative strategies, together with the argumentative analysis relevant to that strategy, yields counter-narratives that are as satisfactory as providing the whole set of counter-narratives.

We also show that a good base model is required for the fine-tuning to have a positive impact. Indeed, for Spanish, the counter-narratives obtained without fine-tuning are mostly unacceptable, and, while fine-tuning improves their overall quality, the performance still remains quite unsatisfactory.

High-quality argumentative information in low resources approaches
improves counter-narrative generation
Damián Furman Pablo Torres José A. Rodríguez',
Diego Letzen", Vanina Martínez', Laura Alonso Alemany
Departamento de Computación, Universidad de Buenos Aires, Argentina
Consejo Nacional de Investigaciones Cientificas y Técnicas, Argentina
3 Facultad de Filosofía, Universidad Nacional de Córdoba, Argentina
4 Artificial Intelligence Research Institute (IIA--SSI), Barcelona, Spain
Facultad de Matemática, Astronomía y Física, Universidad Nacional de Córdoba, Argentina
Abstract
users that generated it. This strategy has two main disadvantages. The first one is that blocking and delcting may prevent a hate message from spread- ing, but does not counter its consequences on those who were already reached by it. The second one is that it can generate overblocking or censorship.
has been shown that high quality fine-tuning boosts the performance of language models, even if the size of the fine-tuning is small. this work we show how highly targeted fine-tuning improves the task of hate speech counter-narrativy generation in user-generated text, even for very small sizes of training (1722 nounter-narrative. for English and 355 for Spanish). Providing a small subset of examples focusing single argumentative strategies, to- gether with the argumentative analysis relevant that strategy, yields counter-narrative that are satisfactory providing the whole of counter-narration We also show that good base model required for the fine-tuning to have positive impact. Indeed, for Spanish, the counter- narrtttvnarr obtained without fine-tuning are mostly unacceptable, and, while fine-tuning improves their overall quality, the performance still remains quite unsatisfactory.
An alternative blocking that has been gaining attention in the last years, is to "oppose hate content with counter-narrative (i.e. informed textual re- sponses)" (Benesch, 2014; Chung al., 2019) this way, the consequences of errors in the hate clas- sification are minimized, overblocking is avoided, and it helps to spread a message against hate that can reach people that are not necessarily convinced, or even not involved in the conversation.
However, the huge volume of online hate mes- sages makes the manual generation counter- narratives an impossible task. this scenario, au- tomating the generation of counter-narratives is an appealing avenue, but the task poses a great challenge due to the complex linguistic and com- municative patterns involved in argumentation.
1 Introduction
Large Language Models (LLMs) have shown im- pressive capabilities to generate acceptable texts in a number of scenarios. This has allowed to tackle tasks that were considered too difficult until very recently. Counter-narratiee generation in particu- lar is very challenging task because it requires reasoning, argumentation and world knowledge, in addition the ability to produce text that is grammatically correct and pragmatically felicitous. Often, counter-narrative resort to pieces of infor- mation that are only indirectly implied in the text. Counter-narrative generation is arising as a valu- able application to mitigate some forms of violence in social media. Indeed, automatically generated counter-narrative have been proposed as a primary input facilitate the task of NGO specialists to counter hate speech (Tekiroğlu et al., 2020).
Traditional machine learning approaches have typically fallen short address argumentation min- ing and generation with satisfactory results. How- ever, Large Language Models (LLMs) seem capa- ble of generating satisfactory text for many tasks. Even so, for counter- narratve generation, specific training seems to be required to obtain nuanced, effective counter-narrative But it is unclear what specific training for this task is, and which informa- tion would have positive effect on this task.
Our hypothesis that additional information about some argumentative aspects of text will have a positive impact in the quality of the generated counter- narratives.
In this paper, we show that some kinds of argu- mentative information of hate speech do enhance the proportion of satisfactory counter-narratives
The predominant strategy adopted so far to counter hate speech in social media is to recog- nize, block and delete these messages and/or the
Noo Hate Speech Movement Campaign: http://ww.. no- hatespocchmovementtorg

---

that LLM generates. We have focused on the do- main of hate speech in social media, and we show that a small dataset of counter-narratives boosts performance, but even smaller subsets have visi- ble impacts improving performance, if they are enriched with adequate argumentative analysis.
In follow-up work to the seminal CONAN work. Tekiroğlu al. (2020) applied LLMs to assist ex- perts in creating the corpus, with GPT-2 gener- ating a set of counter-narratives for a given hate
speech and experts editing and filtering them. Fan- ton et al. (2021) )) iteratively refined LLM where the automatically generated counter-narrativee were filtered and post-edited by experts and then fed them the LLM further training examples fine-tune it, in a number of iterations. Bonaldi et al. (2022) apply this same approach to obtain a machine- -eeeeratee dataset of dialogues between haters and experts in hate countering. As a fur- ther enhancement in the LLM-based methodology, Chung et al. (2021b) enhanced the LLM assistance with knowledge-based retrieval architecture to enrich counter-narratiee generation.
In experiments for English and Spanish, we show that even with very few examples, is the case for Spanish, highly focused examples and relevant ar- gumentative information do have a positive impact.
The rest of the paper is organized as follows. In the next section, we review relevant work related to automated nounter-narrative generation and argu- mentative analysis of hate speech. Then in Section 3 we describe our dataset of counter-narrative with which we carry out the comparison of sce- narios described in Section 4, where we also de- scribe extensively our approach to the evaluation of generated counter- narrativs,,,,,vvs, based on human judgements, and the prompts used to obtain the cuunter-nrrativess Results analyzed in Section 6 show how fine-tuned LLMs and argumentative in- formation provide better results, which we illustrate with some examples.
Ashida and Komachi (2022) use LLMs for gen- eration with a prompting approach, instead of fine- tuning them with manually created or curated exam- ples. They also propose a methodology to evaluate the generated output, based on human evaluation of some samples. This same approach is applied by Vallecillo Vallecillo-Rodrigeez et al. (2023) to create dataset of counter naurter-narratives for Spanish. Both these ap- proaches are targeted to user-generated text, closely related to social media.
2 Related work
Automated counter-narrativy generation has been recently tackled by leveraging the rapid advances in neural natural language generation. As with most natural language generation tasks in recent years, the basic machine learning approach has been to train or fine-tune a generative neural network with examples specific to the target task.
However, none of the aforementioned datasets or approaches to counter-narrativy generation in- cludes or integrates any additional annotated in- formation apart from the hate message, possibly its context, and its response. That is why we con- sider an alternative approach that aims to reach generalization not by the sheer number of exam- ples, but by providing richer analysis of such examples that guide the model in finding adequate generalizations. We believe that information about the argumentative structure of hate speech may be used as constraints to improve automatic counter- narrative generation.
The CONAN dataset (Chung et al., 2019) is, to our knowledge, the first dataset of hate speech counter-narratives. It has 4078 Hate Speech Counter Narrative original pairs manually written by NGO operators, translated to three languages: English, French and Italian. Data was augmented using automatic paraphrasing and translations be- tween languages to obtain 15024 final pairs of hate speech counter-narrative Unfortunately, this dataset is not representative of social media.
Chung et al. (2021a) address an argumenta- tive aspect of hate speech countering. They clas- sify counter-narrative by type, using a LLM, and showing that knowledge about the type of counter- narratives can be successfully transferred across languages, but they do not use this information generate counter-narratives
Similar approaches were carried out by Qian et al. (2019) and Ziems et al. (2020). Qian et al. (2019)'s dataset consists of reddit and Gab conver- sations where Mechanical Turkers identified hate speech and wrote responses. Ziems al. (2020) did not produce new text, but labeled covid- 19 related tweets as hate, counter-speech or neutral based on their hatefulness towards Asians.
To our knowledge, ASOHMO (Furman et al., 2023) is the only corpus where tweets of hate speech have been annotated with argumentative information, and CONEAS is the only dataset

---

of counter-narrativss based the argumentative structure of hate speech. We describe both in the following Section.
Negate Relation Between Justification And Conclusion (Type A) Negate the implied re- lation between the justification and the con- clusion.
3
Creating counter-narrativer associated to argumentative aspects of hate speech
Negate association between Collective and Property (type B) Attack the relation be- tween the property, action or consequence that is being assigned to the targeted group and the targeted group itself.
3.1 Argumentative annotation of hate speech
In ASOHMO we enriched the argumentative tweets of Hateval (Basile et al., 2019) with manual anal- ysis of their argumentative aspects, following an adaptation of the proposal of Wagemans (2016), analytic approach to represent the semantics of the core schemes proposed by Walton al. (2008), with fewer categories based on limited set of general argument features. The following argumen- tative aspects are manually identified in tweets:
Attack Justification based on it is type (Type C) If the justification a fact, then the fact must be put into question or sources must be asked to prove that fact. it is of type "value", it must be highlighted that the premise is actually opinion, possibly rela- tivizing it as xenophobous opinion. If it is "policy", counter policy must be provided. Free Counter-Narrative (type D) All counter-narrative that the annotator comes up with and do not fall within any of the other three types.
Justifications and Conclusions.
Type of Justification and Conclusion: Fact,
Policy or Value.
A Pivot signalling the argumentative relation between Justification and Premise.
An example of each type of counter-narrativn can be seen Figure The dataset consists total of 1722 counter-narratives for 725 argumentative tweets in English and 355 counter- counter-narratives for 144 tweets in Spanish (an average of 2.38 and 2.47 per tweet respectively). The annotation process and work conditions are detailed in Appendix A, and the complete guidelines write counter-narratives can be seen in Appendix
Two domain-specing components: the Col- lective which is the target of hate, and the Property that is assigned to such Collective.
3.2 Counter-narratives based on argument components
In CONEAS CcounterNaarraiive. Exploiting Argu- mentative Structure)(², each argumentative tweet in ASOHMO is paired with counter-narrative of three different types, defined by applying system- atic transformations over argumentative compo- nents of the tweet, and a fourth type consisting of any counter-narrative that does not fall under any of the other three.
Experimental Settings
We designed series of experiments assess the impact of high-quality examples and argumentative information in the automatic generation of counter- narratives via prompting LLMs. We wanted compare the following configurations:
All nounter-narratives. regardless of their type. also follow the guidelines of the Get The Trolls Out project²: don't be aggressive or abusive, don't 't spread hate yourself, try to de-escalate the conver- sation, respond thinking on a wider audience than the person posting the original tweet and try to build a narrative. Annotators were suggested to try to write least one counter-narrativ of each type but only if they came naturally, otherwise they could leave it blank.
Base or XL version of LLM To assess the im- pact of the size of the base LLM, we com- pared the Base (250M parameters) and XL (3B parameters) versions of Flan-T5 (Chung al., 2021b). Given our limited computa- tional resources, only these two LLMs were compared.
Finetuned Fewshot To assess the adequacy of different training strategies, we compared a fine-tuning approach and a fewshot approach.
The instructions to generate each type of counter- narrative are as follows:
For fine-tuning, different sets of exam- ples were used.

---

HATE TWEET:
user deport all illegal migrants india already reeling under constant threat of muslim radicals curb population
Justification: india already reeling under constant threat of muslim radicals curb population (fact)
Conclusion: must deport all illegal migrants (policy)
Collective: illegal migrants
Property: muslim radicals
COUNTER NARRATIVE A (Negate relation between justification and conclusion) Deporting illegal migrants will not mitigate the problems with muslim radicals.
COUNTER NARRATIVE B (Negate relation between collective and property)
Illegal migrants are not necessarily muslim radicals.
COUNTER NARRATIVE C (Negate justification based on type)
It is not true that India is reeling under threat of muslim radicals.
FREE COUNTER NARRATIVE (Free) Deporting illegal migrants without consideration to their circumstances is an inhumane move.
Figure 1: Examples of each type of counter narratives.
For few-shot approaches, we tried build- ing prompts with two, five and ten ex- amples from the training partition of the dataset, enclosed in special tokens. We found that if the prompt was built with five ten examples, the model always generated the same response for all in- puts, usually a paraphrase of one of the counter-narratives used as example. So we only report experiments using two examples on the prompt (see figure 4).
only counter-marratvvs of type (496 training examples in English, 101 Spanish)
only counter-narratives of type B (238 in English, 59 in Spanish)
only counter-narrativer of type C (467 in English, 97 in Spanish)
For each type-specifie fine-tuning, com- pared fine-tuning with the specific subset of counter-narratives only, or enriching them with argumentative information. For each type of counter-narrative, we provided the kind of argumentative information that we deemed relevant for the kind of counter- narrative, that is: Justification and Conclusion for type A, Collective and Property for type B and Justification for type
With without argumentative information
To assess the impact of argumentative information in the quality of the generated counter-narrativs,, we enriched the finetuning or fewshot examples with different combi- nations of the argumentative information available in ASOHMO:
Collective and Property;
Justification and Conclusion;
Fewshot experiments were conducted for Flan- T5 Base (small) and XI. (larger) models. Fine- tuning was only conducted Flan-T5 Base be- cause our limited computational resources.
All argumentative information men- tioned above plus the Pivot
With specific kinds of narratv-narsative To as- sess the impact of high-quality examples, in the fine-tuning approaches we fine-tuned the base model with different subsets of examples, defined by argumentative criteria, namely, the kind of counter-narrative.
We conducted small prospective evaluation find optimal parameters for generation, and we found that using Beam Search with 5 beams yielded the best results, so this the configuration we used throughout the paper. The details can be seen Appendix G.
All counter-narrann

---

4.1
Fine-tuning of the LLM with
ness, Stance (towards the original tweet) and Infor- mativeness (same as Chung et al. (2021b)). Based on these previous works, we manually evaluate the adequacy of counter-narratives four different aspects (further detail is provided in Appendix D):
counter-narrative
To finetune FLAN-TS with our dataset of counter- narratives, we randomly split our dataset in training, development and test partitions, assuring that all counter-narratives for the same hate tweet are con- tained into the same partition. Details can be seen on Table 1. All models were trained starting from in a multilingual setting using mixed English and Spanish examples, with a learning rate of 2c-05 for 8 epochs.
Offensiveness: the tweet is offensive.
Stance: if the tweet supports or counters the specific message of the hate tweet.
Flan-
Informativeness: Evaluates the complexity and specificity of the generated text.
Felicity: Evaluates if the generated text sounds, by itself, fluent and correct.
5 Evaluation method for generated counter-narratives
Each aspect constitutes a Likert-scale with three possible values, ranging from I to 3, where lower values represent the least desirable categories and vice versa. Informativeness is only evaluated if Stance was assigned the higher value possible. Oth- erwise, it is considered as having a value.
Evaluation of counter-narratives is not straightfor- ward. So far, no automatic technique has been found satisfactory for this specific purpose. As co- gently argued by Reiter and Belz (2009), superficial n-gram-based metrics like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) fail to capture impor- tant aspects of generated texts, because they rely strongly words n-grams overlap with manu- ally generated examples and they are inadequate in scenarios where there can be many possible good outputs of the model, with significant differences between themselves, such as our case.
5.1 Prospective evaluation of generated counter-narratives
To assess the quality of counter-narratives gener- ated with each approach (with different kinds of argumentative information and LLM settings), we manually evaluated 360 pairs of hate tweet/counter- narratives in English and 180 in Spanish, following the guidelines described in Appendix D. Evalua- tions are available to the scientific community*.
Specifically for argumentative texts, Hinton and Wagemans (2023) show that LLMs produce low quality arguments that are nevertheless well- formed utterances. A specific, more nuanced eval- uation seems be necessary assess relevant features of generated counter-sarrativss.
We evaluated three random subsets of 20 hate tweets in English and 10 in Spanish. One of the sub- sets contains only tweets associated with counter- narratives of both types A and C on our dataset, and was used to evaluate models finetuned only with these kinds of counter-narratives. Another contains only tweets associated with conter-narrativve of type B and was also used evaluate models fine- tuned only with this type of counter counter-narratives. The last subset contains tweets with counter-narrativ pairs of all types, and with this we evaluated the rest of experiments.
Despite the limitations of n-gram-overlas met- rics, they are still widely used to evaluate counter- speech (Gupta et al., 2023). But, faced with the shortcomings of these metrics, some authors have conducted manual evaluations for automatically generated counter-narrativer in line with the meth- ods of the Natural Language generation community (Shimorina and Belz, 2022). Manual evaluations typically distinguish different aspects of the ad- equacy of given text as a counter-narrative for another. Chung et al. (2021b) evaluate three as- pects of the adequacy of nounter-narrative:: Suit- ableness (if the counter-narrative was suited as response to the original hate message), Informative- ness (how specific or generic is the response) and Intra-coherence (internal coherence of the counter- narrative regardless of the message it is respond- ing to). Ashida and Komachi (2022), on the other hand, assess these three other aspects: Offenssve-
For each tweet in the corresponding evaluation subset and for each combination of features be assessed: fewshot, finetuned, with different kinds of argumentative information and with different sizes of LLM,, we generated one counter-narrative. Then, three annotators labeled each pair tweet counter-narrative according to the four categories described above. Details of the manual evaluation process can be found in Appendix B.

---

English
Spanish
Tweets
CNs
% corpus
B
C
Tweets
CNs
% corpus
A
B
C
Train
509
1201
69.8%
496
238
467
105
257
72.4%
101
59
97
Dev
173
10.0%
38
68
12
27
7.6%
12
Test
145
348
20.2%
138
74
136
27
71
20%
27
21
23
Proportion of tweets with counter-narrative
96%
47%
90%
97%
61 %
89%
Table 1: Size of dataset partitions of English and Spanish datasets. Columns A. B and C show the amount of counter-narrative used for each partition when training only with counter-narrativer a given type.
5.2 A note on the reproducibility of our manual evaluation
categories that can capture distinct, relevant aspects of argumentation. We believe that this will provide more adequate method of evaluation that will enhance the reproduciblity of the evaluation.
As can be expected for such an interpretative task, agreement between judges far from robust. Thus, the results of this prospective evaluation can only be considered indicative, a first, coarse assessment of the quality of generated counter-narratives. But even with these limitations, the lack of better alter- native renders this initial assessment very valuable determine the impact of different factors, spe- cially argumentative information, in the quality of generated cuntter-rrgumntts.
5.3 Aggregated quality score of generated counter-narratives
Given that there were three judgements for each pair hate tweet - counter-narratiin the final value for each category was determined by majority vote, that is, the value assigned by at least two annotators. When there was majority vote for given cate- gory, which happened only in 10 cases, we adopted conservative criterion and assigned the worst of the three possible values.
Table breaks down the agreement scores be- tween the three annotators, calculated using Co- hen's Kappa (Cohen, 1960).
The final score S for each generated counter- narrative calculated by summing the values of the four categorics manual evaluation. Since the smallest possible value is and the highest value is 12, we subtracted 2, so the final score S ranges between and 10, as follows:
I vs 2
vs 3
vs
Offensiveness
0.47
0.40
0.41
Stance 0.63
0.58
0.63
Informativeness
0.49
0.42
0.54
Felicity 0.67
0.37
0.36
Table Agreement scores between annotators 1, and 3 using Cohen's Kappa.
1 Σ Off + Stan, + Felic, Inf 2 (1) n
S
In most cases, agreement ranges from Moder- ate to Substantial (Landis and Koch, 1977). Over- all, the category with best agreement is Stance, which seems to be recognizable with more reliabil- ity across judges. If we factor out the anomalous judgements of Judge 3, who seems to be having a different understanding of the category, Felicity also seems to be reliably recognized. However, Of- fensiveness and Informativeness reach lower levels of agreement. Judges do not seem to manage common concept to assess those categories.
where is the amount of evaluated counter- narratives.
Though didn't 't apply explicit bias when calculating the score, Stance is implicitly more de- terminant than other categories, since any value lower than for Stance will set the value of Infor- mativeness to O automatically.
To facilitate the interpretation of the quality of each approach, we provide another aggregate met- ric, with discrete values Excellent, Good, and None. Good narrativs those with op- timal values for Offensiveness, Stance and Felic- ity. Excellent counter-narrative are those that also have the optimal value for Informativeness. val- ues do not meet the criteria for Good, then the counter-narrative narrative is considered None.
The insights obtained from this first approach will be used improve the evaluation method, as in Teruel al. (2018), aiming for more reproducible evaluations. particular, we will break down the Informativeness category into more fine- -raiied

---

English
Spanish
Score
% Good
% Excellent Approaches
Score
% Good
% Excellent
Fewshot
Base
5,70
40%
0%
2.10
0%
0%
Base All
4.55
25%
5%
2.20
0%
0%
Base Collective
5.65
20%
5%
2.20
0%
0%
Base Premises
5.30
35%
5%
1.90
0%
0%
XL
4.05
25%
0%
2.80
10%
0%
XL All
2.85
10%
5%
2.60
0%
0%
XL Collective
3.70
15%
5%
2.30
0%
0%
XI. Premises
2.90
0%
0%
2.40
0%
0%
Finetuned A Approaches
Base
7.15
60%
30%
6.60
10%
0%
Base All
5.70
25%
10%
7.40
30%
0%
Base Collective
7.40
50%
25%
6.80
30%
0%
Base Premises
6.55
40%
30%
6,40
20%
0%
Base CNs A
5,75
30%
25%
4.80
0%
0%
Base CNs A Premises
6.55
40%
40%
3.50
0%
0%
Base CNs
3.05
0%
0%
2.50
0%
0%
Base CNs B Collective
3.95
10%
10%
3.20
0%
0%
Base CNs C
4.65
10%
0%
5.00
20%
0%
Base CNs C Premises 5.60
30%
20%
3.90
0%
0%
Table 3: Manual evaluation of automatically generated counter-narratives for English and Spanish hate tweets, using different sizes of the model (Base and XL). two learning techniques (fewshot and finetuning). two different training settings (all narrttr-nrrraiivs or only one kind: A, or C) and different combinations argumentative information (no information, Collective and Property, Premises and pivot and all the information available). The score and the proportion of Good and Excellent counter- narratives were calculated by summing the Offensiveness, Stance. Informativeness and Felicity developed in Section D. The detailed score for each category can be seen in Table 4.
assigned by human judges for each manually evaluated counter-narrative
6 Analysis of results
Given the poor performance of the XL model in general, and given our limited computational resources, fine-tuning was carried out only for the Base model.
Results of the manual evaluation of different strate- gies for Dounter-narrative generation for English and Spanish can be seen in Table summary of this table can be seen in Figure 2, which displays the aggregated proportion of Good and Excellent counter-narratives for each strategy.
As be in the lower half of Table fine- tuning has very positive effect in performance. A very valuable conclusion that can obtained from these results is that, for models that perform well enough, a small number of high quality ex- amples produce much bigger improvement in performance than using larger models, which are also more taxing and even inaccessible for some. The results obtained for Spanish seem to indicate that models require minimal performance be improved by additional examples. The question re- mains, how big the model and the fine-tuning need to be to be able to benefit from the information examples.
In the first place, we can see overall that the re- sults for Spanish are much worse than for English, specially when it comes to using the LLM without fine-tuning, only with few-shot approach. Us- ing larger model (XL) has slightly beneficial effect, but in any case the proportions of Good and Excellent counternarratives obtained by fewshot approach for Spanish are almost always null.
For English, we see that the reverse holds: the smaller model (Base) obtains better results for the fewshot approach than the bigger model. Pro- viding argumentative information in this setting seems to negatively affect performance.
For English, the left part of Table 3, we can see that the model with the biggest proportion well- valued counter counter-narrative narratives (60%), Base, does not

---

Ease F ewihho
Base Fowsho!
Booe Fewesol Collecivve
Fawshol
XL Fowishol
XL, Fawssot Collectvve
XL Fewehot Premises
Bare Finctuned
Finetured Colledive
Fineluned Premises
Base Finstuned Chis A
Base Finetuned CNs Premises
Base Finstuned Chs
Base Firetuned B Collcl
Base Fineluned JusiP
50%
100%
Figure 2: Proportion of Excellent (light gray) and Good (light + dark gray) counter-narratives produced by each approach.
rely on any kind of argumentative information, but some kinds of argumentative information produce comparable results. For example, including infor- mation about the Collective targeted by the hate speech yields 50% well-valued counter-narrativer and higher overall score,
(negation of Justification), with almost as many ex- amples as in type A narratives, the smaller ratio of improvement may be attributed to more complex argumentative structures.
In any case, providing the relevant argumentative information, Collective for type and Premises for type C, always produces an improvement. In fact, even when finetuning with all types counter- narratives, providing information about the Collec- tive and Premises yields a improvement higher than providing all available argumentative information, but not good not providing any argumentative information at all.
Moreover, in qualitative inspection we have ob- served that counter- counter-narratives produced by provid- ing argumentative information tend to be more in- formative, as can be seen in Figure 3. However, our current evaluation method is not fine-grained enough to reflect these subtle differences in quality. Future developments of the evaluation method will aim to capture such differences.
For Spanish, the performance of the model with- out specific training is much worse than for English, even with fine-tuning, producing almost no Good and Excellent nounter-narrative. However, when the model is fine-tuned with counter-narratives. even if the number of training examples is very small (257 in total), the improvement in the perfor- mance of the model is notable, even much higher than for the case of English. Indeed, while English goes from score 5.70 for the base model with few shot to a score of 7.15 for fine-tuned, Span- ish goes from 2.1 to 6.6, obtaining much bigger improvement and coming much closer to the perfor- mance of the model for English. The improvement is much more notable when argumentative infor- mation provided, even reaching the score for the best approaches for English (7.4) when all argu- mentative information is provided. However, the proportion Good narratives remains low, and Excellent is nonexistent.
When we fine-tune the model only with counter- narratives of given type (lower quarter of Ta- ble 3), the number of examples for fine-tuning is much smaller, but performance is not proportion- ally affected. Although the proportion of counter- narratives that are valued as Good is lower, the proportion of Excellent increases, especially for counter-narrative of type A (negation of the re- lation between Justification and Conclusion) for which information about the Justification and Con- clusion is provided, even though the training ex- amples are less than half (1201 training examples for the whole corpus vs. 496 examples of counter- narrative A).
For counter-narrativee of type B and C. how- ever, improvements are as good, For counter- narratives of type B, the small number of examples may be the reason for such small improvement in performance. For counter counter-narrative narratives of type C

---

Tweet with argumentative information:
Tweet without argumentative information:
street interview whit italians "send all migrants back where they came from they block streets to pray free speech time :link: @ @user Jus- tification: street interview whit italians "send all migrants back to where they came from they block streets to pray (fact) Conclusion: "send all migrants back to where they came from they block streets to pray (policy) Pivot: migrants they they
street interview whit italians "send all migrants back to where they came from they block streets to pray free speech time :link: @user
Counter-narrative:
Counter-narrative
I don't think it's good idea send all migrants back to where they came from.
don't think it's the right thing to do.
Figure Counter-narratives narratives obtained for the same tweet with different strategies: including argumentative information (left) and without argumentative information (right).
When fine-tuned with types of counter-narrative separately, improvements are smaller, probably due to the very small number of training examples. carried out is
than for English, is encouraging that argumenta- tive information has much higher positive impact than for a scenario with bigger models and more training examples, and will continue to annotate examples for Spanish to improve the generation of counter-narratives.
In any case, the evaluation small, mostly indicative, and these tendencies will be further explored future work.
We will also explore other aspects of the quality of counter-narren narratives,, with a more insightful, more extensive human evaluation. We will also explore the interaction between argumentative information and other aspects, like lexic, level of formality, and culture.
7 Conclusions and future work
We have presented an approach to generate counter- narratives against hate speech in social media by prompting large language models with information about some argumentative aspects of the original hate speech.
A Acknowledgments
We have shown that finetuning a smaller model with a small corpus of high-quality examples of pairs hate speech- counter-narrative yields a big- ger improvement in performance than using larger language models.
This work was supported by funds provided by Agencia Nacional de Promoción Científica y Tec- nológica under grants PICT-2018-0475 (PRH- 2014-0007) and CONICET under grant PIP 1120200101 408CO. The authors also acknowledge support by the Span- ish project PID2022-139835NBB MCIN/AEI/1O. MCIN/AAE//O0.1303//500 13039/55 10. 13303/5500 10001 1033 and by "European Union Nex(GenerationEUUPPRRT"" The development the CONEAS corpus of counternarratives was sup- ported by NAACL as part of the ContraHate chal- lenges. This work used computational resources from CCAD UNC htttss:/ccad.unc.eeee which part of SNCAD MinCyT, Argentina.
and
We have also shown that some kinds of argu- mentative information do have positive impact in generating more specific, more informative counter- narratives. particular, we have found that the types counter-narrative that negate the relation between the Justification and the Conclusion and that negate the Justification have an improvement in performance if argumentative information about the Justification and the Conclusion is provided. Moreover, have also found that argumenta- tive information makes a positive impact in scenar- ios with very few tweets, as shown by our exper- iments for Spanish. Although the quality of the counter-narratin generated for Spanish is lower
funded by 10001 1033, MCIN/AET/

---

Limitations and Ethical Considerations
Finally, have not specifically conducted a study on biases within the corpus, the annotation or the automatic procedures inferred from it, nor on the LLMs that have been applied. We warn researchers using these tools and resources that they may find unchecked biases, and encourage further research in characterizing them.
In the first place, would like to make it clear that the counter-narratives created for this study the result of the subjectivity of the annotators, Al- though they have been instructed through manual and training sessions, counter-narrativ generation is an open-ended task, subject to great variability between persons, Even if we have taken care to avoid it, it can still be the case that the cultural and personal biases of the people generating the may produce some texts that re- sult offensive for some of the collectives involved, or that do not share values with some parts soci- ety.
References
Mana Ashida and Mamoru Komachi. 2022. Towards automatic generation of messages countering online hate speech and microaggressions. In Proceedings of Sixth Workshop on Online Abuse and Harms (WOAH).
Also, it is important to note that the automatic procedures obtained are prone to error, and should not be used blindly, but critically, with attention to possible mistakes and how they may affect users, groups and society.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detec- tion of hate speech against immigrants and women in Twitter. In Proceedings of 13th International Workshop Semantic Evaluation.
Then, it also important to note that the corpus used for this research is very small, specially in the Spanish part, so the results presented in this paper need to be considered indicative. A bigger sample should be obtained and annotated to obtain more statistically significant results.
Susan Benesch. 2014. Countering dangerous speech: New ideas for genocide prevention. United States Holocaust Memorial Museum.
Helena Bonaldi, Sara Dellantonio, Serra Sinem Tekiroglu, and Marco Guerini. 2022. Human- machine collaboration approaches to build dialogue dataset for hate speech countering. In EMNLP.
The findings of this research can potentially in- form the development and improvement of lan- guage models and chatbot systems. However, we emphasize the importance of responsible use and application of our findings. It is essential to en- sure that the identified argumentative components are utilized in manner that promotes reasoned usage and does not contribute to the spread of hate speech or harmful rhetoric. We encourage researchers and developers to consider the ethical implications and societal impact of incorporating argumentative analysis into their systems.
Yi-Ling Chung, Marco Guerini, and Rodrigo Agerri. 202 Multilingual counter narrative type classi- fication. In Proceedings of the 8th Workshop on Argument Mining, pages 125-132, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN COunter NArratives through nichesourcing: a mul- tilingual dataset of responses to fight online hate speech. In ACL.
The data have been adequately anonymized by the original creators of the Hateval corpus.
Yi-Ling Chung. Serra Sinem Tekiroğlu, and Marco Guerini. 2021b. Towards mooleede-groondder counter narrative generation for hate speech. In Findings of the ACL-UICNLP 2021.
Studying hate speech involves analyzing and pro- cessing content that may be offensive, harmful, or otherwise objectionable. We acknowledge the po- tential impact of working with such content and have taken steps to ensure the well-being of the research team involved. We have provided compre- hensive guidelines and training to annotators to mitigate any potential emotional distress or harm that may arise from exposure to hate speech. Ad- ditionally, we have implemented strict measures to prevent the dissemination or further propagation of hate speech during the research process.
Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37.
Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroğlu, and Marco Guerini. 2021. Human-in- the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech. In ACK.
Damián Furman, Pablo Torres, José Rodríguez, Diego Letzen, Vanina Martínez, and Laura Alonso Ale- many. 2023. Which argumentative aspects of hate

---

speech in social media can be reliably identified? In Proceedings of Fourth International Workshop on Designing Meaning Representations, co-located with IWCS 2023.
M.E. Dallecillo-Rodriguer Montejo Ráez, and M.T. Martín Valdivia. 2023. Automatic counter-narrativy generation for hate speech in spanish. Procesamiento del Lenguaje Natural, 71.
Rishabh Gupta, Shaily Desai, Manvi Goel, Anil Bandhakavi, Tanmoy Chakraborty, and Md. Shad
Jean H. M. Wagemans. 2016. Constructing peri- odic table of arguments. In Proceedings of 11th International Conference of the Ontario Society for the Study Argumentation.
Akhtar. 2023. Counterspeeches up my sleeve! in- tent distribution learning and persistent fusion for intent-conditioned counterspeech generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume
Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. CUP.
1: Long Papers), pages 5792-5809, Toronto, Canada. Association for Computational Linguistics.
Caleb Ziems, Bing He, Sandeep Soni, and Srijan Kumar. 2020. Racism is virus: Anti-asian hate and counter- hate in social media during covid-19 19 crisis.
Martin Hinton and Jean H. M. Wagemans. 2023. How persuasive is ai-generated argumentation? anal- ysis of the quality of an argumentative text pro- duced by the gpt-3 ai text generator. Argument and Computation, 1451):59-74
J. Richard Landis and Gary G. Koch. 1977. The mea- surement observer agreement for categorical data. Biometrics, 33(1):159-174
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In ACL.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth M. Beld- ing, and William Yang Wang. 2019. benchmark dataset for learning to intervene online hate speech. CoRR, abs/ htt/1999.042511
Ehud Reiter and Anja Belz. 2009. An investiga-
tion into the validity of metrics for automati- cally evaluating natural language generation systems. Computational Linguistics, 35(4):529-533
Anastasia Shimorina and Anya Belz. 2022. The hu- evaluation datasheet: A template for record- ing details of human evaluation experiments NIP. In Proceedings of the 2nd Workshop Human Evaluation of NLP Systems (HumEval), pages 54 75, Dublin, Ireland, Association for Computational Linguistics.
Serra Sinem Tekiroğlu, Yi-Ling Chung, and Marco Guerini. 2020. Generating counter uarratives against online hate speech: Data and strategies. ACI..
Milagro Teruel, Cristian Cardellino, Fernando Cardellino, Laura Alonso Alemany, and Serena Villata. 2018. Increasing argument annotation reproducibility by using inter-annotator agreement to improve guidelines. In Proceedings of the Eleventh International Conference Language Resources Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

---

A Details of the annotation process for the CONEAS corpus
of the authors of the paper and a PhD in an unre- lated field without previous acquaintance with the task. The two authors were unpaid, the unrelated judge was paid 25USD/hour. In total, each annota- tor labeled 540 hate wwee//counter-narratin pairs. A total of 5 hours was devoted to this evaluation. summing up all 3 evaluators.
Manual generation of CONEAS counter-narratives was conducted by two annotators, a Computer Sci- ence PhD student and an undergraduate Philosophy student. To begin with, annotation guidelines were created by the research team, including annotators, describing how to use the argumentative analysis of hate tweets to write nounter-narrative. using dif- in Appendix
C Guidelines to create counter-narrative
ferent strategies (A, B and C), C. Then each annotator wrote counter- narrter-narrative of each type for the same subsample of 20 tweets in English, then compared their annotations and discussed how to consolidate the counter-narrative writing process to improve:
The following are the instructions for annotators to create counter-narratio for hate tweets against immigrants within the HatEval corpus (Basile et al., 2019), enriched with argumentative annotations in ASOHMO (Furman et al., 2023):
Three types of counter narratives must be writ- ten, based on the argumentative annotation of ASOHMO. Each type of counter narrative will propose a specific hint for the annotator to follow but they must be written to sound like a natural spontaneous response on Twitter. All three types of counter-narratives must be written unless it is not possible write something coherent using the given hint.
Adequacy of the narrate- as ac- ceptable, tailored reply for a given hate tweet; Adequacy to the specific type of counter- narrative;
Robustness of the written narrative.
This process is challenging since the task is in- herently subjective and variability between annota- tors does not necessarily imply that one of them is wrong, but that guidelines need to cover such vari- ability, without being too lax too vague. Then, 50 new tweets were annotated, and difficult issues were discussed to achieve better alignment be- tween annotators, but without a need to update the guidelines. After that, they wrote counter- narratives for all the rest of the dataset. After the whole annotation, third annotator reviewed 50 random pairs of hate tweet/counter-narrate using the guidelines as reference, without finding major inadequacies.
It might be possible that two counter-narrative written following different hints end up being very similar. This is not a concern and in any case, the annotator will try whenever it is possible to emphasize different aspects of the counter narrative to make them be the least similar possible.
Counter narratives should be written following the guidelines of the Get The Trolls Out project: don't 't be aggressive or abusive, don't 't spread hate yourself. try to de-escalate the conversation, re- spond thinking on a wider audience than the person posting the original tweet and try to build a nar- rative (see your answers long term, repetitive process).
The first annotator labeled 20% of the total dataset and was not paid specifically for this task, as it is part of his own reserach project. The sec- ond annotator labeled the other 80% and was paid 25USD/hour for 80 hours of work, partially funded by NAACL part of the organization of the Con- traHate challenge
If you think that there is strong counter- narrative that doesn't it fit any of the three proposed types, add it as a fourth type (CounterNarrativee))
C.1 Negate relation between justification and conclusion (CounterNarraaiveeA)
The basic idea for this type of counter narrative is to negate the relation between the conclusion and the justification, therefore negating the core of the user's reasoning. In most cases the reason given by the user is forced fallacious and there is a direct counter-narrative. However, some cases it's necessary to give an indirect response or to concentrate on particular aspect of the reasoning
B Details of the manual evaluation process
For the manual evaluation of the hate tweet counter-narratives pairs (360 pairs in English and 180 in Spanish), there were three annotators: two

---

that is more vulnerable an attack.
D Guidelines for manual evaluation of counter-narratives
Example:
We manually evaluate? the adequacy of counter- narratives in four different aspects:
We should deport all illegal immigrants because they are breaking the law and the law says they get deported.
Offensiveness: if the tweet is offensive either the larget group, the author of the tweet or any other group or person. Possible values are: Offensive; Possibly Offensive/Not clear; Not offensive.
In this case, directly negating the relation be- tween conclusion and justification would result in something like "just because the law says some- thing doesn't mean you have to do it" but it's not very convincing narrative. If, on the other hand, the perspective is changed, the counter-narrative could state that "Just because the law says something, doesn't mean it's right" or "Laws can change they don't i adjust to the reality of innocent people suffering. What should prevail is our humanity and our will to help the needed and not our will to get rid of them".
Stance: if the tweet supports or counters the specific message of the hate tweet. Possi- ble values are: Supports the original mes- sage; Not clear/Changes subject wrt original tweet; Counters the original message. Stance incorporates certain notion of suitableness, since it assigns value "Changes the subject" if the counter-narrative is responding specif- ically to the standpoint of the original tweet. Informativeness: Evaluates the complexity and specificity of the generated text. Only Nounter-narrativve with "Counters" Stance are evaluated. Possible values are:
C.2 Negate relation between collective and properties associated to them (CounterNarrativee)
1. Generic statement:
replies that don't in- corporate any information mentioned on the tweet and could counter many differ- ent hate messages (e.g "I don't 't think so" "That is not true").
The core of this type of counter-narrative is to negate the relationship between the collective be- ing attacked and the properties, actions or conse- quences related to them. If no collective property were annotated, you can skip this counter-narrative.
2. Specific but not argumentative: the ply is simple statement, possibly com- posed of single sentence without pro- viding justification for the stance but ferring to some specific aspect of the original tweet. Usually they comply with formula composed of a prefix (like "I don't think that" or "Do you have proof that") and verbatim copy of some part of the hate tweet.
C.3
Attack the justification of the claim (ComntrNaarrativeC)
The third type of counter-narration narrative consists of refut- ing the justification of the argument. This refuting must be done considering the type of the justifica- tion premise. If it is of type "fact", then the fact must be put into question sources must be asked prove that fact. If is of type "value", the char- acteristic of the premise of being an opinion must be highlighted and the speaker's opinion itself must be relativized as a xenophobous opinion. If it is a "policy" a counter or opposite policy must be answered.
3. Specific and Argumentative: counter- narratives with some degree of elabora- tion of the information contained the hate message. We identified three com- mon patterns that we associate with this value:
A replies that take more than one ele- ment from the original message and establish some relation between them (c.g. "I don't see the relation between
C.4 Free Counter-narrative (CounterNarrative))
If and only if there is a strong counter-narrative that doesn't on any of the types proposed, the annotator can add it as free caunter-narrative type.
'Results of the evaluation can be found on https://shorturlataacrt

---

(element from the original message) and {other element from the original message)"
"Generate non-hateful response for the following hate messages:
SHS>unbelievable census bureau: 44 6% in ca dont hashtag speakenglish at
B A simple statement declaring stance over single element from the origi- nal tweet but adding second coordi- nated statement with personal appre- ciations about it (e.g. don't 't think we should {some policy mentioned on the Iweet). It is bad idea").
home 35 6% 35 6% in tx 34 5% in nm
31 7% in nj 31 0% in ny our schools
getting 150% funding for every student who doesnt speak hashtag english well parents job hashtag stop immigration https://t co/vi6xxhiill<<HS> And would ending immigration improve English? I don't think so... <EON>
<SCN>
C An argumentative reply that neces- sarily inferred some information not mentioned explicitly the original tweet showing comprehensive derstanding of the meaning of the hate message (e.g. reply to tweet concluding with #BuildTheWal say- ing "Building a wall would cost more to the taxpayers or "Building a wall won't give you more control over il- legal trafficking").
<SHS>100 bangladeshi nationals apprehended near texas border in
3 weeks via @user hashtag noamnesty
hashtag deport illegals hashtag buildthatwall hashtag trump hashtag maga hashtag masa<EHS> => <SCN>Do you know why where they apprehended? Why do you assume they were illegals? <ECN>
Felicity: This category is related to Chung et al. (2021b)'s Intra-Coherenee, but also con- sidering additional dimensions like syntactical and semantic correctness. evaluates inde- pendently of the original tweet, if the gener- ated text sound, by itself, fluent and correct. There are three possible values: The text is incoherent or semantic or syntactically incor- rect; The text is coherent with small errors like in coordination of genre/tense/etc. repeat- ing parts of the original text without adapting them to the text being generated; The text is fluent and sounds correct.
<SHS> " Tweet + "<EHS> <SCN>"
Figure 4: Prompt used for English fewshot counter- narratives generation.
argumentative information. We then generated 10 counter-narrative for random tweets in En- glish that were not among the 20 used for the main evaluation using four decoding strategies: Greedy Search without sampling, with sampling using a temperature value of 0.7, with sampling using Top P with a p value of 0.92 and with Beam Search using 5 beams. Results of this evaluation are shown in Table 5. Results show that Beam Search outperforms the rest of the decoding strate- gies explored, so we used this generation method for all experiments in this work. manual exami- nation of the examples generated shows that beam search produces less diverse results, repeating some words or phrases between replies to different hate tweets. Sampling techniques produce more diverse responses, but also more prone generate in- coherent phrases.
E Breaking down the score calculation
Table shows how the score assigned to each batch HS-CN pairs corresponding to each experimen- tal settings is broken down into the four categories used for the manual evaluation: Offensiveness, Stance, Informativeness and Felicity.
F Prompt for fewshot approaches
In Figure 4 an example prompt for the fewshot approach can be seen.
G Generation settings
We conducted an independent experiment to find the best performant decoding method for the gener- ation of narattvvs.. We finetuned a FLAN- T5 Base model with ASOHMO without using the

---

English
Spanish
Off
Stance
Infor
Fel
Score
Off
Stance
Infor
Fel
Score
Fewshot
Approaches
Base
2.50
2.25
0.40
2.55
5.70
1.30
1.30
0.00
1.50
2.10
Base All
1.95
1.80
0.45
2.35
4.55
1.30
1.20
0.00
1.70
2.20
Base Collective
2.45
2.05
0.30
2.85
5.65
1.20
1.20
0.00
1.80
2.20
Base Premises
2.05
2.05
0.55
2.65
5.30
1.00
1.00
0.00
1.90
1.90
XL
1.65
1.65
0.40
2.35
4.05
1,40
1.40
0.10
1.90
2.80
XL All
1.30
1.30
0.15
2.10
2.85
1,40
1.30
0.00
1.90
2.60
XL Collective
1.70
1.60
0.25
2.15
3.70
1.10
1.10
0.00
2.10
2.30
XL Premises
1.55
1.40
0.00
1.95
2.90
1.30
1.20
0.00
1.90
2.40
Finetuned Approaches
Base
2.55
2.55
1.40
2.65
7.15
2.60
2.60
1.70
1.70
6.60
Base All
2.30
2.15
0.60
2.65
5.70
2.90
2.80
1.60
2.10
7.40
Base Collective
2.55
2.60
1.45
2.80
7.40
2.70
2.70
1.40
2.00
6.80
Base Premises
2.30
2.35
1.20
2.70
6.55
2.60
2.70
1.40
1.70
6.40
Base CNs A
2.35
2.25
0.90
2.25
5.75
2.50
2.30
0.60
1.40
4.80
Base CNs A Premises
2.50
2.35
1.30
2.40
6.55
2.20
2.00
0.20
1.10
3.5
Base CNs B
1.90
1.80
0.05
1.30
3.05
1.70
1.80
0.00
1.00
2.50
Base CNs В Collective
2.15
2.15
0.35
1.30
3.95
2.00
2.00
0.20
1.00
3.20
Base CNs C
2.50
2.20
0.35
1.60
4.65
3.00
2.30
0.3
1.40
5.00
Base CNs C Premises
2.20
2.25
0.85
2.30
5.60
2.20
2.10
0.30
1.30
3.90
Table Score assigned to counter-narratives generated by each experimental setting, broken down by each of the four categories used for evaluation: Offensiveness. Stance, Informativeness and Felicity. Offensiveness, Stance and Felicity range between and 3, while Informativeness ranges hetween O and 3. The Score is the sum of the four previous categories but subtracting 2, so it ranges between 1 and 10.
Ofensiveness
Stance 2
Informativeness
Felicity
Good
Excellent
3
Greedy
10%
20%
70%
0%
40%
60%
10%
20%
30%
10%
40 %
50%
30%
10%
Greedy w/ w/Temp
20%
30%
50%
20%
60%
20%
0%
10%
10%
40%
10 %
50%
10%
10%
Greedy w/Top P
80%
0%
20%
40%
20%
40%
10%
0%
30%
40%
0 %
60%
20%
10%
Beam P
10%
0%
90%
10%
0%
90%
0%
30%
60%
0%
30 % 70% %
70%
50%
Table Results for manual evaluation of different decoding strategies.
Score
% Good
% Excellent
Greedy
7.00
30%
10%
Greedy w/Temp
4.60
20%
10%
Greedy w/Top p
4.90
20%
10%
Beam Search
8.70
70%
50 %
Table 6: Scores for different generation strategies using finetuned Flan T5-Base model.


Reviews:
[{"review_id": "zCkoEaYK9Q", "reviewer": "Reviewer_4FEa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[this is a general comment for all three reviewers] We appreciate the effort made by the reviewers to provide insightful feedback on our work. We understand that it does not exactly conform to the standard experimental paper with standard datasets and standard metrics. We made the decision to address questions that are not straightforward incremental on previous work, and we understand this requires an extra effort from reviewers to identify strong and weak points. We are very thankful for this effort and the very valuable suggestions that they provide. We will be sure to incorporate as much as we can of their work in the final version of this paper.\n\nConcerning the fact that one cannot draw very strong conclusions from the experiments presented here, we agree with the reviewer that the experiments are small and the agreement figures leave room for improvement. As we explain below, we have been working to improve that. However, we believe that these results are worth sharing with the community as they can encourage more research in injecting high-quality information into large language models. Also, they are an illustrating case of a successful combination of symbolic and subsymbolic approaches, moreover, in a task that has been very challenging for long. We will make an effort to put more emphasis on these conclusions in the final version of the paper.\n\nWe are very grateful for the insights from the NLG community with respect to evaluation! Indeed, we were only distantly aware of work in that area, and for the evaluation provided in this work we mainly focused on proposals from argumentation evaluation. We find that evaluation of open-ended tasks is mainly carried out with deeply inadequate, n-gram based methods, which obscures more insightful approaches. We found in ACL 2023 a paper on counter-speech generation [1] that is based solely on such n-gram- or embedding-based automatic metrics of evaluation, as those criticized in [2] (thanks for the pointer!).\n\nSince we got the reviews, we have been skimming through the work of Anja Belz and collaborators and Howcroft and collaborators, and indeed they have provided concepts and operationalizations that are very useful for a second version of our evaluation protocol. Indeed, the evaluation protocol presented in this paper was our first try for the manual evaluation of counter-narratives. We think this protocol enabled a first glimpse on the quality of the obtained counter-narratives, and it is very valuable for us, and hopefully for the community, to show that high-quality argumentative information has a positive impact in improving automatically generated counter-narratives. As such, we believe this is a valuable result to be shared with the community. However, the agreement figures were worrying.\n\nSince we obtained the agreement figures, we have been working on how to improve them, and have come to the same conclusion as the reviewer: that we need to zoom into the dimension we dubbed “Informativeness” to make room for the acceptability of arguments and better capture the persuasiveness of the argument, as developed in the recent analysis by Wagemans and Hinton [3]. We have just finished creating a second, improved version of the guidelines for manual evaluation of counternarratives where Informativeness will be replaced by four dimensions, each representing one of the Gricean maxims: Quantity, Quality, Manner and Relevance. We expect that this granularity will allow for a more accurate evaluation rubric, which will hopefully result in higher agreement between annotators. We plan to carry out an exploratory evaluation of the size of the evaluation presented in this paper and, after that, a more extensive one.\n\nConcerning the pointers provided by the reviewer, we will definitely incorporate the concepts provided by the NLG evaluation people, to try to make the concepts in our current and future evaluation schemes more transparent for the community. Additionally, we found the Human Evaluation Datasheet [4] very useful to systematize the analysis of our task in order to obtain a better evaluation protocol, and will describe some of how it helped in the final version of the paper. Also the analysis and proposal in [5] has been inspiring and will incorporate some of it, together with the analysis in [6]. We will also be incorporating some of the general insights of [7], which we find very cogent and valuable to share in the context of this work.\n\nAgain, we want to thank the reviewer for these pointers that will definitely help to make a more insightful, wider evaluation in the future.\n\nReferences:\n\n[1] Rishabh Gupta, Shaily Desai, Manvi Goel, Anil Bandhakavi, Tanmoy Chakraborty, Md. Shad Akhtar. 2023. Counterspeeches up my sleeve! Intent Distribution Learning and Persistent Fusion for Intent-Conditioned Counterspeech Generation\nhttps://aclanthology.org/2023.acl-long.318/\n\n[2] Ehud Reiter and Anja Belz. 2009. An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems\nhttps://aclanthology.org/J09-4008.pdf\n\n[3] Jean H.M. Wagemans and Martin Hinton. 2022. How persuasive is AI-generated argumentation? An analysis of the quality of an argumentative text produced by the GPT-3 AI text generator\nArgument & Computation\n\n[4] Anastasia Shimorina and Anya Belz. 2022. The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP\nhttps://aclanthology.org/2022.humeval-1.6.pdf\n\n[5] Rudali Huidrom and Anya Belz. 2002. A Survey of Error Annotation Schemes for Human and Machine Generated Text\nhttps://aclanthology.org/2022.gem-1.33.pdf\n\n[6] Anya Belz, Simon Mille, David M. Howcroft. 2020. Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing\nhttps://www.aclweb.org/anthology/2020.inlg-1.24/\n\n[7] Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP\nAnya Belz, Craig Thomson, Ehud Reiter, Simon Mille", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors invested much effort into building the necessary data, their hypotheses are perfectly adequate and the presented methodology is sound. Some findings even bear evidence beyond their topic of generating counter-narratives, namely that enriching input data with linguistic / argumentative information may improve small models above the level of larger models. Their critical position against BLEU, BERTScore, Rogue etc. is fully justified and its been a pleasure to note that the authors did not report any uninformative scores - this is quite rare in times where plain, sometimes unexplainable differences in BLEU scores have become the stereotypical evaluation method.", "reasons_to_reject": "One weakness, two sides of the same medal. The criteria you use for human evaluation (Informativeness, felicity etc.) are not exclusive to this subfield of generation, but are also widely discussed in the generation literature. It would have been a good idea to discuss or at least mention some of the unification ideas of Belz et al. or Howcroft et al. that is supposed to introduce more objectiveness into the terms / concepts, which otherwise remain opaque. (BUT actually, this is not the authors' fault - it is an effort that the whole NLG community has to make in order to define fundamentally improved evaluation metrics.)\n\nThe only other issue is that the conclusions one can draw from the experiments are not yet as clear as one could wish, but hopefully further research will result in a clearer picture (again this is not a conceptual error in the models or experiments - just an inherent issue with empirical studies.).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors deal with highly-targeted fine-tuning for hate speech counter-narrative generation in social media. Counter-narrative generation is the opposite to mere detection and blockage of hate speech in social media, which leads to overblocking or censoring.\nThe authors hypothesize that additional information about argumentative structure of the hate speech instances may improve the counter-narration.\n\nThe authors manually constructed counter-narratives for an existing corpus, dubbing their bilingual dataset CONEAS,  containing 4 different narratives per argumentative tweet. They trained T5 derived models on the data.\n\nSeveral experiments were conducted. \n\nFor controlling the influence of the size of the models, the authors tested several sizes of FlanT5.\nAfter preliminary testing, larger models did not perform significantly better such that tests were aborted for energy-saving reasons.\n\nFurthermore they examined the qualitative difference between fine-tuning and fewshot learning\n\nGiven their main hypothesis about argumentative information enrichment, they perfomed an ablation study on the argumentation information present in their data (ASOHMO)\n\nThey also investigated whether the type of the counter-narrative (4 types) has influence.\n\nFor evaluation, they manually assessed the criteria offensiveness, Informativeness and felicity. \n\nInterestingly, the authors find that on English data, argumentative information enrichment causes the larger model to perform worse than the smaller model!! \nThis is an instance of a pointer that linguistic engineering reduces model size need.\nThe main finding of evaluation is that finetuning a smaller model on a small set of high quality data improves generation of counter-narratives above the level of larger models. Furthermore some kind of argumentative imformation may improve informativity of generated counter-narratives.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Belz, Howcroft etc regarding evaluative criteria for human evaluation"}, "extra_text": {}}, {"review_id": "RaYUDnlIkf", "reviewer": "Reviewer_8o9t", "summary": "", "questions": "Section 3: who were the annotators? one instance of counter-narrative type per item or more? Not enough detail provided here.\n\nl392: score: why subtract 2 and not map onto a scale from 0 to 1? And how exactly is the score calculated? what are the 'values'? A Likert scale?\n\nSection 4: what were the prompts exactly?\n\nSection 4.3: very small subset of the data for the ground through - how was the data chosen? And who were the analysts? Authors of the paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[this is a general comment for all three reviewers] We appreciate the effort made by the reviewers to provide insightful feedback on our work. We understand that it does not exactly conform to the standard experimental paper with standard datasets and standard metrics. We made the decision to address questions that are not straightforward incremental on previous work, and we understand this requires an extra effort from reviewers to identify strong and weak points. We are very thankful for this effort and the very valuable suggestions that they provide. We will be sure to incorporate as much as we can of their work in the final version of this paper.\n\n[This is a shared response for two reviewers] Thank you for pointing out the missing information about the annotation process and the annotators. We will include it in the camera-ready version of the paper, in case it is accepted. The details are as follows:\nManual generation of CONEAS counter-narratives was conducted by two annotators in three phases. The annotators are both from academic backgrounds, one being a Computer Science PhD student and the other, an undergraduate Philosophy student within the area of Theory of Argumentation. Annotation guidelines were created by the research team before the annotation process began, describing how to use the argumentative analysis of hate tweets to write counter-narratives using different strategies (A, B and C). Both annotators participated on a two hour training to get familiar with the guidelines. Then, during the first phase, each annotator wrote one counter-narrative of each type for the same subsample of 20 tweets in English, then compared the results of their annotation and discussed among each other on how to improve the counter-narrative writing process according to three criteria: a - Adequacy of the counter-narrative as an acceptable, tailored reply for a given hate message; b - Adequacy of the counter-narrative to the specific type of counter-narrative; c - Robustness of the written narrative. This process is challenging since the task is inherently subjective and variability between annotators does not necessarily imply that one of them is wrong, but that guidelines need to cover such variability, without being too lax or too vague. After re-writing and consolidating the guidelines, some counter-narratives were re-written according to the updated guidelines. The second phase of the annotation process consisted in annotating 50 new tweets, and discussing issues that may have arisen and some difficult cases, just to achieve a better alignment between annotators, but without a need to update the guidelines. The third phase consisted on writing counter-narratives for all the rest of the dataset. After this third step, a third annotator reviewed 50 random pairs of hate tweet/counter-narrative using the guidelines as reference, without finding major inadequacies. The first annotator labeled 20% of the total dataset and was not paid (he is part of our research team) while the second annotator labeled the other 80% and was paid 25U$D/hour for 80 hours of work. This project was partially funded by NAACL (we plan to add an acknowledgment and sincere thanks for this in the camera ready version of the paper, if accepted). We want to thank the reviewer for pointing this out. We had originally omitted this information because of space constraints in the paper, but will include it even if it is necessary to add an Appendix.\n\nConcerning the manual evaluation of the automatically generated counter-narratives, this is in a first stage of consolidation. We are aware that evaluation of counter-narratives is a highly complex task, and we intend to carry out a more extensive evaluation with the lessons learned from this first experience. That is why this first phase was small. A small set of guidelines was developed. The annotators were the two annotators who created the manual counter-narratives and a third annotator who is also invested in the project, went through the training with the evaluation guidelines and also participated in the discussion of some test examples to build common ground between annotators. For this work, annotators were paid  the same rate per hour as for the creation of the counter-narratives, and a total of 5 hours was devoted to this evaluation, summing up all 3 evaluators.\n\nDespite the NAACL grant, we have a low budget to carry out this project and had to decide what to fund. We decided to spend most of our budget in obtaining a high quality annotated dataset, instead of investing in obtaining more powerful hardware, because we believe that the quality of annotations makes a bigger impact in this kind of problem. Our main research question was to assess whether this kind of high quality argumentative information did have an impact in the performance of models, more than extensive comparison of models, so we invested most of our little budget in obtaining a high quality dataset. We believe that, to obtain a high quality dataset, one has to employ properly educated annotators, provide them with adequate training and revision processes, and provide high quality working conditions, with good pay and as little as possible time pressure. \n\nMoreover, we also believe that, now that the dataset is publicly available, experiments with larger language models, requiring more powerful computing, can also be carried out by other teams with more computing power available.\n\n[This is a shared response for two reviewers] To our understanding, the paper we presented has all the necessary resources to reproduce the experiments described. We included a link to the Github repository (https://github.com/ConeasDataset/CONEAS) containing the annotated dataset along with a copy of the ASOHMO dataset with the argumentative information and a python script that can run the experiment described under any setting. We noted, however, that the repository doesn’t contain a README file explaining the script parameters and how to run it so we added it. Any server with the proper hardware should be able to run it. We also created a new repository with the human evaluation of the counter-narratives under https://github.com/ConeasDataset/CounterNarrativesScores. Again, we thank the reviewers for the improvement that the reviewing process has produced in our work.\n\n[This is a shared response for the three reviewers] Thanks to your comments we understand that the values of the score are difficult to interpret. In the final version of the paper we will provide the algorithm showing how the score is calculated, and will illustrate it with a couple of examples in an Appendix. We believe that a formula will not make the meaning of the score more clear, because it is a sum of the score obtained for each of the four dimensions making up the final score, which are each a Likert scale ranging from 1 to 3 in three cases and 0 to 3 in the fourth, with a maximum possible score of 12. Moreover, one of those dimensions, “Informativeness” only scores more than 0 if the dimension of “Stance” scores at the maximum of 3.\n\nNormalizing the final score to a range from 0 to 1 is definitely a suggestion that we will take, as it makes the score more interpretable. However, seeing the difficulties in interpreting the metric, we will also provide the detail of the score obtained by each approach for each of the four dimensions evaluated, as we believe this provides more transparency for the way the final score is built up.  \n\nConcerning the illustration of prompts, we included an example of the prompt used on the appendix. We don’t know if the reviewer couldn’t find the Appendix or if the provided prompt is not clear enough. We didn’t include it in the main sections of the paper due to space constraints but we will make an effort to include it in the extra page for the main body of text for the camera ready version. We will also include more examples in the Appendix.\n\nThe training, tuning and test partitions were determined by the original ASOHMO test, taking the proposed partitions as standard. For manual evaluation of the generated counter-narratives we selected a random sample of 20 tweets in English and 10 tweets in Spanish for most experimental settings, involving all types of counter-narratives. We couldn’t do a more extensive evaluation because of budget and time constraints, but are currently developing a new set of guidelines with lessons learned from this first phase of the project and will be carrying out a more extensive evaluation with these improved guidelines.\n\nWith respect to the possibility that a wrong counter narrative could be harmful, contributing to the hate in the original message, we totally agree with the reviewer that it must be signaled as a high-cost error. We are not very experienced in writing Ethical Considerations sections and some aspects have skipped our attention. In the final version of the paper, we will include this within the ethical considerations as an error to be handled with special care. Our recommendation is that counter-narrative generation should not be used as a stand-alone tool, but rather as an aid for human experts to moderate the conversation in social media and other sites of such conflicts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is laudable in itself and it is interesting that in the case of the smaller dataset of Spanish, the rise in performance seems significant.", "reasons_to_reject": "The paper is very superficial in some places and I'd need more information in the places specified below. I'd also like to see a paragraph of the ethical implications of this work, i.e. a wrong counter narrative could in principle even enhance the impact of the original message.", "questions_to_authors": "Section 3: who were the annotators? one instance of counter-narrative type per item or more? Not enough detail provided here.\n\nl392: score: why subtract 2 and not map onto a scale from 0 to 1? And how exactly is the score calculated? what are the 'values'? A Likert scale?\n\nSection 4: what were the prompts exactly?\n\nSection 4.3: very small subset of the data for the ground through - how was the data chosen? And who were the analysts? Authors of the paper?", "paper_topic_and_main_contributions": "The paper describes an approach to enhance counter-narratives with argumentative information in the social media post to generate more informative responses.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos/grammar/formatting errors are frequent in the paper, examples: newlines in abstract, capitalisation in counter-narrative types, spacing in l287ff, consistent upper-casing in lists, different indentation in Section 4"}}, {"review_id": "ChMO6qNehp", "reviewer": "Reviewer_GdEP", "summary": "", "questions": "Please see questions and concerns from the weaknesses section and address them.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[this is a general comment for all three reviewers] We appreciate the effort made by the reviewers to provide insightful feedback on our work. We understand that it does not exactly conform to the standard experimental paper with standard datasets and standard metrics. We made the decision to address questions that are not straightforward incremental on previous work, and we understand this requires an extra effort from reviewers to identify strong and weak points. We are very thankful for this effort and the very valuable suggestions that they provide. We will be sure to incorporate as much as we can of their work in the final version of this paper.\n\nWe appreciate the suggestions to improve the presentation, find them very pertinent and will incorporate them in the camera-ready version of the paper if accepted.\n\nWith respect to the fact that only results with Flan-T5 were reported, we have to describe our research context. We work in a low-income country, with a very small budget to carry out this and mostly any research project. All the resources used to run our experiments come from an HPC center in a public university, which granted us access to a cluster with one NVIDIA A10 and one NVIDIA A30, each with 24GB RAM each, and we had to go through a selection process that was time consuming to be granted this access. To be honest with the reviewer (to whom we highly appreciate all the work taken to review our work), one of the reasons (besides the carbon footprint) why we didn’t finetune Flan-T5-XL is because it doesn’t fit on the RAM memory we have available if we want to fine-tune it (if we only want to generate text it does fit, and the same thing happened with FLAN-T5-Large). We have to constantly make decisions about what is the best way to spend the funds we’ve got. In this case, we decided that it was better to spend it on developing the most amount of data of the best quality possible and then use the models that would fit in the cluster we have access to instead of buying computing resources or devoting our time to write proposals to obtain access to computing resources provided pro bono by other institutions. Although we agree that it would be interesting to fine-tune Flan-T5-XL, we think that one advantage of this experiment settings is that they allow the experiment to be reproduced without the need of costly hardware. One main standpoint of our work is that quality data is preferred over bigger models, so if future researchers with limited funds like us have to choose between either spending on annotation or spending on hardware we can provide evidence to support the first. To prove that, we believe that comparing a finetuned base model against a non-finetuned large model is sufficient. \n\nHowever, we did experiment with different architectures of models of smaller sizes. To be able to conduct our experiment according to the four categories of analysis that we proposed we needed models with at least two sizes available: a base size small enough to be finetuned without an excessive computational effort (around or less than 500M params) and a larger version used for comparison. We discarded all models that didn’t have a version lighter than 3b parameters such as Llama and that were not OpenSource, like ChatGPT and GPT4. We decided to only use open-source models available for the research community that allows us to reproduce the experiment afterwards. We chose three models complying with these specifications: Bloom (560M and 3B parameters versions), GPT2 (137M and 1.5B parameters versions) and Flan-T5 (base-248M and XL-3b parameters). We conducted small experiments with each of these models, generating counter-narratives for 20 random English tweets using a few-shot approach. We carried out a manual inspection of the results and found that Flan-T5 produced the most coherent responses by far, at least using this approach and for the base model. Besides that, only Flan-T5 and Bloom are multilingual, and we were interested in the approach working straightforwardly for Spanish. We didn’t include these results because of space constraints but mostly because we didn’t evaluate these results thoroughly using our proposed four categories of analysis. We agree with the reviewer, however, that it is important to add justification about why we chose the model that we used for our experiments and we will add both a description of the comparison we did between those models and the evaluation of the generated results using our four categories of analysis for the camera-ready if our work is accepted.\n\n[This is a shared response for the three reviewers] Thanks to your comments we understand that the values of the score are difficult to interpret. In the final version of the paper we will provide the algorithm showing how the score is calculated, and will illustrate it with a couple of examples in an Appendix. We believe that a formula will not make the meaning of the score more clear, because it is a sum of the score obtained for each of the four dimensions making up the final score, which are each a Likert scale ranging from 1 to 3 in three cases and 0 to 3 in the fourth, with a maximum possible score of 12. Moreover, one of those dimensions, “Informativeness” only scores more than 0 if the dimension of “Stance” scores at the maximum of 3.\n\nNormalizing the final score to a range from 0 to 1 is definitely a suggestion that we will take, as it makes the score more interpretable. However, seeing the difficulties in interpreting the metric, we will also provide the detail of the score obtained by each approach for each of the four dimensions evaluated, as we believe this provides more transparency for the way the final score is built up. \n\n[This is a shared response for two reviewers] Thank you for pointing out the missing information about the annotation process and the annotators. We will include it in the camera-ready version of the paper, in case it is accepted. The details are as follows:\nManual generation of CONEAS counter-narratives was conducted by two annotators in three phases. The annotators are both from academic backgrounds, one being a Computer Science PhD student and the other, an undergraduate Philosophy student within the area of Theory of Argumentation. Annotation guidelines were created by the research team before the annotation process began, describing how to use the argumentative analysis of hate tweets to write counter-narratives using different strategies (A, B and C). Both annotators participated on a two hour training to get familiar with the guidelines. Then, during the first phase, each annotator wrote one counter-narrative of each type for the same subsample of 20 tweets in English, then compared the results of their annotation and discussed among each other on how to improve the counter-narrative writing process according to three criteria: a - Adequacy of the counter-narrative as an acceptable, tailored reply for a given hate message; b - Adequacy of the counter-narrative to the specific type of counter-narrative; c - Robustness of the written narrative. This process is challenging since the task is inherently subjective and variability between annotators does not necessarily imply that one of them is wrong, but that guidelines need to cover such variability, without being too lax or too vague. After re-writing and consolidating the guidelines, some counter-narratives were re-written according to the updated guidelines. The second phase of the annotation process consisted in annotating 50 new tweets, and discussing issues that may have arisen and some difficult cases, just to achieve a better alignment between annotators, but without a need to update the guidelines. The third phase consisted on writing counter-narratives for all the rest of the dataset. After this third step, a third annotator reviewed 50 random pairs of hate tweet/counter-narrative using the guidelines as reference, without finding major inadequacies. The first annotator labeled 20% of the total dataset and was not paid (he is part of our research team) while the second annotator labeled the other 80% and was paid 25U$D/hour for 80 hours of work. This project was partially funded by NAACL (we plan to add an acknowledgment and sincere thanks for this in the camera ready version of the paper, if accepted). We want to thank the reviewer for pointing this out. We had originally omitted this information because of space constraints in the paper, but will include it even if it is necessary to add an Appendix.\n\nConcerning the manual evaluation of the automatically generated counter-narratives, this is in a first stage of consolidation. We are aware that evaluation of counter-narratives is a highly complex task, and we intend to carry out a more extensive evaluation with the lessons learned from this first experience. That is why this first phase was small. A small set of guidelines was developed. The annotators were the two annotators who created the manual counter-narratives and a third annotator who is also invested in the project, went through the training with the evaluation guidelines and also participated in the discussion of some test examples to build common ground between annotators. For this work, annotators were paid  the same rate per hour as for the creation of the counter-narratives, and a total of 5 hours was devoted to this evaluation, summing up all 3 evaluators.\n\nDespite the NAACL grant, we have a low budget to carry out this project and had to decide what to fund. We decided to spend most of our budget in obtaining a high quality annotated dataset, instead of investing in obtaining more powerful hardware, because we believe that the quality of annotations makes a bigger impact in this kind of problem. Our main research question was to assess whether this kind of high quality argumentative information did have an impact in the performance of models, more than extensive comparison of models, so we invested most of our little budget in obtaining a high quality dataset. We believe that, to obtain a high quality dataset, one has to employ properly educated annotators, provide them with adequate training and revision processes, and provide high quality working conditions, with good pay and as little as possible time pressure. \n\nMoreover, we also believe that, now that the dataset is publicly available, experiments with larger language models, requiring more powerful computing, can also be carried out by other teams with more computing power available.\n\nConcerning how limited the few-shot learning approach was, we actually conducted experiments for the few-shot approach providing more examples in the prompt for the Flan-T5 model. In particular, we tried prompts with 3, 5 and 10 examples. In all cases, both for base and XL models, when 5 or more examples are used, the text generated by the model is always the same no matter the input taken. In these cases, the model reproduces, either verbatim or with small variations, one of the counter-narratives defined on the examples used on the prompt. Although in some cases, the counter-narrative generated by the model could be considered more or less accurate, we consider this to be mere chance, since the model always returns the same, no matter the input. We hypothesized that this behavior might be due to the model falling in a local minimum. To force the model to produce variability in the output, we ran a small test using different generation settings (particularly, using top-sampling and temperature) and checked that in these cases there is higher variation among the responses, but the quality of the generated text was noticeably worse. Since we couldn’t devote the time to properly understand the underlying reason for this anti-intuitive behavior, and considering that manual evaluation of the generated counter-narratives is costly, we decided not to evaluate these cases, and because of space constraints, we didn’t include a description of this phenomenon in the paper. \n\nWe chose to use only two examples and low temperature considering that in a preliminary observation, this seemed to be the number that yielded good but also diverse replies. However, we understand now with the help of the comments made by the reviewer, that it is important to report this and we are planning on doing so. We will describe the experiments we carried out with more examples in the prompt, showing the average distance between the sentence embedding of one generated counter-narrative to the sentence embedding of all the rest. Experiments where all the generated responses are the same will have an average distance of zero, indicating that the model always returns the same. This indicator will be useful to motivate the choice of n=2 for the number of examples provided in the prompt. \n\n[This is a shared response for two reviewers] To our understanding, the paper we presented has all the necessary resources to reproduce the experiments described. We included a link to the Github repository (https://github.com/ConeasDataset/CONEAS) containing the annotated dataset along with a copy of the ASOHMO dataset with the argumentative information and a python script that can run the experiment described under any setting. We noted, however, that the repository doesn’t contain a README file explaining the script parameters and how to run it so we added it. Any server with the proper hardware should be able to run it. We also created a new repository with the human evaluation of the counter-narratives under https://github.com/ConeasDataset/CounterNarrativesScores. Again, we thank the reviewers for the improvement that the reviewing process has produced in our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles a very important application of NLP in a real-world scenario which is the generation of counter-narratives against hate speech (and fake news probably as well). The motivation is clear, and readers would easily appreciate the efforts of the authors done for the study. The paper is also relatively well written (but requires some level of structuring and reorganization to make it more readable to the wider audience) and provides an acceptable level of analysis of results. More than that, I believe two of the datasets constructed for this study, the CONEAS data plus the additional ground truth dataset used for testing, would be beneficial to the NLP community.\n", "reasons_to_reject": "While there are definitely strong points to the paper, I still have some concerns about the technical rigor and how the paper details some of its experimental procedures. Addressing these would help improve the quality of the work.\n\n1. The annotation process and the annotators should be described in more detail, including reliability testing, payment, and background qualifications--especially since subjectivity is one major concern for the annotation of these types of tasks. The annotation process should be clearly described, including the instructions given to the annotators and the quality control measures that were used. This information was provided for the creation of the ground truth dataset, but not for the construction of the CONEAS dataset. \n\n2. The proposed few-shot learning seems to be sensible for the study, but the experiment done for the paper seems to be very limited to me. If only two examples are used at a time, the model will not be able to learn as much as it could if more examples were available. I believe it would be interesting to see if there are positive changes in the model's performance as more or fewer examples are given to it during training. Especially if, in the discussion section, the authors highlighted some improvements in using a smaller T5 model compared to the XL one.\n\n3. The authors seem to be overusing the claim of reducing carbon footprints to not perform further experiments to further ground the findings. While this reason may be noble in some sense, I don’t find it as an acceptable excuse to limit the freedom of the authors to do more training that will support their results. Especially for not finetuning Flan-T5 XL which leaves the whole experiment recipe to be incomplete.\n\n4. There should be justification as to why Flan-T5 was used for the task compared to other existing generative models that are easy to finetune such as GPT2. Is there some special feature with Flan-T5 that makes it better for the task? \n", "questions_to_authors": "Please see questions and concerns from the weaknesses section and address them.\n", "paper_topic_and_main_contributions": "The paper describes work on the addition of human-annotated argumentative aspect information to the standard finetuning and few-shot learning to train generative models to combat hate speech by generating counter-narratives. The paper derives its motivation from the fact that current approaches in the field only ever make use of the actual hate speech text itself for finetuning, and no one has yet explored on incorporating context in the form of extracting argumentative information and using this in for training models. The paper then collects four types of argumentative information from texts, including negations between justification and conclusion, property and collective, attack justification, and free counter-narrative. The paper explores training only on a very limited set of models and experimental procedures, citing concerns on carbon footprint and limited computational resources. Results show that the use of additional argumentative information does improve performances even for languages such as Spanish with relatively small datasets compared to English and that smaller models have produced more favorable effectivity than larger models for the task.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The title is grammatically incorrect. It sounds more acceptable as “low resource approaches” than “low resources approaches”. \n\nIt might be better for the EMNLP audience as well as general readers for the authors to formalize the scoring mechanism developed in a mathematical formula. I suggest this as this can serve as one of the main contributions of the paper. The impression I’m getting from the scoring is that there may be two formulas to be added since it was mentioned in the criteria that for Informativeness, only counter-narratives with Stance are evaluated. All of these should be properly formalized to avoid confusion.\n\nIn Section 4.3, I suggest adding one table containing the details of the new ground truth dataset created. Maybe add this in the Appendix just to have another reference for readers who need quick details about your dataset.\n\nI appreciate that the authors provided in-depth discussion from the results they obtained in Table 3. However, condensing all of these in one long paragraph reduces the readability of the discussion. I suggest the authors to add subsections to split the different parts of discussions into something that in targets from the experiment. For examples: Subsection A. Effect of Model Size to Performance, Subsection B. Fewshot vs. Finetuning, etc.\n"}}]

Author rebuttal:

Meta-review: This work tackles counter-narrative generation for hate speech on social media. Unlike existing works largely leveraging the hate speech itself, this work uses argumentative structures from the context. Results show that adding argumentative structures helps boost the results even in a few-shot setting.

Reviewers appreciated the importance of the task, effective use of argument information and the two datasets constructed for this study.

At the same time, reviewers are concerned about the limited details on the annotation process, the discussion on evaluation, and the limited experiments conducted to underpin the main claims of the paper. More discussions on the annotation and evaluation will be helpful to the community.
Decision: Accept-Findings