Finding Authentic Counterhate Arguments: A Case Study with Public Figures
Authors: Abdullah Albanyan
Ahmed Hassan
Eduardo Blanco
Venue: EMNLP 2023 Main (2023)
Keywords: Hate Speech
Counterhate
Social Media

Abstract: We explore authentic counterhate arguments for online hateful content toward individuals. Previous efforts are limited to counterhate to fight against hateful content toward groups. Thus, we present a corpus of 54,816 hateful tweet-paragraph pairs, where the paragraphs are candidate counterhate arguments. The counterhate arguments are retrieved from 2,500 online articles from multiple sources. We propose a methodology that assures the authenticity of the counter argument and its specificity to the individual of interest. We show that finding arguments in online articles is an efficient alternative to counterhate generation approaches that may hallucinate unsupported arguments. We also present linguistic insights on the language used in counterhate arguments. Experimental results show promising results. It is more challenging, however, to identify counterhate arguments for hateful content toward individuals not included in the training set.

Finding Authentic Counterhate Arguments:
A Case Study with Public Figures
Abdullah Albanyan," Ahmed Hassan, and Eduardo Blanco"
"College of Computer Engineering and Sciences, Prince Sattam Bin Abdulaziz University "Department of Communication and Information Engineering, Zewail City University 'Department of Computer Science, University of Arizona
a. al banyan@psau. edu. sa
5 s-ahned_ hassanézewailcis wwwp:aaiiiittttutteeeeeee edu.eg eduardobl Geuardoblancc@arizzn
Abstract
We explore authentic counterhate arguments for online hateful content toward individuals. Previous efforts are limited counterhate to fight against hateful content toward groups. Thus, we present a corpus of 54,816 hateful tweet-paragraph pairs, where the paragraphs are candidate counterhate arguments. The counterhate arguments are retrieved from 2,500 online articles from multiple sources. We propose a methodology that assures the authenticity of the counter argument and its specificity
Messi racist!!!! Hope he gets suspended. #c
Synthetic counterhate argument (generic, generated on demand by experts automatically):
This kind of unsubstantiated statements not al- lowed as it demeans and insults others.
Authentic counterhate argument from Dailypost.ng:
www. dailypost og/2812/45/11/ [-] --ire--aadk-drinthaee claies
"The player [Messi] has always shown a maximum re- spect and sportmanship towards his rivals, something which has been recognized by his [...]'
the individual of interest. We show that finding arguments in online articles an efficient alternative counterhate generation approaches that may hallucinate unsupported arguments. We also present linguistic insights on the language used in counterhate arguments. Experimental results show promising results. is more challenging, however, to identify counterhate arguments for hateful content toward individuals not included the training set.
Authentic counterhate argument from Quora.com:
Vis. quora, Newii--tass-racist
He [Messi] could be harsh and that's due to the frus- tration during the game [..]. it's all love from Messi.
Figure 1: Hateful tweet (top) and three replies with counterhate. Previous work targets synthetic counter- hate arguments that tend be generic. In this paper, find authentic counterhate arguments that address the hateful claims the tweet hand.
1 Introduction
Social media platforms such as Twitter and Face- book are used by many derson (2021) report that 72% of U.S. adults use at least social media site. Content in social media can spread fast and help community organi- zations. For example, the first report of the 2017 Las Vegas shooting was tweet posted ten minutes after the event started (Blankenship and Graham, 2020). More recently, Twitter and hashtags have been credited with facilitating the organization of protests and the social movement that followed the killing of George Floyd (Wirtschafter, 2021).
hate. Hateful content in social media is widespread. In 2017, a study conducted on 4,248 U.S. adults showed that (a) 41% were personally subjected to online hate speech and (b) 66% witnessed hate speech directed toward others (Duggan, 2020).
Auxier and An-
Hateful content in social media has conse- quences in the real world. For example, there is evidence that several celebrities have committed suicide after being the target of hateful content social media (Today, 2020). Additionally, an analy- sis of the Japanese daily death registry and one mil- lion tweets between 2010 and 2014 revealed that there is a correlation between reports of celebrity suicides in social media and suicides committed by regular people (Ueda al., 2017).
Social media has several positive effects includ- ing the ability share information at low cost and being a vehicle for free speech where anyone can express their views. also has negative ef- fects including the spread of misinformation and
There have been many efforts to identify hateful content online (Section 2). Social media platforms

---

have reported spending billion per year on these efforts. Regardless of how hate is identified, one could keep the content but flag it, delete the content, or ban the author. An alternative compatible with freedom of speech is to counter the hateful content, a strategy shown to be effective at minimizing the spread and limiting the consequences of hateful content (Gagliardone et al., 2015). Consider the hateful tweet and three counterhate arguments in Figure 1. Previous work, discussed in Section 2, has targeted synthetic counterhate arguments that (a) are generated on demand by experts (Chung et al., 2019; Fanton et al., 2021) or (b) condemn the language but do not address the specific hateful claims and as result generate generic counterhate arguments (e.g.. "Use of this language is not toler- ated and it is uncalled for," (Qian et al., 2019)). In this paper, we instead target authentic counterhate arguments that address the specific hateful claims, as exemplified in the bottom two arguments Fig- ure We are inspired by previous work outside of the counterhate domain pointing out that (a) URLs increase perceived trust (ODonovan et al., 2012; Morris et al., 2012) and (b) good arguments have to appeal to logic by including factual, testimonial, or statistical evidence (Habernal and Gurevych, 2016). Therefore, authentic counterhate arguments with sources have the potential be more effective than generic statements condemning hate.
The main contributions are:
a corpus 250 hateful tweets toward 50 pub- lie figures (5 cach) and 2,500 candidate arti- cles for authentic counterhate arguments; annotations indicating whether the 54,816
paragraphs in the 2,500 candidate articles are authentic counterhate arguments;
analysis characterizing the language used make authentic counterhate arguments; experimental results showing that the task can be partially automated; and qualitative analysis describing when the task is the hardest to automate.
Previous Work
Identifying hateful content in user-generated con- tent has received substantial attention in recent years (Fortuna and Nunes, 2018). Rescarchers have presented several datasets and models for hate de- tection in Twitter (Waseem and Hovy, 2016; David- son et al., 2017), Yahoo (Warner and Hirschberg, 2012; Djuric et al., 2015; Nobata et al., 2016). Face- book (Kumar al., 2018), Gab (Mathew et al., 2021), and Reddit (Qian et al., 2019).
Previous efforts have also worked on identify- ing the target of hate (e.g., a group, an individ- ual, an event or object). For example, Basile et al. (2019) differentiate between groups and individ- uals. and between immigrant women and other groups. Zampieri et al. (2019a), differentiate be- tween groups, individuals, and others (e.g., an event or object). Similarly, Ousidhoum et al. (2019) de- tect hateful content in several languages and iden- tify the target from a set of 15 groups
The work presented here works with hateful tweets toward public figures and authentic counter- hate arguments published online. A hateful tweet is, according the Twitter guidelines, any implicit or explicit tweet that attacks an individual's gender, religion, race, ideology, or social class. Our defi- nition of authentic counterhate argument borrows from previous work on counterhate (Mathew et al., 2019) and argumentation (Habernal and Gurevych, 2016). We define counterhate direct response that counters hate speech and an authentic argu- ment paragraph that appeals to logic by in- cluding factual, Iestimonial, or statistical evidence. In the remainder of this paper, we may refer to authentic counterhate argument as counterhate to save space. The work presented here could be un- derstood as specialized information retrieval task where the goal is to identify authentic counterhate arguments against hateful tweet in (a) online arti- cles and (b) paragraphs within the articles.
Countering Hate Previous work on countering hate can be divided into two groups: detection and generation. Counter hate detection consists in iden- üfying whether a piece of text counters hateful con- tent. For example, Mathew et al. (2020) identifies counterhate in the replies to hateful tweets. They rely in a simple pattern to detect hateful tweets: hate <group>. Garland et al. (2020) work with German tweets from two well-known groups and define hate and counterhate based on the group the authors belong to. He et al. (2021) work with tweets related to COVID-19 and identify hate and counterhate using 42 keywords and hashtags.
Generating counterhate is arguably more chal- lenging than detecting it. Qian et al. (2019) crowd-
"htms://hit.ly3JJ9PDD
hhtp:///hhhhb.ommmmaaaaaaoounnnc puragraph

---

source a collection of counter hate interventions and present models to generate them. Similarly, Fanton et al. (2021) direct their focus to other groups including Jews, LGBTQ+, and migrants. Chung et al. (2019) present CONAN, a collection of anti-Muslim hateful content and counterhate replies. Both the hateful content and counterhate replies in CONAN were written by expert operators on demand; thus, is unclear whether this synthetic dataset would transfer to content written by regu- lar people. More related to our work, Chung et al. (2021) use external knowledge and GPT-2 to gener- ate counterhate arguments against specific hateful claims, similar to what we define here as authentic counterhate. Generating counterhate is certainly a valid strategy, but GPT-2 and other large pretrained models are known for hallucinating facts. In this paper, we bypass this issue and instead retrieve au- thentic counterhate arguments from online articles. We find counterhate arguments for 72% of hateful content we consider, making this strategy viable. Further, all previous efforts are limited to counter- hate to fight against hateful content toward groups Unlike them, we explore counterhate for hateful content toward individuals.
individuals. The second step is necessary ensure that we do not have hateful tweets that do not target the individual at hand. For example, You are all a bunch of fu *rss. is the greatest actor ever is hateful but is not hateful toward X.
Selecting Hateful Tweets First, we retrieve tweets containing the name of any of the 50 individ- uals using the Tweepy Python library (Roesslein, 2020). Second, we feed these tweets to hate clas- sifier, HateXPlain model (Mathew et al.. 2021), to automatically identify the tweets that are hateful.
Identifying Hate Segments toward Individuals Hateful tweets mentioning an individual are not necessarily hateful toward that individual. In order to ensure that the tweets we work with target one of the 50 individuals, we define several patterns grounded on part-of-speech tags. The first pattern consists of the name of the individual followed by is and a phrase headed by either noun or an adjective (e.g., X is a big ole X is dumb as rocks). Appendix B provides the part-of-speech tags we consider for phrases; they are a combina- tion of determiners, adjectives, adverbs, and either noun or an adjective. The second pattern is in- spired by Silva et al. (2016) and consists of I, an optional adverb, a hateful verb, and the name of the individual. The list of hateful verbs includes hate, despise, detest, and abhor among many others.
3 A Collection of Hateful Tweets and Authentic Counterhate Arguments
We start our study creating a corpus of hateful tweets and authentic counterhate arguments against the hateful content. We focus on hateful tweets toward 50 public figures which we will refer to individuals. The supplementary materials list the 50 individuals; we collected their names from online sources. 3 The list of 50 individuals is not sender-balancee (male: 30, female: 20) but in- cludes people working in professions such as politi- cian, comedian, singer, and athlete. also includes, for example, politicians with opposing views in- cluding Joe Biden and Donald Trump. Appendix A details the 50 individuals.
We use segment to refer to the text matching pattern. Despite segments come from hateful tweets mentioning an individual, there is no guar- antee that the segment is hateful toward the indi- vidual or even hateful at all. In order to avoid this issue, we (a) run Patan on the segment and (b) keep hateful segments and discard the rest. Here is an example of a hateful tweet mentioning an in- dividual that we discard because there is no hateful segment toward the individual (underlining indi- cates the segment): Y'all are ignorant piece of #* X is fu *inn gorgeous (segment is offensive but not hateful toward X).
3.1 Collecting Hateful Tweets toward Individuals
We use the part-of-speech tagger in spaCy (Neu- mann et al., 2019). Table presents the number of individuals and average number of hateful tweets per profession. Politicians and comedians receive more hate (8.6 and 8.0 on average), and hosts and journalists the least (1.0). The average number of hateful tweets is 4.72 across all professions. Based on this average and to ensure we work with hateful tweets targeting a variety of individuals, we re-
We automatically collect hateful tweets toward the 50 individuals as follows. First, we identify hateful tweets that mention the name of each individual. Second, we impose some filters these tweets to ensure that the hateful content targets one of the 50
https://bt.lyy3355ccc. https//bit.ll//kk6oss

---

Profession
#Individuals
Avg. #hateful tweets
Authentic Counterhate?
Politician
15
8.60
no
yes
Comedian
8.00
Paragraph
52,451 (95.7%)
2,365
(4.3%)
Actor
3.56
Article 1,961 (78.4%)
539
(21.6%%
Singer
11
3.54
Athlete
3,40
Table 2: Label percentages at the paragraph and article levels. We work with 54,816 paragraphs and 2,500 articles. The distribution is biased toward no at both levels (i.e., authentic counterhate arguments rare).
Entrepeneur
3.00
Model
2.00
Host
1.00
Journalist
1.00
All
50
4.72
We use these queries with Google Search API and retrieve (a) the top-5 online articles from Quora and (b) the top-5 online articles from other sources. We put an emphasis on Quora articles after we con- ducted a manual examination and discovered that Quora often has civil conversations where people present their opposing views about public figure.
Table 1: Number of individuals and average number of hateful tweets profession of the 50 public figures we work with. The final collection process resulted in hateful tweets per individual.
peat the process to collect hateful tweets described above until we obtain 5 tweets containing unique hateful segments toward each of the 50 individuals. By unique, we mean up to segment matching noun or an adjective in the first pattern (X is f... ] noun_or caim_o__adjecciie adjective) and one segment matching the second pattern (I optional _ddverb hateful_ _erefull_erb_a X) per individual. The total number of tweets (and unique segments) is thus 250.
Extracting Text and Paragraphs We extract the text from Quora articles using customized parser built using BeautifulSoup." For other articles, we use Goose, 5 a Python library that abstracts away the parsing details required to extract text from html documents. We then split text into paragraphs using the <p> html tag. The result is 54,8 6 paragraphs from 2,500 articles (50 individuals x 5 hateful tweets X 10 articles = 2,500).
We manually validated the results of the filtering with sample of 200 discarded hateful tweets and discovered that most of the tweets (83%) were not hateful toward the individual. Here are two more examples: X is not currently running for political office, dirty haters (no segment) and Just so you know: X hates racists (segment is not hateful).
3.3
Validating Candidates for Authentic Counterhate Arguments
The last step to collect hateful tweets and authentic counterhate arguments is to manually validate the candidate arguments. We used Label Studio 6 as an annotation tool. The tool showed the hateful tweet with the hateful segment highlighted, and it guided annotators through each paragraph in an article. Annotators decided whether the paragraph was counterhate argument following the provided an- notation guidelines. These guidelines gave enough details on when to label paragraph as a counter- hate paragraph. For example, a paragraph is anno- tated as counterhate when criticizes the hateful claim or provides a fact that contradicts the hateful claim. Two annotators participated in the annota- tion task; both are active social media users and have worked before in the hate and counterhate do- main. They invested 180 hours in the annotations. In the first phase of annotations, both of them anno- tated all paragraphs from ten candidate articles for
3.2 Collecting Candidates for Authentic Counterhate Arguments
Finding authentic counterhate arguments counter- ing hateful content is harder than it may seem. While finding positive content about anyone with public presence generally easy (e.g.. X will save our country from ruin), finding an argument that directly counters the hateful segment more nu- anced. We design a two-step procedure to first find candidate arguments and then validate them.
Retrieving Candidate Articles For each of the 250 unique hateful segments, we retrieve 10 online articles. We use the following search queries:
"X is not adjective / noun" for the first pattern,
X is [...] adjective_or__t sorrrcttvve_ooooun.__
"Reasons to like X" for the second pattern, I optienal_adverb adverb verb X.
" http:///www.../m///oiiiehhhhhooo https::/github.bo hhtp:://githuh..cm/haaatttttaaae http://hww....oomm/aaaeett////aa

---

Hateful Tweet: Angelina Jolie is a horrible person with no acting ability
Counterhate paragraph (yes label); To those people who load nasty comments about celebrities, [--]. I would challenge any of you to do what Jolie does; be a mother, actor, director and a humanitarian while dealing with your own severe health issnes. Her work as amhassador for United Nations has created awareness of the struggles of both children and adult refugees.
Nos Counterhate paragraph (and negative; no label): have no idea, am TV critic & know she shouldn't be considered a A list actor, believe she has that label because of her dad Being Jon Voit [..]-
Not Counterhale paragraph (and positive; no label): She is beautiful and has amazing appearance.
Table 3: Hateful tweet and three paragraphs from our corpus. The first one contains a counterhate argument. Note that consider counterhate arguments statements that counter the specific hateful claims (underlined). The second and third ones are negative and positive toward Angelina Jolie but neither one counters the hateful tweet.
a hateful segment toward each individual (500 arti- cles; 20% of them). The inter-annotator agreement (Cohen's 's k) was 0.75, which indicates substantial agreement; coefficients between 0.60 and 0.80 are considered substantial agreement and above 0.80 (almost) perfect (Artstein and Poesio, 2008). Given the high agreement, we split the remaining articles in half and each annotator annotated one half (all paragraphs in 1,000 articles each).
2 and 264, respectively, while the average number of counterhate arguments per individual 47.4.
We present a hateful tweet and three paragraphs from our corpus in Table 3. The tweet is a hateful tweet toward Angelina Jolie (i.e., a horrible per- son). Annotators chose yes for the first paragraph as it presents reasons not to believe that Angelina Jolie is a horrible person, the hateful segment in the tweet. On the other hand, the next two para- graphs are not authentic counterhate. The second paragraph is negative toward Angelina Jolie and criticizes her acting abilities-it doesn't 't address the hateful segment. The third paragraph is posi- tive toward Angelina Jolie but does not counter the hateful tweet with a convincing argument- being bcautiful and having good appearance are arguably not incompatible with being horrible person.
4 Analyzing Counterhate Arguments
Table 2 shows the label percentages (no and yes) at the paragraph and article levels. Out of the 54,816 paragraphs in the 2,500 online articles, only 4.3% are authentic counterhate arguments. In other words, authentic counterhate arguments are rare despite we designed the retrieval of online articles improve the chances of finding counterhate. The number of articles having at least one paragraph containing an authentic counterhate argument is higher (21.6%). Despite the low percentages, we note that our collection includes at least one para- graph containing an authentic counterhate argu- ment for 72% of hateful tweets. This is true de- spite limiting ourselves finding authentic coun- terhate arguments in 10 online articles per hate- ful tweet. We also note that it is more likely to find counterhate arguments for certain professions. For example, the total number of counterhate ar- guments is low for professions such as journalists and hosts (averages are 4 per individual and 6.4 per individual, respectively). On the other hand, the number is high for professions such as athletes and politicians (averages are 135.8 per individual and 1 per individual, respectively). Additionally, regarding the number of counterhate arguments per individual, we find that the lowest and the highest number of counterhate arguments per individual are
4.1 Linguistic Analysis
We conduct a linguistic analysis comparing the lan- guage in (a) paragraphs that contain authentic coun- terhate arguments (yes) and those who do not (no); and (b) articles that have at least one paragraph containing an authentic counterhate argument and those who do not (Table 4). We tokenize and part- of-speech tag text count pronouns and proper nouns using spaCy (Neumann et al., 2019). In order to identify negation cues, we use a RoBERTa- based cue detector trained with CD-SCO (Morante and Blanco, 2012). We use profanity lexicon to identify profanity words, 7 and consider words misspelled if they do not appear in the Brown cor- pus (Francis and Kucera, 1979) or the lexicon of profanity words. Finally,
sentiment lex- icon containing positive and negative words (Mo- hammad and Turney, 2013) and TextBlob to ob-
'htps://bitlly/3pddyyP

---

Paragraph
Article
report results with random 70/10/20 split. Then. explore a more realistic- and challenging- scenario: using in the test set candidate counterhate arguments for hateful tweets toward wnseen indi- viduals during training (Section 5.2).
Number of.
tokens
pronouns
11
first person
second person
44
Baselines We also present results with two base- lines: majority and random labels. The majority label is no for both paragraphs and articles.
proper nouns
111
negation cues
11
111
question marks
444
profanity words
41
44
misspellings positive words
444
Paragraph-Level Neural Classifier We exper- iment with neural classifiers built top of BERT-based transformer. Specifically, we use the RoBERTa transformer (Liu et al., 2019), which is pretrained with 800M words from the BooksCorpus (Zhu et al., 2015), English Wikipedia (2,500M words), CC-News (63M English news articles) (Nagel, 2016), Open (Gokaslan et al., 2019), and Stories (Trinh and Le, 2018). The neural architecture consists of RoBERTa, fully connected layer with 128 neurons and ReLU tivation, and another fully connected layer with 2 neurons and a softmax activation which outputs the prediction (no or yes). We tried several input choices (individually and combinations):
tt
negative words
44
Polarity score Subjectivity score
44
111
111
Table 4: Linguistic analysis counterhate arguments at the paragraph and article levels. Arrow direction indicates whether higher values indicates counterhate (up) or (down). Number of arrows indicate the p- value (t-test; one: < 0.05, two: 0.01, and three: p < 0.001). All tests pass the Bonferroni correction.
tain polarity and subjectivity scores. We make several interesting observations:
Longer articles are more likely to contain counterhate, but paragraph length is not good indicator of counterhate.
the article title,
the hateful tweet,
the hateful segment, and
The more pronouns the more likely is para- graph or article to contain counterhate. The more recond-person pronouns a paragraph, however, the less chances of being counter- hate. This is because these pronouns (you, your) are often used to attack the author of the hate rather than the hateful content.
the candidate paragraph.
In order feed to the network with different combinations of inputs, we concatenate them with the separator special token 
.
A Article-Leve Neural Classifier We experiment with the same architecture the paragraph-level classifier, however, we use the Longformer trans- former (Beltagy et al., 2020) instead of RoBERTa. The Longformer can handle longer input texts. We also consider several inputs:
Negation cues, positive words, and subjective language indicate counterhate in both para- graphs and articles.
Profanity, misspellings, and negative words are rare in paragraphs containing counterhate.
the article alone.
the hateful tweet, and
the hateful segment.
5 Experiments and Results
We also experiment with a system that aggre- gates the paragraph- predictions. This system reuses the neural classifier at the paragraph level and outputs yes if any of the paragraphs are identi- fied to be an authentic counterhate argument.
We experiment with classifiers to determine (a) whether paragraph is an authentic counterhate argument against a hateful tweet and (b) whether an article contains a paragraph that is an authen- tie counterhate argument against hateful tweet. All the neural classifiers are built using pretrained language models from HuggingFace (Wolf al.. 2020) and Pytorch (Paszke al., 2019). We tune hyperparameters with the training and development splits, and report results with the test split. We first
5.1 Quantitative Results
First, we present experimental results with ran- dom 70/10/20 split (Table 5). Second, we present results using 70/10/20 split in which there is no overlap between individuals (toward whom the hate

---

P
R
F1
somewhat identifiable despite it is unknown what hate they are countering. Surprisingly, combining the paragraph with the article title yields much bet- ter results (F1: 0.65). Recall stays roughly constant for any combination of two or more inputs that include the paragraph (R: 0.75-0.77) Increasing precision, however, requires the hateful tweet or segment in the input. Either one yields roughly the same results (P: 0.57 vs. 0.65 and 0.68; F1: 0.70
Paragraph-Level Predictions
Majority baseline
0.00
0.00
0.00
Random baseline
0.04
0.50
0.08
RoBERTa trained with...
article title
0.54
0.19
0.28
hateful tweet
0.00
0.00
0.00
article title
0.59
0.22
0.32
hateful segment
0.44
0.06
0.10
article title
0.61
0.21
0.32
and 0.71). These results lead to the conclusion that unnecessary to identify the hateful segment prior to feeding the tweet to the network.
paragraph
0.62
0.46
0.53
article title
0.57
0.76
0.65
hateful tweet
0.65
0.76
0.70
+ article title
0.64
0.77
0.70
We also explore pretraining with complementary tasks, but it is barely beneficial. Our rationale is to leverage existing datasets with counterhate amples, which are the minority class in our corpus (Table 2). We adopt the method by Shnarch et al. (2018), which incorporates annotated data from related tasks using different ratios in each train- ing epoch. In the first epoch, all instances from the related task are used for training, and the ratio is decreased all the way to zero in the last epoch. The corpora pretrain with are CONAN (Chung et al., 2019) and Sultitarget-OONAN (Fanton al., 2021). CONAN consists of 6,654 hate speech ex- amples (related to Islamophobia) paired with syn- thetic counterhate statements written by experts. Multitarget-CONAL
hateful segment
0.68
0.75
0.71
article title
0.67
0.76
0.71
pretraining
0.69
0.75
0.72
Article-Level Predictions
Majority baseline
0.00
0.00
0.00
Random baseline
0.21
0.50
0.30
LongFormer trained with...
article
0.70
0.24
0.36
hateful tweet
0.62
0.30
0.41
+ hateful segment
0.67
0.29
0.41
Aggregate par.-level preds.
0.65
0.85
0.74
Table 5: Results for the yes label. We present results for identifying counterhate arguments paragraphs and articles using several inputs. Aggregate paragraph-level predictions predicts yes if any the paragraph- level predictions for article is yes. The appendix provides additional results with more input combinations and pretraining with more related tasks.
similar but includes hateful content toward multiple targets. It includes 5,003 examples collected with a cemi-automatic proce- dure. Pretraining is barely beneficial (F1: 0.72 vs. 0.71). We hypothesize that it is due to two facts. First, CONAN and Multitarget-CNNAN include hateful content toward groups (Muslims, LGBTQ+, etc.) rather than individuals. Second, the coun- terhate in these corpora synthetic and does not address the hateful claims with what we refer to as an argument; it condemns the hateful content without arguing against it.
is directed) in the training, development, and test splits (Section 5.2). The results in the paper are for the most important label: yes (i.e., identifying authentic counterhate arguments). The appendices present detailed results including Precision, Recall, and F1-measure for and yes labels.
Paragraph-Level Counterhate Table 5 presents the results at the paragraph level (top block), i.e.. identifying whether paragraph is an authentic counterhate argument for given hateful tweet. The article title alone outperforms the baselines but obtains modest results (F1: 0.28). Either the hateful tweet or hateful segment by itself yields poor results (F1: 0.00, 0.10), leading to the conclu- sion that the hateful content (both the full tweet or only the segment) is not good signal for whether one is more likely to find a counterhate argument.
A Article-Level Counterhate ticle level (Table 5, bottom block) present similar trends. The Longformer using the text in the article alone outperforms the baseline (F1: 0.36), and in- cluding either the hateful tweet or just the segment brings the F1 to 0.41. In other words, while the model can identify good arguments just by looking at the article, it is beneficial to know the hateful content the counterhate argument is for.
The results at the
We obtain the best results (F1: 0.74 vs.0.41) ag- gregating the paragraph-level predictions. In this setup, we output yes at the article level if any of
The paragraph alone obtains much better results (0.53), signaling that counterhate arguments are

---

P
R
F1
Error Type
%
Paragraph-Level Predictions
Undesirable counterhate
33
Random baseline
0.03
0.53
0.06
Positive but not countering hateful claims Unsupported counterhate statement
15
RoBERTa trained with..
10
paragraph
0.44
0.30
0.36
Ambiguous, mix of hate and counterhate
8
+ article title
0.42
0.36
0.39
Intricate text
32
hateful tweet
0.49
0.32
0.39
Implicit counterhate argument Mix of named entities
14
+ article title
0.33
0.47
0.39
hateful segment
0.45
0.38
0.41
article title
0.41
0.43
0.42
General world knowledge
11
pretraining
0.43
0.41
0.42
Rhetorical question
5
Table 6: Results for the yes label testing with candidate counterhate arguments for hateful tweets toward public figures not seen during training. We present results with the best performing systems from Table 5. Results much lower (FI: 0.72 vs. 0.42), showing that the task is challenging for unseen individuals.
Table 7: Error types made by the best performing model (RoBERTa trained with paragraph+ hateful segmen! article title and pretrained with related tasks, Table 5), provide the percentages each error type.
Hateful tweet: Messi is racist!!! [...] Paragraph: Messi is one of the best soccer players [...] Messi's goals makes him the high- est goalscorer. Gold: no, Pred.: yes.
the paragraphs are predicted yes by the paragraph- level classifier. These result lead to the conclusion that the Longformer faces challenges finding coun- terhate paragraphs in long articles, as many articles have many paragraphs (over 21 on average) and only few, if any, contain authentic counterhate.
Unsupported counterhate that not a counter- hate argument according to our definition:
Hateful tweet: Pelosi is an EVIL B**C!!!!! Paragraph: She isn T. Gold: no, Pred. yes.
5.2 Is it Harder to Find Counterhate Arguments for Unseen Individuals?
Mix of hate and counterhate arguments:
Hateful tweet: Demi Lovato is fat b**ch and hate her like there is no tomorrow.
Yes, is (Table 6). The results in terms of relative improvements depending on the combination of inputs show similar trends, but all of them are sub- stantially lower (best Fl: 0.42 vs. 0.72). Despite making sure to have unique hateful segments in our corpus (Section 3), these results show that learning to identify authentic counterhate arguments for new individual is challenging. We hypothesize that the reason is that the same counterhate argument can be reused counter against several hateful claims toward the same individual.
Paragraph: I think at certain times in her life she has been very attractive [...]. But then there have been times when she has been somewhat over weight. Gold: no, Pred.: yes. The second most common error types fall under intricate text (32%):
Implicit counterhate arguments:
Hatcful tweet: Reminder that Mel Gibson is a Jew-hating racist skid-mark on [...]
Paragraph: Mel has worked in Hollywood for years and Hollywood has a prominent Jewish community. If remember correctly Mel also had Jewish friends that defended him after the drunk driving incident. Gold: yes, Pred.: no Mix of named entities:
5.3 Qualitative Analysis
While our best model obtains somewhat high re- sults at the paragraph level (F1: 0.72; first block in Table 5), we conduct a qualitative analysis to de- scribe when it makes the most errors. Table lists the error types and their frequency in random sam- ple of 100 errors made by the best model. These error types overlap (i.e., a paragraph may contain many named entities and rhetorical question).
Hateful tweet: This why dislike Bush and lost a lot of respect for the man.
Paragraph: No. Ronald Reagan was the best President [...] I cannot comment on any presi- dents prior to 1970. Gold: no, Pred.: yes.
The most common error types (33%) fall under undesirable counterhate argument including:
Lastly, identify two additional error types: Counterhate that requires world knowledge:
Positive comments about the individual not countering the hateful claims:
Hateful tweet: Bill Gates is an evil [...]
PPaagraph: people don't 't know facts about

---

bill gates, and are instead believing in false conspiracy theories. Gold: yes, Pred.: no,
References
Ron Artstein and Massimo Poesio. 2008. Survey article: Inter- agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
Rhetorical questions:
Hateful tweet: Ronaldo is wayyyyy better
[..] monkey Messi is selfish and a dickhead. Paragraph: How selfish of him to donate to people who need it! Gold: yes, Pred.: no.
Brooke Auxier and Monica Anderson. 2021. Social
media use in 2021.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2015 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
6 Conclusions
Countering hate is effective at minimizing the spread of hateful content (Gagliardone et al., 2015). Unlike blocking content or banning users, counter- ing hate does not interfere with freedom of speech. Previous work (a) works with hateful content to- ward groups and (b) focuses generic, synthetic counter arguments written on demand by experts or the crowd. In this paper, we focus on authen- tie counterhate arguments against hateful claims toward individuals. Authentic arguments, unlike generic ones, address specific hateful claims and include factual, testimonial, or statistical evidence.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprim: 25X5v:2004.05150
Mary Blankenship and Carol Graham. 2020. How mis- information spreads on twitter.
Yi-Ling Chung. Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN COunter NArratives through nichesourcing: a mul- tilingual dataset of responses fight online hate speech. In Proceedings of the 57th Anmual Meet- ing of the Association for Compulational Linguistics, pages 2819-2829, Florence, Italy. Association for Computational Linguistics.
We present a collection of 250 hateful tweets toward 50 individuals and candidate counterhate ar- guments. Candidates come from 54,816 16 paragraphs from 2,500 online articles. Our annotation effort shows that authentic counterhate arguments rare (4.3%). Experimental results show promising classification results. It is more challenging. how- ever, to identify authentic counterhate arguments for a hateful claim toward an unseen individual.
Yi-Ling Chung, Serra Sinem Tekiroğlu, and Marco Guerini. 2021. Towards mhowledge-groundde counter narrative generation for hate speech. In Find- ings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 899-9144 Online. Associa- tion for Computational Linguistics.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detec- tion and the problem of offensive language. Proceed- ings of the International AAAI Conference on Web and Social Media, 11(1).
Limitations
The work presented here has several limitations. First, our retrieval approach assumes that an online article with an authentic counterhate argument for a given hateful tweet is available. Empirically, we found this is the case for 72% of hateful tweets despite we limit our retrieval to 10 online articles.
Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Gr- bovic. Vladan Radosavijevic, and Narayan Bhamidi- pati. 2015. Hate speech detection with comment em- beddings. In Proceedings of the 24th International Conference on World Wide Web, WWW '15 Compan- ion, page 29-30, New York, NY, USA. Association for Computing Machinery.
That said, there is no guarantee that the approach would work for a brand new tweet.
Another limitation of this study is the restrictions imposed on hateful tweets. We use patterns and classifiers to identify hateful tweets and segments. Neither one is 100% perfect and may introduce bi- ases. Lastly, the results with hateful tweets toward unseen individuals during training show a large drop in results. More robust models are needed in this scenario.
Maeve Duggan. 2020. Online harassment 2017.
Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroğlu, and Marco Guerini. 2021. Human-in-ther loop for data collection: a multi-target counter narra- tive dataset to fight online hate speech. In Proceed- ings the 59th Annual Meeting the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 9226-3240, Online. Association for Computational Linguistics.

---

Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput, Sure. (4)..
shalt not hate: Countering online hate speech. Pro- ceedings of the International AAAI Conference on Web and Social Media, (3(01))369-333
W. N. Francis and H. Kucera. 1979. Brown corpus manual. Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. Proceedings of the AAAI Conference on Artificial Intelligence, 35((7):14867-1477
Iginio Gagliardone, Danit Gal, Thiago Alves, and Gabriela Martinez. 2015. Countering online hate speech. Unesco Publishing.
Saif M Mohammad and Peter D Turney. 2013. Crowd- sourcing a word-emotion association lexicon. Com- putational intelligence, 29(3):43-446..
Joshua Garland, Keyan Ghazi-Zahedi, Jean-Gabriel Young, Laurent Clebert-Dufresne and Mirta Galesic. 2020. Countering hate on social media: Large scale classification of hate and counter speech. In Pro- ceedings of the Fourth Workshop on Online Abuse and Harms, pages 102-112, Online. Association for Computational Linguistics.
Roser Morante and Eduardo Blanco. 2012. *SEM 2012 shared task: Resolving the scope and focus of nega- tion. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics- - Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth Interna- tional Workshop on Semantic Evaluation (SemEvvl 2012), pages 265-274, Montréal, Canada. Associa- tion for Computational Linguistics.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Ste- Fanie Tellex. 2019. Openwebtext corpus.
Tvan Habernal and Tryna Gurevych. 2016. What makes convincing argument? empirical analysis and de- tecting attributes of convincingness in web argumen- tation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing. pages 1214-1223 Austin, Texas. Association for Computational Linguistics.
Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia Schwarz. 2012. Tweeting is believing? understanding microblog credibility per- ceptions. In Proceedings of the ACM 2012 confer- ence on computer supported cooperative work, pages 441-450.
Bing He, Caleb Ziems, Sandeep Soni, Naren Ramakr- ishnan, Diyi Yang, and Srijan Kumar. 2021. Racism a virus: anti-asian hate and counterspeech in social media during the covid-19 crisis. In Proceedings of the 2021 IEFF/ACM International Conference on Advances in Social Networks Analysis and Mining. pages 90-94.
Sebastian Nagel. 2016.
Co-news. URL: hupp//web. archive. org/auve/att///commmmeeww
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing, In Pro- ceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyber- bullying (TRAC-2018), pages 1-11, Santa Fe, New Mexico, USA. Association for Computational Lin- guistics.
Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language de- tection in online user content. In Proceedings of the 25th International Conference on World Wide Web. WWW 11, page 145-153, Republic and Canton of Geneva, CHE. International World Wide Web Con- ferences Steering Committee.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1699..
John ODonovan, Byungkyu Kang. Greg Meyer, Tobias Höllerer, and Sibel Adali.. 2012. Credibility in con- text: An analysis of feature distributions in twitter. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 293-301.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.
Binny Mathew, Navish Kumar, Pawan Goyal, and Ani- mesh Mukherjee. 2020. Interaction dynamics be- tween hate and counter users twitter. In Proceed- ings of the 7th ACM IKDD CoDS and 25th COMAD, CoDS COMAD 2020, page l 16-114, New York, NY, USA. Association for Computing Machinery.
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang. Yangqiu Song, and Dit- Yeung. 2019. Multi- lingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 4675- 4684, Hong Kong, China. Association for Computa- tional Linguistics.
Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary Veiito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang- Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high- seeh-peformance deep learning library. In Advances in Neurul Information Process- ing Systems 32. pages 8024-8035. Curran Associates, Inc.
William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Me- dia, pages 19-26, Montréal, Canada. Association for Computational Linguistics.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88-93, San Diego, California. Association for Computational Linguis- tics.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. benchmark dataset for learning to intervene in online hate speech. In Empirical Proceedings of the 2019 Conference
Valerie Wirtschafter: 2021. How george floyd changed the online conversation around blm.
Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing GGMNLP-IJCNLP" pages 4755- 4764, Hong Kong, China. Association for Computa- tional Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguisties.
Joshua Roesslein. 2020. Tweepy: Twitter for python. URL: https://githu.. com/tweepy/vveepy. 957:84.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017.
SemEval-2217 task 4: Sentiment analysis Twitter. In Proceedings of the 11th International Workshop on Semantie Evaluation (SemEval-20117 pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguistics.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1415-1420, Minncapolis, Minnesota. Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Mar- tin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 599-605, Melbourne, Australia. Association for Computational Linguistics.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and cat- egorizing offensive language in social media (Or- fensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75-86, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Leandro Silva, Mainack Mondal, Denzil Correa, Fabrí- cio Benevenuto, and Ingmar Weher. 2016. Analyzing the targets of hate in online social media. In Tenth international AAAI conference on web and social media.
Today. 2020. Japan man probed over insults tied to netflix star's suicide.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEÉ International Con- ference on Computer Vision (ICCV), pages 19-27.
Trieu H Trinh and Quoc V Le. 2018 A simple method for commonsense reasoning, arXiv preprint arXiv:7806.02847.
Michiko Ueda, Kota Mori, Tetsuya Matsubayashi, and Yasuyuki Sawada. 2017. Tweeting celebrity suicides: Users' reaction to prominent suicide deaths on twitter and subsequent increases in actual suicides. Social Science & Medicine, 189:158-166
A Individuals
Table 8 presents the 50 individuals used in our study. The table shows their names, professions, and genders. The 50 individuals cover various pro- fessions such as models, politicians, actors, en- trepreneurs, hosts, journalits comedians, singers, and athletes.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection En- glish tweets. In Proceedings of The 12th Interna- tional Workshop Semantic Evaluation, pages 39- 50, New Orleans, Louisiana. Association for Compu- tational Linguistics.

---

Individual
Profession
Gender
Individual
Profession
Gender
Ellen Degeneres
Host
Female
Sean Hannity
Host
Male
3
Nancy Grace
Host
Female
Jay Leno
Host
Male
5
Tucker Carlson
Host
Male
Chrissy Teigen
Model
Female
7
Kim Kardashian
Model
Female
Bill Gates
Entrepreneur
Male
Bill Cosby
Comedian
Male
10
Piers Morgan
Journalist
Male
11
Jimmy Carter
Politician
Male
12
Newt Gingrich
Politician
Male
13
Justin Trudeau
Politician
Male
14
Chris Christic
Politician
Male
15
Nancy Pelosi
Politician
Female
David Duke
Politician
Male
17
Joe Biden
Politician
Male
18
Donald Trump
Politician
Male
19
George Bush
Politician
Male
20
Dick Cheney
Politician
Male
21
Boris Johnson
Politician
Male
22
Hillary Clinton
Politician
Female
23
Barack Ohama
Politician
Male
24
Bill Clinton
Politician
Male
25
Mitch McConnell
Politician
Male
26
Ann Coulter
Actor
Female
27
Mel Gibson
Actor
Male
28
Lindsay Lohan
Actor
Female
29
Meghan Markle
Actor
Female
30
Angelina Jolie
Actor
Female
31
Will Smith
Actor
Male
Demi Lovato
Actor
Female
33
Amber Heard
Actor
Female
34
Sandra Bullock
Actor
Female
35
Justin Bieher
Singer
Male
36
Nicki Minaj
Singer
Female
37
Jennifer Lopez
Singer
Female
38
Madonna
Singer
Female
39
Chris Brown
Singer
Male
40
Beyonce
Singer
Female
41
Britney Spears
Singer
Female
42
Avril Lavigne
Singer
Female
43
Kanye West
Singer
Male
44
Anne Hathaway
Singer
Female
45
John Mayer
Singer
Male
46
LeBron James
Athlete
Male
47
Floyd Mayweather
Athlete
Male
48
Lionel Messi
Athlete
Male
49
Neymar
Athlete
Male
50
Cristiano Ronaldo
Athlete
Male
Table 8: List of the 50 individuals we work with.
B Part-of-Speech Patterns
pre-processed data is then fed to RoBERTa and Longformer models where RoBERTa and Long- former tokenizers were used respectively tok- enize tweets and to obtain the ids and attention masks. RoBERTa model was used at the paragraph level while the Longformer model was used at the article level. All models were trained using learn- ing rate of 1e-5, a batch size of 16, an Adam W optimizer (Loshchilov and Hutter, 2017), and sparse categorical cross-entropy loss function. The RoBERTa model used a maximum sequence length of 256 tokens (padding shorter sequences), whereas the Longformer model used a maximum sequence length of 2048 tokens, as articles are much longer than paragraphs. All models were trained for 6 epochs while saving a checkpoint of the model parameters after the epoch in which the model achieved the lowest validation loss.
Table shows the patterns grounded on part-of- speech tags. These patterns are used identify the hate segments toward individuals (Section 3.1 1.1 in the main paper). Patterns 1 1-21 consist of the name of the individual followed by is and a noun or an adjective phrase. The second kind of patterns, 22 and 23, was inspired by Silva et al. (2016).
C Inter-Feature Correlations
Figures and present the linguistic features used in Section of the main paper. We present corre- lations the paragraph and article levels. Most correlation coefficients are low, meaning that our linguistic features capture different kinds of lan- guage.
D Implementation Details
We used the Python's Transformers library to load the base RoBERTa (Liu al., 2019) and Long- former (Beltagy al., 2020) models. Our dataset was pre-processed by removing URLs, removing symbols, removing any additional spaces, and at the end, converting all words to lower-case. The
E Detailed Results
Tables 10 and 11 show the detailed results comple- menting Tables 5 and the paper. We provide Precision, Recall, and F1-measure for the yes and no labels for the paragraph and article levels using

---

#
Pattern
# Pattern
findividual's name, is. 'J'']
[individual's name, is. 'JJ', 'NN']
3
[individual's name, is, 'NN']
4
[individual's name, is, 'NN', 'NN'] findividual's name, is, 'DT', 'JJ", 'NN'] [individual's name, is, 'DT", 'JJS"] [individual's name, is, 'DT', 'NN']
5
findividual's name, is. 'DT', 'J3"]
6
[individual's name, is, 'DT", 'JJ', 'NN', 'NN']
9
findividual's name, is, 'DT', 'JJS', 'NN']
10
11
findividual's name, is, 'DT', 'NN', 'NN'] (individual's name, is. 'DT", 'NNP"] jindividual's name, is, 'DT', "VBG", 'NN] findividual's name, is. 'DT', 'RBS', 'VBN'
12
findividual's name, is, 'DT', 'NN', 'IN', 'NN' [individual's name, is, 'DT", 'NNP", 'VBG') [individual's name, is, 'DT', 'RBS', 'D1 [individual's name, is, 'PDT" 'DT", 'J] [individual's name, is, 'PDT', 'DT', 'NN']
15
16
17
18
19
lindividual's name, is, 'PDT", 'DT", 'J", 'NN']
20
21
jindividual's name, is, 'RB', 'JJ']
22
[I. <hateful verb>, individual's name]
RB, <hateful verb>, individual's name]
RB,
Table 9: Lists of sequences of part-of-speech tags used to identify hate segments. DT: Determiner, NN: Singular noun, JJ: Adjective, JJS: Superlative Adjective, RB: Adverb, RBS: Superlative Adverb, VBG: Present participle verb, VBN: Past participle verb, PRP: Pronoun, and PDT: Predeterminer.
Takens
0:49
0:32
0:09
Takkns
0.03
0:23
Risk purson pronoure
0.49
6:32
8:1
6.2
0.04
0.03
0.78 0.34
0.71
0.1
First person pronoure
04
0:49
0:32
0.18
5:74
0.11
6:83
0.08
0:14
0.01
6.1
person pronouns
0.31
0.85
0.17
0:08
Second person pronouns
0:49
0.27
0:30
0.92
Propert
0.37
0:35
0.29
0.03
0.03
0:73 0.66
4.22
mouns
0.75
0.22
0.43
0.2
0:33
0.0
0.03 D.04
Negation cues
0:46
0.3
0:27
0:12
5.09
0.06 0.26
0.29
0.32
00
0.12
Negation cues
0:7
0.56
0.23
0.32
0.49
0.58
0.59
0.11
Profanity words
0.14
0.95
0.b0
C.94
5.4
0.02
Profonity words
0.4
0.14
0.1
0:44
sapellings
0:72
5.28
0:37
0:72
0.08
3.1
0.02
Misspallins
0.88
0.41
0.23
0:1
0:70
0.02
Pesttvv 0.77
0.34
0:22
0.60
0.08
0.69
0.1
0:91
0:49
0.23
0.39
0.08
0:24
Negatie wards
0:02
2.75
8:38
03:31
3:17
0:04 0.38
0.05 0.01
4018
0.00
Megation words
0.84
0.44
0.34
0.67
0.15
2:71 0.22
0.24
0.70
.2.03
8.15
Polarity score
6.52
3.03
Poleniy 0:83
0.0L 0.05
Suljectivig 0.09
01
6:38
4.00
c:01
0.01
b.03
0.02
0.25
Subjectiviy
0:3
3.1
2.12
0.04 0.0
0.11
0.15
0.13
Tollens
Fist person pronoums
Socond porson pronouns
Duesston mark
hotterity wonts
repelling
Negattv words
Ph lrryy score
Sublectrviy score
Tollons
peron pronoums
Second pronouns
Proper menn
Negetton cues
Duesston mark
Irotenity words
Mospelligg
Poalone words
raggttiv words
Poarty score
score
Hegelton
Posit
Figure Correlation coefficients between features used the linguistic analysis. The heatmap shows the
correlations with paragraphs that counterhate (left) and counterhate paragraphs (right).

---

first person pronours
0.74
0:77
0.52
2.72
5.93
0.32
39
0.71
0.33
01
4.39
First person prenours
6:34
0.3
6.7
0.83
0.30
0.22
person pronours
0:5
0.61
6.87
0.41
503
Second person pienours
066
0.77
0.37
0.61
0.63
0.4
0.40
2.00
Proper
0.84
0:51
0:44
5.57
0.29
0.52
0:54
0.53
0:33
63
Nogation cues
0.86
0.72
0.30
0.6
6.03
0:45
000
0.74
16
4.03
Negation cums
0:54
0:71
0:65
0.61
0.6
0.3
0.67
0.78
0.0
5:41
Queston
062
040
0.49
0330
0LVL
Profanity words
0.48
0.32
0:41
6.11
0:41
Profenity words
6:26
0.36
0.28
1 espellings 0.92
0.59
0:50
5.97
0.43
1
0.87 0.85
0.14 6:16
Missaalings
6:92
0.65
0:48
0.91
0:67
0.99
6.25
6.9
0.85
0.02
4.02
Negadive words
0.93
0.59
0:52
0.73
3.73
0.99
1.5
0.69 0.60
0.29
4.12
Megative werds
0:83
0.64
0.77
0.8
0:57
0.29
0.85
4.3
0.04
Polarity
0.1 5..4
0.36
Polarity 4.00
0.03
2.09
Subjectivit 4.50
5,41
4.16
0.36
Subjectivity
4.44 0.c6
0.03
0.05
0.06
4.64
Tokens
Frrst persen promoune
Secord persen prorouns
Question mark
Frotenty worrds
Neopatiing
Neqaaivve worrds
Polenty score
Subjectisty score
Tokents
Hrople nou-
Nagution curs
Quend mark
Peditive words
Neqalive words
Poluiity score
Sunjectialy scae
begaitan
First fistppeson
Second
Figure Correlation coefficients between features used the linguistic analysis. The heatmap shows the
correlations with articles that counterhate (left) and counterhate articles (right).
RoBERTa and Longformer.
The top block of table 10 provides additional results for pretraining RoBERTa with more existing corpora:
hate: tweets containing and not containing
hateful content (Basile al., 2019).
sentiment: tweets that are positive, neutral, or
negative (Rosenthal et al., 2017).
irony: irony and not irony tweets (Van Hee et al., 2018).
offensive: offensive or not offensive tweets (Zampieri et al., 2019b).
CONAN and Multitarget-CONNa were dis- cussed in details in Section 5.1.

---

Yes
No
Weighted Avg.
P
F
F
F
Paragraph-Level Predictions
Majority baseline
0.00
0.00
0.00
0.96
1.00
0.98
0.92
0.96
0.94
Random haseline
0.04
0.50
0.08
0.96
0.50
0.66
0.92
0.50
0.63
RoBERTa trained with
article title
0.54
0.19
0.28
0.96
0.99
0.98
0.95
0.96
0.95
hateful tweet
0.00
0.00
0.00
0.96
1.00
0.98
0.92
0.96
0.94
+ article title
0.59
0.22
0.32
0.97
0.99
0.98
0.95
0.96
0.95
hateful segment
0.44
0.06
0.10
0.96
1.00
0.98
0.94
0.96
0.94
+ article title
0.61
0.21
0.32
0.97
0.99
0.98
0.95
0.96
0.95
paragraph
0.62
0.46
0.53
0.98
0.99
0,98
0.96
0.96
0.96
+ article title
0.57
0.76
0.65
0.99
0.97
0.98
0.97
0.96
0.97
+ hateful tweet
0.65
0.76
0.70
0.99
0.98
0.99
0.97
0.97
0.97
+ article title
0.64
0.77
0.70
0.99
0.98
0.99
0.97
0.97
0.97
+ hateful segment
0.68
0.75
0.71
0.99
0.98
0.99
0.97
0.97
0.97
+ article title
0.67
0.76
0.71
0.99
0.98
0.99
0.97
0.97
0.97
pretrained with
Hate
0.62
0.78
0.69
0.99
0.98
0.98
0.97
0.97
0.97
Irony
0.61
0.78
0.69
0.99
0.98
0.98
0.97
0.97
0.97
Offensive
0.67
0.73
0.70
0.99
0.98
0.99
0.97
0.97
0.97
Sentiment
0.64
0.78
0.70
0.99
0.98
0.99
0.97
0.97
0.97
CONAN and
Multitarget.CONNN
0.69
0.75
0.72
0.99
0.99
0.99
0.98
0.97
0.98
Article-Level Predictions
Majority baseline
0.00
0.00
0.00
0.78
1.00
0.88
0.61
0.78
0.69
Random baseline
0.21
0.50
0.30
0.77
0.48
0.60
0.65
0.49
0.53
LongFormer trained with...
article
0.70
0.24
0.36
0.82
0.97
0.89
0.79
0.81
0.77
+ hateful tweet
0.62
0.30
0.41
0.83
0.95
0.89
0.78
0.81
0.78
+ hateful segment Aggregate. lerelevel preds.
0.67
0.29
0.41
0.83
0.96
0.89
0.79
0.81
0.79
0.65
0.85
0.74
0.96
0.87
0.91
0.89
0.87
0.87
Table 10: Detailed results (P, and F) for RoBERTa and Longformer predict whether paragraph and article is counterhate. The results complement Table 5 in the main paper.
Yes
No
Weighted Avg.
P
R
F
P
R
F
P
F
Paragraph-Leve Predictions
Random baseline
0.03
0.53
0.06
0.97
0.50
0.66
0.95
0.50
0.64
RoBERTa trained with
paragraph
0.44
0.30
0.36
0.98
0.99
0.99
0.97
0.97
0.97
article title
0.42
0.36
0.39
0.98
0.99
0.98
0.97
0.97
0.97
hateful tweet
0.49
0.32
0.39
0.98
0.99
0.99
0.97
0.97
0.97
article title
0.33
0.47
0.39
0.98
0.97
0.98
0.97
0.96
0.96
hateful segment
0.45
0.38
0.41
0.98
0.99
0.98
0.97
0.97
0.97
article title
0.41
0.43
0.42
0.98
0.98
0.98
0.97
0.97
0.97
pretraining
0.43
0.41
0.42
0.98
0.98
0.98
0.97
0.97
0.97
Table 11: Detailed results (P, R. and F) for RoBERTa whether paragraph is counterhate for unseen individuals. The results complement Table 6 in the main paper.


Reviews:
[{"review_id": "JmBOYIGNiJ", "reviewer": "Reviewer_bvb9", "summary": "", "questions": "- A: L201 -- how many tweets are retrieved each time? If all tweets targeting an individual are retrieved the first time, how can this process be repeated (as stated on L246)?\n- B: do you validate the hate annotations for the non-discarded tweets in 3.1?\n- C: L336 -- Were there any individuals without any counterhate arguments? Were there specific professions that had the most countering?\n- D: L458 -- does this mean classifying a paragraph as counter hate in a tweet without seeing either the tweet or the paragraph? \n- E: L479 -- did you experiment with pretraining for the hateful tweet + article title combination? If not, why? It seems odd to only have pretraining on the last combination. Do you think the lack of benefit from pretraining could be in part due to the differences in data because the new data has specific patterns of hate which may not be in the other datasets?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons to Reject**\n\nRE Although the authors acknowledge this limitation, they should include some analyses of potential biases or more discussion of this.\n- We will expand on the discussion in the extra page if accepted. Thank you for the suggestions\n\nRE The claim that previous work has only considered generic denouncing arguments or synthetic data is not correct. See note in missing references.\n- Thank you, we will rephrase and include the additional references.\n\n**Questions for The Authors**\n\n- Answer to Q(A):\nIn each iteration of the collection process, we retrieve a batch of tweets per individual (tweets containing the name of any of the 50 individuals) until we collect 5 tweets that contain unique hateful segments toward each individual (using the HateXplain model). Once we have 5 unique hateful segments toward an individual, we remove the individual from the collection process and continue repeating these steps for the remaining individuals. We will clarify: the process is repeated for each individual.\n\n- Answer to Q(B):\nYes, we manually validated that 1) the tweets are actually hateful, and 2) we have 5 unique hateful segments for each individual.\n\n- Answer to Q(C):\nSome hateful tweets do not have any counterhate arguments; however, each individual has many counterhate arguments (each individual has 5 hateful tweets).\n  - The lowest number of counterhate arguments per individual is 2\n  - The highest number of counterhate arguments per individual is 264\n  - The average number of counterhate arguments per individual is 47.4 \n\n  And regarding the number of counterhate arguments per profession, we found the number of counterhate arguments is low for some professions like journalists and hosts while high for professions like politicians and athletes. It is interesting to include this kind of analysis in the final version of the paper. Thank you.\n\n- Answer to Q(D):\nYes. Let us clarify that this is (probably) the simplest baseline. We include it to show that the problem cannot be reduced to \"look at the title and then just tell me whether the article has a counterhate argument.\n\n- Answer to Q(E):\nNo, we found that (hateful tweet + article) yields a lower F1-score compared to (hateful tweet + segment), so we decided to do the pretraining with this combination (hateful tweet + segment); additionally, the combination (hateful tweet + segment) obtains the best results compared to all other inputs.\nRegarding the second part of the question, “Do you think the lack of…..“ we believe this is due to the fact that the two datasets that we do the pretraining with (CONAN and Multitarget-CONAN) are towards groups instead of individuals, and their counterhate arguments are synthetic as stated in L498-L504.\n\n**Grammar / Typos**\n\nThank you. We will fix the grammar/typos issues and include the ethics section if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset contains good annotations and will likely be useful for other researchers on identifying and also generating or analyzing counterspeech.\n- The focus on individuals is novel and an important area for counterspeech.", "reasons_to_reject": "- The data was constructed with only specific patterns of hate, which may introduce biases. Although the authors acknowledge this limitation, they should include some analyses of potential biases or more discussion of this.\n- The claim that previous work has only considered generic denouncing arguments or synthetic data is not correct. See note in missing references.", "questions_to_authors": "- A: L201 -- how many tweets are retrieved each time? If all tweets targeting an individual are retrieved the first time, how can this process be repeated (as stated on L246)?\n- B: do you validate the hate annotations for the non-discarded tweets in 3.1?\n- C: L336 -- Were there any individuals without any counterhate arguments? Were there specific professions that had the most countering?\n- D: L458 -- does this mean classifying a paragraph as counter hate in a tweet without seeing either the tweet or the paragraph? \n- E: L479 -- did you experiment with pretraining for the hateful tweet + article title combination? If not, why? It seems odd to only have pretraining on the last combination. Do you think the lack of benefit from pretraining could be in part due to the differences in data because the new data has specific patterns of hate which may not be in the other datasets?\n", "paper_topic_and_main_contributions": "The authors construct a corpus of hateful tweets towards individual public figures with related articles. The articles are labeled at the paragraph and article level for whether they contain substantiated (not unsupported) counterhate arguments for a specific hateful segment in the tweet. The tweets are collected by identifying specific syntactic patterns of hate. The accompanying articles are taken from a google search with emphasis placed on responses from Quora. A subset of the paragraphs from articles are labeled by two annotators with strong agreement and the remaining are labeled by a single annotator. The authors conduct an analysis of the language in countering hate and also train classifiers for the task of predicting whether a paragraph or argument is countering hate in a tweet. The results show that the task is quite hard, especially for individuals not seen during training. \n\nMain contributions: new data resource", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- The claim on L067 that previous work only has generic condemnations isn't true. Both Chung et al., 2019 and Mathew et al., 2019 (already cited) include analysis of different hate countering strategies. Furthermore, Allaway et al., 2022 (https://arxiv.org/pdf/2303.16173.pdf) construct counterhate statements that also target specific components of the hatespeech. This should be included in the references. \n- There are also non-synthetic arguments considered in Mathew et al., 2019, as well as the works cited on L139-147 (in fact this paragraph describes how they are not synthetic). The task as presented in the paper is very much a classification task, so it does not make sense to only consider generation approaches (which are generating synthetic data) as comparisons for that.", "Typos_Grammar_Style_and_Presentation_Improvements": "- The authors should include an ethics section discussing ethical issues related to studying hate specifically targeting individuals. In particular, the data could be used to train a model to produce hate targeted at individuals.\n- L099 (\"It\") and L112 (\"task\"): the task is really clearly defined before the first reference to it, and although it can be inferred from the rest of the paper, there really should be a clear definition in the introduction. \n- L069: \"result generic\" --> \"result has generic\"\n- L158: I would argue that statements constructed by experts are not synthetic because they are human written. \n- L225: \"come from\" --> \"coming from\"\n- L227: \"segment is hateful\" --> \"segments are hateful\"\n- L249-253: it is not clear whether this sentence means the tweet has to contain both types of segments or just one. From the example on 261 it seems like just one, but this should be clarified.\n- L329: \"despite we designed\" --> \"despite having designed\"\n- L337: \"we limit ourselves\" --> \"limiting ourselves\"\n- L341: \"Angelina Jolie (horrible person)\" -- this phrasing makes it sound like it is your opinion that AJ is a horrible person. I think just remoe the parenthetical.\n- L354: this would be easier to read if it was a subsection instead of a paragraph heading.\n- L366: \"misspellings\" --> \"misspelled\"\n- L518: \"This results lead to\" --> \"This result leads to\"\n- L529: \"we took care\" --> \"taking care\"\n- There are weird floating line numbers on the last two pages (497 in Table 5, 563 in table 7, 612 in L562, and 529 in L579).\n"}}, {"review_id": "bLlkZMQRj3", "reviewer": "Reviewer_pTGV", "summary": "", "questions": "1. Do you have any numbers on how HateXPlain performs on the data? e.g., how many filtered tweets are hateful/not hateful toward each profession/individual?\n2. Are there statistics of counterhate arguments found/annotated (as yes) for each profession/individual? e.g., is it easier to find counterhate arguments for certain professions/individuals?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Annotation guidelines.** More details about the annotation instructions and guidelines will be added in the extra page if accepted. Thank you for pointing this out.\n\n**Answer for Q1**  \nWe do not know the numbers per profession/individual but in general, we retrieved approximately 41,000 tweets for the 50 individuals. Then, as discussed in L204, we fed these tweets to the HateXPlain classifier. We ended this process with 2,040 hateful tweets (5% of total tweets), approximately 38,960 tweets were not classified as hateful.\n\n\n**Answer for Q2**  \nYes, but it is not included in the current version of the paper. For example, it is more likely to find counterhate arguments for a politician than a journalist. We believe that such insights should and can be included in the corpus analysis section of the paper. Thank you, we will include them using the extra page if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new dataset consists of 250 hateful tweets paired with ~2400 counterhate arguments/paragraphs.\n2. Linguistic analysis of authentic counterhate characteristics and error analysis of when the classification task is hard.", "reasons_to_reject": "1. I appreciate the effort done in all the experiments on the performance of classifiers across different strategies for getting relevant counterhate arguments (section 5.1), but I don't think it adds much new knowledge to what we already knew based on findings from information retrieval. \n2. Detail about the annotation guideline is missing", "questions_to_authors": "1. Do you have any numbers on how HateXPlain performs on the data? e.g., how many filtered tweets are hateful/not hateful toward each profession/individual?\n2. Are there statistics of counterhate arguments found/annotated (as yes) for each profession/individual? e.g., is it easier to find counterhate arguments for certain professions/individuals?", "paper_topic_and_main_contributions": "This paper tackles the problem of retrieving relevant counterhate (counterspeech) arguments to counter online hateful content targeting individuals. The authors present a collection of 250 hateful tweets toward 50 individuals plus counterhate arguments retrieved from  2,500 online articles. They also tested various strategies for developing classifiers to identify potential counterhate articles.\nAs counterhate arguments are rare and existing datasets mostly are synthetic, the dataset (not provided for review) and experiments provided can help characterise counterhate in-the-wild and hate mitigation.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In the paper, it is not clear whether the annotated counterhate paragraphs are useable directly in response to hateful tweets as they are provided or not. I would suggest adding a few sentences explaining if further editing on the paragraphs is needed to be used as responses.\nAlso, the detail about the annotation guideline is very little - what is the criteria for a paragraph to be annoataed as counterhate argument? What qualifies as positive statistical evidence (line 307) - when numbers are mentioned? It is better to elaborate on it more."}}, {"review_id": "jAfQTf5zoZ", "reviewer": "Reviewer_xM6k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review.\n\n**Reasons to Reject**\n- RE Targeting celebrities:\nIt is certainly subjective, but we would argue that addressing hate speech targeting celebrities would improve online discourse in general. Doing so could benefit all participants (less hate), not just the celebrities (by minimizing potential consequences the celebrities may suffer).\n\n**RE grammar.**  We will fix the grammar/typos issues. We really appreciate the time you took to point out these typos and grammar issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well-written (no significant typos or grammar mistakes) and has a clear structure. All experiments are well-motivated. The authors are also very specific about their implementation (always mentioning the tools they used along their configuration). The experimental set-up looks mostly sound. The dataset is carefully constructed. The chosen annotators look sufficiently qualified which is also confirmed by the acceptable interannotation agreement. The classifiers that are chosen including the baselines look appropriate for these data. The classification results look fairly plausible. I also think that the results are appropriately discussed and interpreted. I particularly appreciate the brief linguistic analysis in Table 4 which helps the reader get a better idea what type of language counter hate represents. I also think that the experiments on unseen individuals (Section 5.2) is important. It shows that one only gets reasonable classification performance if one has training data for the public figure that one is going to find counter hate for.", "reasons_to_reject": "Despite the apparent technical soundness of the approach, I really wonder how pressing the given task is. Counter hate regarding identity groups and private citizens seems much more pressing to me. In other words, the negative effects on those groups caused by hate speech seem to be much more severe. It comes with the territory of being a public figure that those individuals are often subject to intensive criticism. This, of course, does not excuse in any way the use of hate speech, insults or defamation directed towards them. My impression is simply that these people are much more prepared of what they face on the web than private individuals which usually are much less resilient. I acknowledge and appreciate that the authors try to give some motivation and also cite Ueda et al. (2017). Still, I find it not that convincing that celebrity suicides also causes an increase of suicide of private individuals. (Regarding Ueda et al. (2017), I am a bit sceptical in how far these results are representative in general and particularly of other cultural areas others than the one addressed in that publication, i.e. Japan.) My pessimistic guess is that given that public figures are often significantly less protected by the law against insults than private citizens, operators of social media sites would be more reluctant to invest in software that specifically combats insults/hate speech regarding public figures.\n\nOne important insight of the paper is that the classification approach is only effective when the test data focuses on public figures that have already been observed in the training data. Given this insight, I wonder whether the proposed method could in generally scale up to a fully-fledged commercial application. Would it be worthwhile to have such a tool if one can only effectively act against hate speech directed towards people for whom one has collected specific training data?\n\nFinally, I am a bit worried that the extraction of the hate tweets and the retrieval of online articles might have restricted the scope of possible instances of hate speech and also possible instances of counter speech. Both hate tweets and online articles were retrieved using fairly simple surface-based lexical patterns. I wonder whether this approach might have biased the resulting hate tweets and online articles since the underlying query terms are likely to be lacking in lexical diversity.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the detection of counter hate arguments with a focus on insults/hate speech targeting public figures. The authors introduce a collection of 250 hateful tweets towards 50 individuals combined with 2,500 online articles (each addressing one of the 50 individuals) in which each paragraph has been labelled as being counter hate or not. The hateful tweets and the additional online articles are obtained using surface patterns and information retrieval. Apart from a brief descriptive linguistic analysis on their novel dataset, the authors present classification experiments using that dataset. They address two tasks: detection of counter hate on the paragraph level and counter hate on the article level. The classifiers the authors examine are based on finetuning transformers (RoBERTa and Longformer). For the paragraph-level classification, the authors also consider the difficult setting in which training and test data use different individuals. The resulting classifiers perform notably worse.\n\nThe paper basically makes 3 contributions: It introduces a new data resource, it carries out a computationally-aided linguistic analysis on it and presents classification results (NLP engineering experiment).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tThe authors use the word “despite” incorrectly. It is a preposition rather than a subordinate conjunction. Please write something like “Despite the fact that …”\n\n-\tLines 300-306, “The tool showed … on the annotation guidelines.”: I found this sentence hard to read, please rephrase.\n\n-\tThere is something wrong with the line indices of line 497, 612 and 717.\n"}}]

Author rebuttal:

Meta-review: All the reviewers agree on the fact that the paper provides valuable and novel contributions to research in countering hate speech messages online. A corpus of hateful tweets towards individual public figures with related articles is created, and annotated with useful annotations to identify and analyze counterspeech. The experimental set-up looks also solid and sound.
The main concerns (not answered in the rebuttal) are related to the fact that data was constructed with only specific patterns of hate, which may introduce biases, as well as missing details about the annotation instructions and guidelines. A discussion on how far the results provided in the paper are representative in general is also missing.
Decision: Accept-Main