SlowGuess/NAACL_2022_OCR huggingface dataset:
Xinchen Yu  
University of North Texas  
xinchenyu@my.unt.edu
Eduardo Blanco  
Arizona State University  
eduardo.blanco@asu.edu
Lingzi Hong  
University of North Texas  
lingzi.hong@unt.edu
Hate speech is plaguing the cyberspace along with user-generated content. This paper investigates the role of conversational context in the annotation and detection of online hate and counter speech, where context is defined as the preceding comment in a conversation thread. We created a context-aware dataset for a 3-way classification task on Reddit comments: hate speech, counter speech, or neutral. Our analyses indicate that context is critical to identify hate and counter speech: human judgments change for most comments depending on whether we show annotators the context. A linguistic analysis draws insights into the language people use to express hate and counter speech. Experimental results show that neural networks obtain significantly better results if context is taken into account. We also present qualitative error analyses shedding light into (a) when and why context is beneficial and (b) the remaining errors made by our best model when context is taken into account.
The advent of social media has democratized public discourse on an unparalleled scale. Meanwhile, it is considered a particularly conducive arena for hate speech (Caiani et al., 2021). Online hate speech is prevalent and can lead to serious consequences. At the individual level, the victims targeted by hate speech are frightened by online threats that may materialize in the real world (Olteanu et al., 2018). At the societal level, it has been reported that there is an upsurge in offline hate crimes targeting minorities (Olteanu et al., 2018; Farrell et al., 2019).
There are two common strategies to combat online hate: disruption and counter speech. Disruption refers to blocking hateful content or users. To scale this strategy, researchers have proposed methods to identify hate (Waseem and Hovy, 2016; Davidson et al., 2017; Nobata et al., 2016). While these interventions could de-escalate the impact of
Parent	As an average height male, idgaf how tall you are, if that's your issue then spend the money and get a better seat, or just f**king make the seat selection online to get more space.
Target	Found the short guy!
-Target is Neutral if considering only Target.
-Target is Hate if considering Parent and Target.
Parent	I deal with women all day with my job and this is how they are - extremely stupid, hate-filled, bizarre and they appreciate nothing.
Target	Maybe you're an a**hole if they treat you like that?
-Target is Hate if considering only Target.
-Target is Counter-hate if considering Parent and Target.
Table 1: Reddit comments (Targets) deemed to be Hate, Neutral, or Counter-hate depending on whether one takes into account the previous comment (Parent).
hate speech, they may violate online free speech (Mathew et al., 2019). Additionally, attacks at the micro-level may be ineffective as hate networks often have rapid rewiring and self-repairing mechanisms (Johnson et al., 2019). Counter speech refers to the "direct response that counters hate speech" (Mathew et al., 2019). It has been shown to be more effective in the long term than disruption in theoretical and empirical studies (Richards and Calvert, 2000; Mathew et al., 2020). Identifying hate and counter speech in natural conversations is critical to understand effective counter speech strategies and the generation of counter speech.
Most corpora with either hate speech (Hate) or counter speech (Counter-hate) annotations do not include conversational context. Indeed, they annotate a user-generated comment as Hate or Counter-hate based on the comment in isolation (Davidson et al., 2017; Waseem and Hovy, 2016; Mathew et al., 2019; He et al., 2021). Therefore, systems trained on these corpora fail to consider the effect of contextual information on the identification of Hate and Counter-hate. Recent studies have shown that context affects annotations in toxic-
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5918 - 5930 July 10-15, 2022 ©2022 Association for Computational Linguistics
ity and abuse detection (Pavlopoulos et al., 2020; Menini et al., 2021). We further investigate the effect of context on the task of identifying Hate and Counter-hate. Table 1 shows examples where a comment, denoted as Target, is Hate, Neutral or Counter-hate depending on whether the preceding comment, denoted as Parent, is taken into account. $^{1}$
In the top example, the Target goes from Neutral to Hate when taking into account the Parent: it becomes clear that the author is disparaging short men. In the bottom example, the Target goes from Hate to Counter-hate as the author uses offensive language to counter the hateful content in the Parent. This is a common strategy to express counter speech (Mathew et al., 2019).
In this study, we answer the following questions:
1. Does conversational context affect if a comment is perceived as Hate, Neutral, or Counter-hate by humans? (It does.)
2. Do models to identify Hate, Neutral, and Counter-hate benefit from incorporating context? (They do.)
To answer the first question, we create a collection of (Parent, Target) Reddit comments and annotate the Targets with three labels (Hate, Neutral, Counter-hate) in two independent phases: showing annotators (a) only the Target or (b) the Parent and the Target. We limit context to the parent comment. While the full conversation could provide additional information, it is also known to affect annotators' stance (Dutta et al., 2020) and introduce biases. We find that human judgments are substantially different when the Parent is shown. Thus the task of annotating Hate and Counter-hate requires taking into account the context.
To answer the second question, we experiment with context-unaware and context-aware classifiers to detect if a given Target is Hate, Neutral, or Counter-hate. Results show that adding context does benefit the classifiers significantly.
In summary, the main contributions of this paper are:2 (a) a corpus with 6,846 pairs of (Parent, Target) Reddit comments and annotations indicating whether the Targets are Hate, Neutral, or Counterhate; (b) annotation analysis showing that the problem requires taking into account context, as the ground truth changes; (c) corpus analysis detailing the kind of language people use to express Hate
and Counter-hate; (d) experiments showing that context-aware neural models obtain significantly better results; and (e) qualitative analysis revealing when context is beneficial and the remaining errors made by the best context-aware model.
Hate speech in user-generated content has been an active research area recently (Fortuna and Nunes, 2018). Researchers have built several datasets for hate speech detection from diverse sources such as Twitter (Waseem and Hovy, 2016; Davidson et al., 2017), Yahoo! (Nobata et al., 2016), Fox News (Gao and Huang, 2017), Gab (Mathew et al., 2021) and Reddit (Qian et al., 2019).
Compared to hate speech detection, few studies focus on detecting counter speech (Mathew et al., 2019; Garland et al., 2020; He et al., 2021). Mathew et al. (2019) collect and hand-code 6,898 counter hate comments from YouTube videos targeting Jews, Blacks and LGBT communities. Garland et al. (2020) work with German tweets and define hate and counter speech based on the communities to which the authors belong. He et al. (2021) use a collection of hate and counter hate keywords relevant to COVID-19 and create a dataset containing 359 counter hate tweets targeting Asians. Another line of research focuses on curating datasets for counter speech generation using crowdsourcing (Qian et al., 2019) or with the help of trained operators (Chung et al., 2019; Fanton et al., 2021). However, synthetic language is rarely as rich as language in the wild. Even if it were, conclusions and models from synthetic data may not transfer to the real world. In this paper, we work with user-generated content expressing hate and counter-hate rather than synthetic content.
Table 2 summarizes existing datasets for Hate and Counter-hate detection. Most of them do not include context information. In other words, the preceding comments are not provided when annotating Targets. Context does affect human judgments and has been taken into account for Hate detection (Gao and Huang, 2017; Pavlopoulos et al., 2020; Menini et al., 2021; Vidgen et al., 2021). Gao and Huang (2017) annotate hateful comments in the nested structures of Fox News discussion threads. Vidgen et al. (2021) introduce a dataset of Reddit comments with annotations in 6 categories taking into account context. However, the inter annotator agreement is low (Fleiss' Kappa 0.267) and the
1The examples in this paper contain hateful content. We cannot avoid it due to the nature of our work.
$^{2}$ Code and data available at https://github.com/xinchenyu/counter_context
Authors	Source	Size	Labels	Context?	Counter?
Waseem and Hovy (2016)	Twitter	1,607	Sexism/Racism/Normal	×	×
Davidson et al. (2017)	Twitter	24,783	Hate/Offense/Neither	×	×
Nobata et al. (2016)	Yahoo!	2,000	Hate/Derogatory/Profanity/Clean	×	×
Mathew et al. (2021)	Gab	1,1093	Hateful/Offensive/Normal	×	×
Gao and Huang (2017)	Fox News	1,528	Hateful/Non-hateful	preceding comment	×
Qian et al. (2019)	Reddit	22,324	Hate/Non-hate	full conversation	×
Pavlooulos et al. (2020)	Wikipedia	20,000	Toxic/Non-toxic	preceding comment	×
Menini et al. (2021)	Twitter	8,018	Abuse/Non-abuse	preceding comment	×
Mathew et al. (2019)	YouTube	13,924	Non-counter/Counter (6,898)	×	✓
He et al. (2021)	Twitter	2,400	Hate/Neutral/Counter (517)	×	✓
Vidgen et al. (2021)	Reddit	27,494	Taxonomy including Counter (220)	full conversation	✓
Ours	Reddit	6,846	Hate/Neutral/Counter (1,622)	preceding comment	✓
Table 2: Comparison of corpora with hate (above dashed line) and counter-hate annotations (below dashed line, some also include hate). Vidgen et al. (2021) is the only one considering counter-hate and context, but they only have 220 instances of counter hate. Numbers between parenthesis indicate the number of counter-hate instances.
number of Counter-hate instances is small (220). Moreover, both studies use contextual information without identifying the role context plays in the annotation and detection. Pavlopoulos et al. (2020) allow annotators to see one previous comment to annotate Wikipedia conversations. They find context matters in the annotation but provide no empirical evidence showing whether models to detect toxicity benefit from incorporating context. Menini et al. (2021) re-associate an existing corpus to investigate the role of context in abusive language. They found context does matter. Utilizing conversational context has also been explored in text classification tasks such as sentiment analysis (Ren et al., 2016), stance (Zubiaga et al., 2018) and sarcasm (Ghosh et al., 2020). In this paper, we investigate the role of context in Hate and Counter-hate detection.
3 Dataset Collection and Annotation
We first describe our procedure to collect (Parent, Target) pairs, where both Parents and Targets are Reddit comments in English. Then, we describe the annotation guidelines and the two annotation phases: showing annotators (a) only the Target and (b) the Parent and Target. The two independent phases allow us to quantify how often context affects the annotation of Hate and Counter-hate.
3.1 Collecting (Parent, Target) pairs
In this work, we focus on Reddit, a popular social media site. It is an ideal platform for data collection due to the large size of user populations and many diverse topics (Baumgartner et al., 2020). We use a list of hate words to retrieve Reddit conversations to keep the annotation costs reasonable while creating a (relatively) large corpus
of counter speech. We start with a set of 1,726 hate words from two lexicons: Hatebase<sup>3</sup> and a harassment corpus (Rezvan et al., 2018). We remove ambiguous words following ElSherief et al. (2018). To collect (Parent, Target) pairs, we use the following steps. First, we retrieve comments containing at least one hate word (commentw/ hateword). Second, we create a (Parent, Target) pair using commentw/ hateword as Target and its preceding comment as Parent. Third, we create a (Parent, Target) pair using commentw/ hateword as Parent and each of its replies as Target. Lastly, we remove pairs if the same author posted the Parent and Target. We retrieve 6,846 (Parent, Target) pairs with PushShift (Baumgartner et al., 2020) from 416 submissions. We also collect the title of the discussion from which each pair is retrieved.
3.2 Annotation Guidelines
To identify whether a Target is Hate, Neutral, or Counter-hate, we crowdsourced human judgments from non-experts. Our guidelines reuse the definitions of Hate by Ward (1997) and Counter-hate by Mathew et al. (2019) and Vidgen et al. (2021):
- Hate: the author attacks an individual or a group with the intention to vilify, humiliate, or incite hatred;
- Counter-hate: the author challenges, condemns the hate expressed in another comment, or calls out a comment for being hateful;
- Neutral: the author neither conveys hate nor opposes hate expressed in another comment.
Annotation Process We chose Amazon Mechanical Turk (MTurk) as the crowdsourcing platform. We replace user names with placeholders (User_A
<sup>3</sup>http://hatebase.org/
and User_B) owing to privacy concerns. The annotations took place in two independent phases. In the first phase, annotators are first shown the Parent comment. After a short delay, they click a button to show the Target and then after another short delay they submit their annotation. Delays are at most a few seconds and proportional to the length of the comments. Our rationale behind the delays is to "force" annotators to read the Parent and Target in order. In the second phase, annotators label each Target without seeing the preceding Parent comment. A total of 375 annotators were involved in the first phase and 299 in the second phase. There is no overlap between annotators thus we eliminated the possibility of biased annotators remembering the Parent in the second phase.
Annotation Quality Crowdsourcing may attract spammers (Sabou et al., 2014). For quality control, we first set a few requirements for annotators: they must be located in the US and have a  $95\%$  approval rate over at least 100 Human Intelligence Tasks (HITs). We also block annotators who submit more than 10 HITs with an average completion time below 5 seconds (half the time required in our pilot study). As the corpus contains vulgar words, we require annotators to pass the Adult Content Qualification Test. The reward per HIT is $0.05.
The second effort is to identify bad annotators and filter out their annotations until we obtain substantial inter-annotator agreement. We collect five annotations per HIT and compute MACE (Hovy et al., 2013, Multi-Annotator Competence Estimation) for each annotator. MACE is devised to rank annotators by their competence and adjudicate disagreements based on annotator competence (not the majority label). Then, we use Krippendorff's  $\alpha$  (Krippendorff, 2011) to estimate inter-annotator agreement:  $\alpha$  coefficients at or above 0.6 are considered substantial (above 0.8 are considered nearly perfect) (Artstein and Poesio, 2008). We repeat the following steps until  $\alpha \geq 0.6$ :
1. Use MACE to calculate the competence score of all annotators.
2. Discard all the annotations by the annotator with the lowest MACE score.
3. Check Krippendorff's  $\alpha$  on the remaining annotations. Go to (1) if  $\alpha < 0.6$ .
The final corpus consists of 6,846 (Parent, Target) pairs and a label assigned to each Target (Hate, Counter-hate, or Neutral). The ground truth we experiment with (Section 5) is the label obtained
Without Parent
Hate	Counter-hate	Neutral
With	Hate	57.4	8.4	34.2
Counter-hate	18.7	26.2	55.1
Neutral	9.7	8.1	82.2
Table 3: Confusion matrix (percentages) showing annotation changes depending on whether annotators are shown the Parent of the Target comment.
Example	With	Without
Parent: That chick needs a high-five in the face with a chair. Damn her for making us look bad!
Target: A brick is more effective.	Hate	Neutral
Parent: If I knew her I would sh*t in her mailbox.
Target: The poor mail carrier in that neighborhood doesn't deserve that.	Counter	Neutral
Parent: Go watch your incest porn on your own time.
Target: You're a sick person.	Counter	Hate
Table 4: Examples of Target comments whose labels change depending on whether annotators are shown the Parent of the Target comment (with and without).
taking into account the Parent (first phase). The second phase, which disregards the Parent, was conducted for analysis purposes (Section 4). We split the corpus into two subsets: (a) Gold (4,751 pairs with  $\alpha \geq 0.6$ ) and (b) Silver (2,095 remaining pairs). As we shall see, the Silver pairs are useful to learn models.
Does conversational context affect if a comment is perceived as Hate or Counter-hate? Yes, it does. Table 3 presents the percentage of labels that change and remain the same depending on whether annotators are shown the Parent, i.e., the context. Many Target comments that are perceived as Hate or Counter-hate become Neutral (34.2% and 55.1% respectively) when the Parent is provided. More surprisingly, many Target comments are perceived with the opposite label (from Hate to Counter-hate (8.4%) or from Counter-hate to Hate (18.7%)) when the Parent comments are shown.
We show examples of label changes in Table 4. In the first example, annotators identify the Target ("A brick is more effective.") as Neutral without seeing the Parent. In fact, a female is the target of hate in the Parent, and the author of Target replies
Title	Parent	Target
p-value	Bonferroni	p-value	Bonferroni	p-value	Bonferroni
Textual factors
Total tokens	↓↓	×	↑↑↑	✓
Question marks	↑↑↑	✓
1st person pronouns	↓↓↓	✓
2nd person pronouns	↑↑↑	✓	↑↑	×
Sentiment and cognitive factors
Prophanity words	↑↑↑	✓	↓↓↓	✓
Problem-solving words	↑↑↑	✓
Awareness words	↑↑↑	✓
Negative words	↓	×	↑↑↑	✓	↓↓↓	✓
Disgust words	↓↓↓	✓
Enlightenment words	↑↑↑	✓
Conflicting words	↓↓↓	✓
Table 5: Linguistic analysis comparing the Titles, Parents and Targets in Counter-hate and Hate Target comments. Number of arrows indicate the p-value (t-test; one:  $\mathrm{p} < {0.05}$  ,two:  $\mathrm{p} < {0.01}$  ,and three:  $\mathrm{p} < {0.001}$  ). Arrow direction indicates whether higher values correlate with Counter-hate (up) or Hate (down). A check mark (✓) indicates that the test passes the Bonferroni correction.
Figure 1: Label distribution in Targets depending on whether commentw/ hateword is the Parent or the Target.
with even more hatred (and the ground truth label is Hate). In the second example, the Target alone is insufficient to tell if it is Counter-hate. When annotators see the Parent, however, they understand the author of Target counters the hateful content in the Parent by showing empathy towards the mail carrier. In the last example, the Target alone is considered Hate because it attacks someone by using the phrase "sick person". When the Parent is shown, however, the annotators understand the Target as calling out the Parent to be inappropriate.
Label distribution and linguistic insights Figure 1 shows the label distribution for all pairs (right-most column in each block) and for pairs in which  $\text{comment}_{\text{w}/\text{hateword}}$  (i.e., the comment containing at least one hate word) is the  $\text{Parent}$  or  $\text{Target}$ . The most frequent label assigned to  $\text{Target}$  comments is Neutral (49%) followed by Hate (28%) and Counter-hate (23%). While  $\text{Target}$  comments containing a hate word are likely to be Hate (45%), some are Counter-hate (19%) with context.
We analyze the linguistic characteristics of Titles, Parents and Targets when the Targets are Hate or
Counter-hate with context to shed light on the differences between the language people use in hate and counter speech. We combine the set of hate words with profanity words to check for profanity words.4 We analyze sentiment and cognitive factors using the Sentiment Analysis and Cognition Engine (SEANCE) lexicon, a popular tool for psychological linguistic analysis (Crossley et al., 2017). Statistical tests are conducted using unpaired t-tests between the groups, of which the Targets are Counter-hate or Hate (Table 5). We also report whether each feature passes the Bonferroni correction as multiple hypothesis tests are performed. We draw several interesting insights:
- Questions marks in Target signal Counter-hate. They are often rhetorical questions.
- Fewer 1st person pronouns (e.g., I, me) and more 2nd person pronouns (e.g., you, your) in the Parent signal that the Target is more likely to be Counter-hate. This is due to the fact that people tend to target others in hateful content.
- High profanity count in the Parent signals that the Target is Counter-hate, while high profanity count in the Target signals Hate.
- More words related to awareness, enlightenment and problem-solving in the Target signal Counter-hate.
- When there are more negative words in the Parent, the Target tends to be Counter-hate. Targets labeled as Counter-hate contain fewer negative and disgusting words.
4https://github.com/RobertJGabriel/ google-profanity-words-node-module/blob/ master/lib/profanity.js
Hate	Counter-hate	Neutral	Weighted Average
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Majority Baseline	0.00	0.00	0.00	0.00	0.00	0.00	0.51	1.00	0.67	0.26	0.51	0.34
Trained with Target	0.56	0.55	0.56	0.41	0.36	0.38	0.67	0.71	0.69	0.58	0.59	0.58
+ Silver	0.58	0.55	0.57	0.44	0.42	0.43	0.69	0.72	0.70	0.60	0.61	0.61
+ Related task	0.56	0.55	0.56	0.51	0.41	0.45	0.68	0.74	0.71	0.61	0.61	0.61
+ Silver + Related task	0.55	0.56	0.56	0.49	0.53	0.51	0.67	0.69	0.70	0.61	0.61	0.61
Trained with Parent_Target	0.56	0.62	0.59	0.52	0.38	0.44	0.68	0.72	0.70	0.61	0.62	0.61
+ Silver†	0.58	0.57	0.57	0.49	0.51	0.50	0.72	0.71	0.72	0.63	0.63	0.63
+ Related task‡	0.55	0.66	0.60	0.54	0.43	0.48	0.71	0.70	0.71	0.63	0.63	0.63
+ Silver + Related task‡	0.55	0.65	0.60	0.54	0.52	0.53	0.74	0.68	0.71	0.64	0.64	0.64
Table 6: Results obtained with several systems. We indicate statistical significance (McNemar's test (McNemar, 1947) over weighted average) with respect to the model trained with the Target only using neither Silver nor pretraining on related tasks as follows:  $\dagger$  indicates  $p < 0.05$  and  $\ddagger$  indicates  $p < 0.01$ . Training with the Parent and Target coupled with blending Silver annotations and pretraining with stance corpora yields the best results. The supplementary materials detail the results pretraining with all related tasks we consider.
We build neural network models to identify if a Target comment is Hate, Counter-hate, or Neutral. We randomly split Gold instances (4,751) as follows:  $70\%$  for training,  $15\%$  for validation and  $15\%$  for testing. Silver instances are only used for training.
Neural Network Architecture We experiment with neural classifiers built on top of the RoBERTa transformer (Liu et al., 2019). The neural architecture consists of a pretrained RoBERTa transformer, a fully connected layer (768 neurons and Tanh activation), and another fully connected layer (3 neurons and softmax activation) to make predictions (Hate, Counter-hate, or Neutral). To investigate the role of context, we consider two textual inputs:
- the Target alone (Target), and
- the Parent and the Target (Parent_Target).
We concatenate the Target and the Parent with the [SEP] special token. We conduct multiple runs of experiments, which show consistent results. The hyperparameters and other implementation details are presented in the Appendix. We also experiment models that take the title of the discussion as part of the context, but it is not beneficial.
We implement two strategies to enhance the performance of neural models: