Venue: ACL 2020

# Toxicity Detection: Does Context Really Matter?

John Pavlopoulos\({}^{\dagger}\), Jeffrey Sorensen\({}^{\ddagger}\)

**Lucas Dixon\({}^{\ddagger}\), Nithum Thain\({}^{\ddagger}\), Ion Androutsopoulos\({}^{\dagger}\) \({}^{\dagger}\) Department of Informatics, Athens University of Economic and Business, Greece annis,ion@aueb.gr \({}^{\ddagger}\) Google sorenj,ldixon,nthain@google.com**

###### Abstract

Moderation is crucial to promoting healthy online discussions. Although several 'toxicity' detection datasets and models have been published, most of them ignore the context of the posts, implicitly assuming that comments may be judged independently. We investigate this assumption by focusing on two questions: (a) does context affect the human judgement, and (b) does conditioning on context improve performance of toxicity detection systems? We experiment with Wikipedia conversations, limiting the notion of context to the previous post in the thread and the discussion title. We find that context can both amplify or mitigate the perceived toxicity of posts. Moreover, a small but significant subset of manually labeled posts (5% in one of our experiments) end up having the opposite toxicity labels if the annotators are not provided with context. Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers, having tried a range of classifiers and mechanisms to make them context aware. This points to the need for larger datasets of comments annotated in context. We make our code and data publicly available.

+
Footnote †: 9HmSzc). For other languages, see for example the German GermEval ([https://goo.gl/u2Eerk](https://goo.gl/u2Eerk)).

+
Footnote †: 4296

## 1 Introduction

Systems that detect abusive language are used to promote healthy conversations online and protect minority voices Hosseini et al. (2017). Apart from a growing volume of press articles concerning toxicity online,1 there is increased research interest on detecting abusive and other unwelcome comments labeled 'toxic' by moderators, both for English and other languages.2 However, the vast majority of current datasets do not include the preceding comments in a conversation and such context was not shown to the annotators who provided the gold toxicity labels. Consequently, systems trained on these datasets ignore the conversational context. For example, a comment like "nope, I don't think so" may not be judged as rude or inflammatory by such a system, but the system's score would probably be higher if the system could also consider the previous (also called _parent_) comment "night it be that I am sincere?". Table 1 shows additional examples of comments that are not easily judged for toxicity without the parent comment. Interestingly, even basic statistics on how often context affects the perceived toxicity of online posts have not been published. Hence, in this paper we focus on the following two foundational research questions:

Footnote 1: Following the work of Wüclzyn et al. (2017) and Borkan et al. (2019), _racticity_ is defined as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion” Wüclzyı et al. (2017).

* RQ1: _How often does context affect the toxicity of posts as perceived by humans in online conversations? And how often does context amplify or mitigate the perceived toxicity?_
* RQ2: _How often does context affect the toxicity of posts as perceived by humans in online conversations? And how often does context amplify or mitigate the perceived toxicity?_

\begin{table}
\begin{tabular}{|p{38.3pt}|p{38.3pt}|} \hline Parent & All of his arguements are nail perfect, you’re inherently stupid. The lead will be changed. \\ Target & **Great argument!** \\ \hline \hline Parent & Really? It’s schuncks like you (and Bush) who turn the world into the shithole it is today! **I’d be interested in the reasoning for that comment, personally. (boundness)** \\ \hline \hline Parent & Indeed. Hitter was also strongly anti-porongraphy [...] it sure looks like Hitler is a hot potato that nobody wants to be stuck with. **Well I guess they won’t approve the slogan “Hitler hated porn”**.** \\ \hline \hline Parent & ?? When did I attack you? I definitely will present this to the arbocom, you should mind WP-CIVIL when participating in discussions in Wikipedia. **I blame you for my alcoholism add that too** \\ \hline Target & **J became you for my alcoholism add that too** \\ \end{tabular}
\end{table}
Table 1: Comments that are not easily labeled for toxicity without the ‘parent’ (previous) comment. The ’target’ comment is the one being labeled.



* RQ2: _Does context actually improve the performance of toxicity classifiers, when they are made context-aware? And how can toxicity classifiers be made context-aware?_

To investigate these questions we created and make publicly available two new toxicity datasets that include context, which are based on discussions in Wikipedia Talk Pages Hua et al. (2018). The first one is a small dataset of 250 comments, created in an AB test fashion, where two different groups of annotators (crowd-workers) were employed. One group annotated the comments without context, while the other group was given the same comments, this time along with the parent comment and the title of the thread as context. We used this dataset to show that the perceived toxicity of a significant subset of posts (5.2% in our experiment) changes when context is (or is not) provided. We conclude that a small but significant subset of manually labeled posts end up having wrong toxicity labels if the annotators are not provided with context. We also found that context can both amplify (approximately 3.6% of comments in our experiment) and mitigate (approx. 1.6%) the perceived toxicity. Examples of comments that were differently labeled with and without context are shown in Table 2.

To investigate the second question, concerning the effect of context on the performance of toxicity classifiers, we created a larger dataset of 20k comments; 10k comments were annotated out of context, 10k in context. This time we did not require the _same_ comments to be annotated with and without context, which allowed us to crowd-source the collection of a larger set of annotations. These two new subsets were used to train several toxicity detection classifiers, both context-aware and context-unaware, which were evaluated on held out comments that we always annotated in context (based on the assumption that in-context labels are more reliable). Surprisingly, we found no evidence that context actually improves the performance of toxicity classifiers. We tried a range of classifiers and mechanisms to make them context aware, and having also considered the effect of using gold labels obtained out of context or by showing context to the annotators. This finding is likely related to the small number of context-sensitive comments. In turn this suggests that an important direction for further research is how to efficiently annotate larger corpora of comments in context. We make our code and data publicly available.3

Footnote 3: [https://github.com/ipavlopoulos/context_toxicity](https://github.com/ipavlopoulos/context_toxicity)

## 2 Related Work

Toxicity detection has attracted a lot of attention in recent years Nobata et al. (2016); Pavlopoulos et al. (2017b); Park and Fung (2017); Wulczyn et al. (2017). Here we use the term 'toxic' as an umbrella term, but we note that the literature uses several terms for different kinds of toxic language or related phenomena: 'offensive' Zampieri et al. (2019a), 'abusive' Pavlopoulos et al. (2017a), 'hateful' Djuric et al. (2015); Malmasi and Zampieri (2017); ElSherief et al. (2018); Gamback and Sikdar (2017); Zhang et al. (2018), etc. There are also taxonomies for these phenomena based on their directness (e.g., whether the abuse was unambiguously implied/denoted or not), and their target (e.g., whether it was a general comment or targeting an individual/group) Waseem et al. (2017). Other hierarchical taxonomies have also been defined Zampieri et al. (2019a). While most previous work does not address toxicity in general, instead addressing particular subtypes, toxicity and its subtypes are strongly related, with systems trained to detect toxicity being effective also at subtypes, such as hateful language van Aken et al. (2018). As is customary in natural language processing, we focus on aggregate results when hoping to answer our research questions, and leave largely unanswered the related epistemological questions when this does not preclude using classifiers in real-world applications.

Table 3 lists all currently available public datasets for the various forms of toxic language that we are aware of. The two last columns show that

\begin{table}
\begin{tabular}{|p{42.7pt}|p{142.3pt}|} \hline Comment with toxicity AMPLIFIED in context & But what if the user is a lesbian’ Then what? \\ Target & ”Pigs Are People Too”, “**Avant-grade a clue**” \\ \hline Comment with toxicity MITIGATED in context & Hmmm. The flame on top of the gay pride emblen can probably be interpreted in a manner that I did non consider. Perhaps one icon on each and using? \\ Target & **Hi Gadget, interpreted in what manner? Flaming says? Or Burn a gay?** \\ \hline \end{tabular}
\end{table}
Table 2: Examples of comments that the annotators labeled differently when the previous (parent) comment was (or not) provided. In the top example, the target comment (the one being annotated) was labeled as toxic only when context was given. In the bottom example, the target comment was considered toxic only without its parent comment.

 

no existing English dataset provides both context (e.g., parent comment) and context-aware annotations (annotations provided by humans who also considered the parent comment).

Both small and large toxicity datasets have been developed, but approximately half of them contain tweets, which makes reusing the data difficult, because abusive tweets are often removed by the platform. Moreover, the textual content is not available under a license that allows its storage outside the platform. The hateful language detection dataset of Waseem and Hovy (2016), for example, contains 1,607 sexism and racism annotations for ids of English tweets. A larger dataset was published by Davidson et al. (2017), containing approx. 25k annotations for tweet-ids, collected using a lexicon of hateful terms. Research on forms of abusive language detection is mainly focused on English (6 out of 10 datasets), but datasets in other languages also exist, such as Greek Pavlopoulos et al. (2017a), Arabic Mubarak et al. (2017), and German Ross et al. (2016); Wiegand et al. (2018).

A common characteristic of most of the datasets listed in Table 3 is that, during annotation, the human workers were not provided with, nor instructed to review, the context of the target text. Context such as the preceding comments in the thread, or the title of the article being discussed, or the discussion topic. A notable exception is the work of Gao and Huang (2017), who annotated hateful comments under Fox News articles by also considering the title of the news article and the preceding comments. However, this dataset has three major shortcomings. First, the dataset is very small, comprising approximately 1.5k posts retrieved from the discussion threads of only 10 news articles. Second, the authors did not release sufficient information to reconstruct the threads and allow systems to consider the parent comments. Third, only a single annotator was used for most of the comments, which makes the annotations less reliable.

Two other datasets, both non English, also include context-aware annotations. Mubarak et al. (2017) provided the title of the respective news article to the annotators, but ignored parent comments. This is problematic when new comments change the topic of the discussion and when replies require the previous posts to be judged. Pavlopoulos et al. (2017a) used professional moderators, who were monitoring entire threads and were thus able to use the context of the thread to judge for the toxicity of the comments. However, the plain text of the comments for this dataset is not available, which makes further analysis difficult. Moreover, crucially for this study, the context of the comments was not released in any form.

In summary, of the datasets we know of (Table 3), only two include context Gao and Huang (2017); Mubarak et al. (2017), and this context is limited to the title of the news article the comment was about. As discussed above, Gao and Huang (2017) include the parent comments in their dataset, but without sufficient information to link the target comments to the parent ones. Hence _no toxicity dataset includes the raw text of both target and parent comments with sufficient links between the two_. This means that toxicity detection methods cannot exploit the conversational context when being trained on existing datasets.

Using previous comments of a conversation or preceding sentences of a document is not uncommon in text classification and language modeling. Mikolov and Zweig (2012), for example, used lda to encode the preceding sentences and pass the en

\begin{table}
\begin{tabular}{|c|c|c|c|c||c|c|} \hline
* \\ \hline cwtk & Wikipedia Toxicity Kaggle & 223,549 & Toxicity sub-types & EN & ✗ &
- \\ \hline Davidson et al. (2017) & Twitter & 24,783 & HateOfense & EN & ✗ &
- \\ \hline Zampieri et al. (2019a) & Twitter & 14,100 & Offense & EN & ✗ &
- \\ \hline Waseem and Hovy (2016) & Twitter & 1,607 & Sexism/Racism & EN & ✗ &
- \\ \hline Gao and Huang (2017) & Fox News & 1,528 & Hate & EN & ✓ & Tate \\ \hline Wiegand et al. (2018) & Twitter & 8541 & Insult/Abuse/Profatuity & DE & ✗ &
- \\ \hline Ross et al. (2016) & Twitter & 470 & Hate & DE & ✗ &
- \\ \hline Pavlopoulos et al. (2017a) & Gazzetta.gr & 1,6M & Rejection & EL & ✓ &
- \\ \hline Mubarak et al. (2017) & Aljazeera.net & 31,633 & Obscene/Offense & AR & ✓ & Title \\ \hline \end{tabular}
\end{table}
Table 3: Publicly available datasets for toxicity detection. The Size column shows the number of comments. Column \(C_{a}\) shows if annotation was context-aware or not. Column \(C_{t}\) shows the type of context provided. Pavlopoulos et al. (2017a) used professional moderator decisions, which were context-aware, but context is not included in their dataset. The datasets of Gao and Huang (2017) and Mubarak et al. (2017) include context-aware labels, but provide only the titles of the news articles being discussed.
 

coded sentence history to an rnn language model Blei et al. (2003). Their approach achieved state of the art language modeling results and was used as an alternative solution (e.g., to lstms) for the problem of vanishing gradients. Sordoni et al. (2015) experimented with concatenating consecutive utterances (or their representations) before passing them to an rnn to generate conversational responses. They reported gains up to \(11\%\) in bleu Papineni et al. (2002). Ren et al. (2016) reported significant gains in Twitter sentiment classification, when adding contextual features.

## 3 Experiments

### Experiments with cat-small for ro1

To investigate how often context affects the perceived toxicity of posts, we created cat-small, a small Context-Aware Toxicity dataset of 250 randomly selected comments from the Wikipedia Talk Pages (Table 4). We gave these comments to two groups of crowd-workers to judge their toxicity. The first group gc Group with Context) was also given access to the parent comment and the discussion title, while the second group gn Group with No context) was provided with no context. No annotator could belong to both groups, to exclude the case of an annotator having seen the context of a post and then being asked to label the same post without its context. We used the Figure Eight crowd-sourcing platform, which provided us with these mutually exclusive groups of annotators.4 We collected three judgments per comment, per group. All comments were between 10 and 400 characters long. Their depth in their threads was from 2 (direct reply) to 5.

Footnote 4: See [https://www.figure-eight.com/](https://www.figure-eight.com/). The annotators were high-performing workers from previous jobs. The demographics and backgrounds of the crowdworkers are detailed in Posch et al. (2018).

We used the parent comment and discussion title only, instead of a larger context (e.g., the entire thread), to speed up our machine learning experiments, and also because reading only the previous comment and the discussion title made the manual annotation easier. In preliminary experiments, we observed that including more preceding comments had the side effect of workers tending to ignore the context completely.5 We addressed this problem by asking the annotators an extra question: "Was the parent comment less, more, or equally toxic?"

Footnote 5: We experimented with providing the gc annotators with all the parent comments in the discussion. We also experimented with preselection strategies, such as employing the score from a pre-trained toxicity classifier for a stratified selection and using a list of terms related to minority groups.

For each comment and group of annotators, the toxicity scores of the annotators were first averaged and rounded to the nearest binary decision, as in Table 4. Figure 1 shows that the toxicity ratio (toxic comments over total) of cat-small is higher when annotators are given context gc compared to when no context is provided gn. A one-sided Wilcoxon-Mann-Whitney test shows this is a statistically significant increase. This is a first indication that providing context to annotators affects their decisions. The toxicity ratio increases by 2 percentage points (4.4% to 6.4%) when context is provided, but this is an aggregated result, possibly hiding the true size of the effect of context. The perceived toxicity of some comments may be increasing when context is provided, but for other comments it may be decreasing, and these effects may be partially cancelling each other when measuring the change in toxicity ratio.

To get a more accurate picture of the effect of

Figure 1: Toxicity ratio (%) of the comments of cat-small when using the toxicity labels of gn (annotators with no context) or gc (annotators with context). The difference is statistically significant (\(P<.01\)).

 

context, we measured the number of comments of cat-small for which the (averaged and rounded) toxicity label was different between the two groups (gn, gc). We found that the toxicity of 4 comments out of 250 (1.6%) decreased with context, while the toxicity of 9 comments (3.6%) increased. Hence, perceived toxicity was affected for 13 comments (5.2% of comments). While the small size of cat-small does not allow us to produce accurate estimates of the frequency of posts whose perceived toxicity changes with context, the experiments on cat-small indicate that context has a statistically significant effect on the perceived toxicity, and that context can both amplify or mitigate the perceived toxicity, thus making a first step to addressing our first research question (rq1). Nevertheless, larger annotated datasets need to be developed to estimate more accurately the frequency of context-sensitive posts in online conversations, and how often context amplifies or mitigates toxicity.

### Experiments with cat-large for rq2

To investigate whether adding context can benefit toxicity detection classifiers, we could not use cat-small, because its 250 comments are too few to effectively train a classifier. Thus, we proceeded with the development of a larger dataset. Although the best approach would be to extend cat-small, which had two mutually exclusive groups of annotators labeling each comment, we found that the annotation process was very slow in that case, largely because of the small size of annotator groups we had access to in Figure Eight (19 and 23 for gc and gn respectively).6 By contrast, when we did not request mutually exclusive annotator groups, we could get many more workers (196 and 286 for gc and gn respectively) and thus annotation became significantly faster.

Footnote 6: Figure Eight provided us with the two mutually exclusive annotator groups, which could not grow in size.

For this larger dataset, dubbed cat-large, we annotated 20k randomly selected comments from Wikipedia Talk Pages. 10k comments were annotated by human workers who only had access to the comment in question (group with no context, gn). The other 10k comments were annotated by providing the annotators also with the parent comment and the title of the discussion (group with context, gc). Each comment was annotated by three workers. We selected comments of length from 10 and 400 characters, with depth in thread from 2 (direct reply) to 5. Inter-annotator agreement was computed with Krippendorff's alpha on 123 texts, and it was found to be 0.72% for gn and 0.70% for gc.

Figure 2 shows that the toxicity ratio increased (from 0.6% to 1.5%) when context was given to the annotators. A one-sided Wilcoxon-Mann-Whitney test shows this is a statistically significant increase (\(P<.001\)). Again, the change of toxicity ratio is an indication that context does affect the perceived toxicity, but it does not accurately show how many comments are affected by context, since the perceived toxicity may increase for some comments when context is given, and decrease for others. Unlike cat-small, in cat-large we cannot count for how many comments the perceived toxicity increased or decreased with context, because the two groups of annotators (gn, gc) did not annotate the same comments. The toxicity ratios of cat-large (Fig. 2) are lower than in cat-small (Fig. 1), though they both show a trend of increased toxicity ratio when context is provided. The toxicity ratios of cat-large are more reliable estimates of toxicity in online conversations, since they are based on a much larger dataset.

We used cat-large to experiment with both context-insensitive and context-sensitive toxicity classifiers. The former only consider the post being rated (the target comment), whereas the latter also consider the context (parent comment).

#### Context Insensitive Toxicity Classifiers

BilstOur first context-insensitive classifier is a bidirectional lstm[1]. On top of the concatenated last states (from the two directions) of the bilstm, we add

Figure 2: Toxicity ratio (%) of the comments of cat-large-n (10k comments annotated with no context, left) and cat-large-c (10k other comments annotated with context, right). For each comment, the toxicity scores of the annotators were first averaged and rounded to the nearest binary decision, as in Table 4. The difference is statistically significant (\(P<.001\)).

 

a feed-forward neural network (ffnn), consisting of a hidden dense layer with 128 neurons and \(\tanh\) activations, then a dense layer leading to a single output neuron with a sigmoid that produces the toxicity probability. We fix the bias term of the single output neuron to \(\log{T\over N}\), where \(T\) and \(N\) are the numbers of toxic and non-toxic training comments, respectively, to counter-bias against the majority (non-toxic) class.7 This bilstm-based model could, of course, be made more complex (e.g., by stacking more bilstm layers, and including self-attention), but it is used here mainly to measure how much a relatively simple (by today's standards) classifier benefits when a context mechanism is added (see below).

Footnote 7: See an example in [http://tiny.cc/m572gz](http://tiny.cc/m572gz).

**bert** At the other end of complexity, our second context-insensitive classifier is bert[4], fine-tuned on the training subset of each experiment, with a task-specific classifier on top, fed with bert's top-level embedding of the [cls] token. We use bert-base pre-trained on cased data, with 12 layers and 768 hidden units. We only unfreeze the top three layers during fine-tuning, with a small learning rate (2e-05) to avoid catastrophic forgetting. The task-specific classifier is the same ffnn as in the bilstm classifier.

**bert-cctk** We also experimented with a bert model that is the same as the previous one, but finetuned on a sample (first 100k comments) of the cctk dataset (Table 3). We used the general toxicity labels of that dataset, and fine-tuned for a single epoch. The only difference of this model, compared to the previous one, is that it is fine-tuned on a much larger training set, which is available, however, only without context (no parent comments). The annotators of the dataset were also not provided with context (Table 3).

**perspective** The third context-insensitive classifier is a cnn-based model for toxicity detection, trained on millions of user comments from online publishers. It is publicly available through the perspective api.8 The publicly available form of this model cannot be retrained, fine-tuned, or modified to include a context-awareness component. Like bert-cctk, this model uses an external (but now much larger) labeled training set. This training set is not publicly available, it does not include context, and was labeled by annotators who were not provided with context.

Footnote 8: [https://www.perspectiveapi.com/](https://www.perspectiveapi.com/)

### Context Sensitive Toxicity Classifiers

ca-bilstm-bilstm In a context-aware extension of the context-insensitive bilstm classifier, dubbed ca-bilstm-bilstm, we added a second bilstm to encode the parent comment (Fig. 3). The vector representations of the two comments (last states from the two directions of both bilstms) are concatenated and passed to a ffnn, which is otherwise identical to the ffnn of the context-insensitive bilstm.

#### ca-bilstm-bert

We also used a bilstm to encode the parent in a context-aware extension of the bert-based classifier, called ca-bilstm-bert (Fig. 4). Now bert encodes the target comment, whereas a bilstm (the same as in ca-bilstm-bilstm) encodes the parent. (We could not use two bert instances to encode both the parent and the target comment, because the resulting model did not fit in our gpu.) The concatenated representations of the two comments are passed to a ffnn,

Figure 4: Illustration of ca-bilstm-bert. bert encodes the target comment. bilstm (shown unidirectional for simplicity) encodes the parent comment. The vector representations of the two comments are concatenated and passed to a ffnn.

Figure 3: Illustration of ca-bilstm-bilstm. Two bilstms, shown unidirectional for simplicity, encode the parent and target comment. The concatenation of the vector representations of the two comments is then passed to a ffnn.

 

which is otherwise the same as as in previous models. bert is fine-tuned on the training subset, as before, and the bilstm encoder of the parent is jointly trained (with a larger learning rate).

**ca-sep-bert** We also experimented with another context-aware version of the bert-based classifier, dubbed ca-sep-bert. This model concatenates the text of the parent and target comments, separated by bert's [sep] token, as in bert's next sentence prediction pre-training task (Fig. 5). Unlike ca-bilstm-bert, it does not use a separate encoder for the parent comment. The model is again fine-tuned on the training subset.

**ca-conc-bert-cctk,****ca-conc-perspective** These are exactly the same as bert-cctk and perspective, respectively, trained on the same data as before (no context), but at test time they are fed with the concatenation of the text of the parent and target comment, as a naive context-awareness mechanism.

### Context Sensitive vs. Insensitive Classifiers

Table 5 reports roc aux scores, averaged over a 5-fold Monte Carlo (mc) cross-validation, i.e., using 5 different random training/development/test splits (Gorman and Bedrick, 2019); we also report the standard error of mean over the folds. The models are trained on the training subset(s) of cat-large-n (@n models) or cat-large-c (@c models), i.e., they are trained on comments with gold labels obtained _without_ or _with_ context shown to the annotators, respectively. All models are always evaluated (in each fold) on the test subset(s) of cat-large-c, i.e., with gold labels obtained _with_ context shown to annotators, assuming that those labels are more reliable (the annotators had a broader view of the discussion). In each fold (split) of the mc cross-validation, the training, development, and test subsets are 60%, 20%, and 20% of the data, respectively, preserving in each subset the toxicity ratio of the entire dataset. We always use the test (and development) subsets of cat-large-c, as always noted. We report roc aux, because both datasets are heavily unbalanced, with toxic comments being rare (Fig. 2). 9

Footnote 9: Recall that we also fix the bias term of the output neuron of each model (apart from perspective) to \(-\log{T\over N}\), to bias against the majority class. We also tried under-sampling to address class imbalance, but this technique worked best.

A first observation from Table 5 is that the best results are those of perspective, bert-cctk, and their context-aware variants (last four rows). This is not surprising, since these systems were trained (fine-tuned in the case of bert-cctk) on much larger toxicity datasets than the other systems (upper two zones of Table 5), and bert-cctk was also pre-trained on even larger corpora.

What is more surprising is that _any kind of information about the context does not lead to any consistent (or large) improvement in system performance_. perspective and bert-cctk seem to improve slightly with the naive context-awareness mechanism of concatenating the parent and target text during testing, but the improvement is very small and we did not detect a statistically significant difference.10 Training with gold labels obtained from annotators that had access to context (@c models) also leads to no consistent (or large) gain, compared to training with gold labels obtained out of context (@n models). This is probably due to the fact that context-sensitive comments are few (5.2% in the experiments on cat-small) and, hence, any noise introduced by using gold labels obtained out of context does not significantly affect the performance of the models.

Footnote 10: We used single-tailed stratified shuffling (Drør et al., 2018; Smucker et al., 2007), \(P<0.01\), 10,000 repetitions, 50% swaps in each repetition.

There was also no consistent (or large) improvement when encoding the parent comments with a bilstm (ca-bilstm-bilstm, ca-bilstm-bert) or directly as in bert's next sentence prediction pre-training task (ca-sep-bert). This is again probably a consequence of the fact that context-sensitive comments are few. The small number of context-sensitive comments does not allow the bilstm- and bert-based classifiers to learn how to use the context encodings to cope with

Figure 5: Illustration of ca-sep-bert. A single bert instance encodes the parent and target comments, separated by [sep]. The top-level representation of the [cls] token is passed to a frnn.

 

context-sensitive comments, and failing to cope with context-sensitive comments does not matter much during testing, again since context-sensitive comments are so few.

We conclude for our second research question (rq2) that we found no evidence that context actually improves the performance of toxicity classifiers, having tried both simple (bilstm) and more powerful classifiers (bert), having experimented with several methods to make the classifiers context aware, and having also considered the effect of gold labels obtained out of context vs. gold labels obtained by showing context to annotators.

## 4 Conclusions and Future Work

We investigated the role of context in detecting toxicity in online comments. We collected and share two datasets for investigating our research questions around the effect of context on the annotation of toxic comments (rq1) and its detection by automated systems (rq2). We showed that context does have a statistically significant effect on toxicity annotation, but this effect is seen in only a narrow slice (\(5.2\%\)) of the (first) dataset. We also found no evidence that context actually improves the performance of toxicity classifiers, having tried both simple and more powerful classifiers, having experimented with several methods to make the classifiers context aware, and having also considered the effect of gold labels obtained out of context vs. gold labels obtained by showing context to the annotators. The lack of improvement in system performance seems to be related to the fact that context-sensitive comments are infrequent, at least in the data we collected.

A limitation of our work is that we considered a narrow contextual context, comprising only the previous comment and the discussion title.11 It would be interesting to investigate in future work ways to improve the annotation quality when more comments in the discussion thread are provided, and also if our findings hold when broader context is considered (e.g., all previous comments in the thread, or the topic of the thread as represented by a topic model). Another limitation of our work is that we used randomly sampled comments. The effect of context may be more significant in conversations about particular topics, or for particular conversational tones (e.g. sarcasm), or when they reference communities that are frequently the target of online abuse. Our experiments and datasets provide an initial foundation to investigate these important directions.

Footnote 11: The discussion title was used only by the human annotators that examined context, not by context-aware systems, which considered only the parent comment.

## Acknowledgments

We thank the anonymous reviewers for their comments. This research was funded in part by Google.

## References

* van Aken et al. (2018) B. van Aken, J. Risch, R. Krestel, and A. Loser. 2018. Challenges for toxic comment classification: An indepth error analysis. In _2nd Workshop on Abusive Language Online_, pages 33-42, Brussels, Belgium.
* Blei et al. (2003) D. M. Blei, A. Y Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. _Journal of Machine Learning Research_, 3(Jan):993-1022.
* Borkan et al. (2019) D. Borkan, L. Dixon, J. Sorensen, N. Thain, and L. Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In _Companion Proceedings of The 2019 World Wide Web Conference_, WWW '19, page 491-500. Association for Computing Machinery.
* Coutinhois et al. (2017) 

T. Davidson, D. Warmsley, M. Macy, and I. Weber (2017) Automated hate speech detection and the problem of offensive language. In ICWSM, pp. 512-515. External Links: Link Cited by: SS3.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, Minneapolis, MN, USA, pp. 4171-4186. External Links: ISBN 978-3-319-14503-3 Cited by: SS1.
* N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and N. Bhamidipati (2015) Hate speech detection with comment embeddings. In ICWWW, pp. 29-30. Cited by: SS1.
* R. Dror, G. Baumer, S. Shlomov, and R. Reichart (2018) The hitchhiker's guide to testing statistical significance in natural language processing. In ACL, Melbourne, Australia, Cited by: SS1.
* M. ElSherief, V. Kulkarni, D. Nguyen, W. Y. Wang, and E. Belding (2018) Hate lingo: a target-based linguistic analysis of hate speech in social media. arXiv preprint. Cited by: SS1.
* B. Gamback and U. K. Sikdar (2017) Using convolutional neural networks to classify hate-speech. In 1st Workshop on Abusive Language Online, Vancouver, Canada, Cited by: SS1.
* L. Gao and R. Huang (2017) Detecting online hate speech using context aware models. In RANLP, pp. 260-266. Cited by: SS1.
* Kyle Gorman and Steven Bedrick (2019) We need to talk about standard splits. In ACL, pp. 2786-2791. Cited by: SS1.
* S. Hochreiter and J. Schmidhuber (1997) Long short-term memory. Neural Computation9 (8), pp. 1735-1780. Cited by: SS1.
* H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran (2017) Deceiving Google's perspective api built for detecting toxic comments. arXiv preprint. Cited by: SS1.
* Y. Hua, C. Danescu-Niculescu-Mizil, D. Taraborelli, N. Thain, J. Sorensen, and L. Dixon (2018) Wikiconv: A corpus of the complete conversational history of a large online collaborative community. arXiv preprint. Cited by: SS1.
* R. Kumar, A. K. Ojha, S. Malmasi, and M. Zampieri (2018) Benchmarking aggression identification in social media. In TRAC, Santa Fe, USA, Cited by: SS1.
* S. Malmasi and M. Zampieri (2017) Detecting hate speech in social media. In RANLP, pp. 467-472. Cited by: SS1.
* T. Mikolov and G. Zweig (2012) Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp. 234-239. Cited by: SS1.
* H. Mubarak, K. Darwish, and W. Magdy (2017) Abusive language detection on arabic social media. In 1st Abusive Language Workshop, Vol. , San Diego, California, pp. 52-56. Cited by: SS1.
* C. Nobata, J. Tetreatult, A. Thomas, Y. Mehdad, and Y. Chang (2016) Abusive language detection in online user content. In ICWWW, pp. 145-153. Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311-318. Cited by: SS1.
* J. H. Park and P. Fung (2017) One-step and two-step classification for abusive language detection on twitter. In 1st Workshop on Abusive Language Online, pp. 41-45. Cited by: SS1.
* J. Pavlopoulos, P. Malakasiotis, and I. Androutsopolos (2017) Deep learning for user comment moderation. In 1st Workshop on Abusive Language Online, pp. 25-35. Cited by: SS1.
* J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos (2017) Deeper attention to abusive user content moderation. In EMNLP, Copenhagen, Denmark, pp. 1125-1135. Cited by: SS1.
* L. Posch, A. Bleier, F. Flock, and M. Strohmaier (2018) Characterizing the global crowd workforce: a cross-country comparison of crowdworker demographics. arXiv preprint. Cited by: SS1.
* Y. Ren, Y. Zhang, M. Zhang, and D. Ji (2016) Context-sensitive twitter sentiment classification using neural network. In 30th AAAI Conference on Artificial Intelligence, Cited by: SS1.
* B. Ross, M. Rist, G. Carbonell, B. Cabrera, N. Kurowsky, and M. Wojatzki (2016) Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In NLP4CMC, Bochum, Germany, Cited by: SS1.
* M. D. Smucker, J. Allan, and B. Carterette (2007) A comparison of statistical significance tests for information retrieval evaluation. In CIKM, pp. 623-632. Cited by: SS1.
* A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan (2015) A neural network approach to context-sensitive generation of conversational responses. arXiv preprint. Cited by: SS1.
* Z. Waseem, T. Davidson, D. Warmsley, and I. Weber (2017) Understanding abuse: a typology of abusive language detection subtasks. In 1st Workshop on Abusive Langauge Online, Vancouver, Canada, Cited by: SS1.
* Z. Waseem and D. Hovy (2016) Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In NAACL SRW, pp. 88-93. Cited by: SS1.
* M. Wiegand, M. Siegel, and J. Ruppenhofer (2018) Overview of the germeval 2018 shared task on the identification of offensive language. In Proceedings of GermanEval, Cited by: SS1.
* E. Wolczyn, N. Thain, and L. Dixon (2017) Ex machina: personal attacks seen at scale. In ICWWW, pp. 1391-1399 

M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In _NAACL_.
* Zampieri et al. (2019b) M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019b. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In _SemEval_.
* Zhang et al. (2018) Z. Zhang, D. Robinson, and J. Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In _Lecture Notes in Computer Science_. Springer Verlag.

## Appendix A Data Annotation

Annotators were asked to judge the toxicity of each comment, given the following definitions:

* very toxic: A very hateful, aggressive, disrespectl comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective.
* toxic: A rule, disrespectl, unreasonable comment or otherwise somewhat likely to make a user leave a discussion or give up on sharing their perspective.
* unsure: Due to polysemy, lack of context or other reasons.
* not toxic: Not containing any toxicity.

For annotation, we used the 'Figure Eight' platform and we invested 5 cents per row.13 For the cat-small we employed high accuracy annotators (i.e., from zone 3), selected from 7 English speaking countries (i.e., UK, Ireland, USA, Canada, New Zealand, South Africa, Australia), and only ones allowing explicit content (we also warned about the explicit content in the title). 62 quiz questions were used. For the cat-large, we invested the same amount of money but all the annotators were able to participate (again, they ware warned for the explicit content). Inter annotator agreement was measured on the quiz questions with Krippendorff's alpha and was found to be 70% and 72% for the C and N sets.

Footnote 13: [https://www.figure-eight.com/](https://www.figure-eight.com/)

gc annotators had one more question, which was asking them to compare the toxicity of the target comment to that of the parent comment. The main scope of that question was to make it less easy for annotators to ignore the parent comment.

## Appendix B Hyper parameters

All systems were trained for 100 epochs with patience of 3 epochs. We performed early stopping by monitoring the validation ROC AUC.

### bilstm

The hidden size of the LSTM cells had size 128. We used batch size 128, max length 512, and we concatenated the forward and backward last hidden states before the FFNN. We used binary cross entropy for loss and Adam optimizer was used with default parameters (learning rate 1e-03).

#### ca-bilstm-bilstm

We used the same hyper-parameters with bilstm but included one more bidirectional LSTM to encode the parent text. The parent biLSTM had 64 hidden nodes and we concatenated the forward and backward last hidden states. The parent and the target embeddings (the ones generated by the two biLSTMS) were concatenated before being passed to the FFNN.

### bert

We used a learning rate of 2e-05 for BERT and only unfroze the top three layers during training to our data. On top of the BERT [CLS] representation, we added a FFNN of 128 hidden nodes and a sigmoid to yield the toxicity probability. 128 tokens were used as maximum sequence length.

#### ca-sep-bert

A [SEP] token separated the two texts and the [CLS] token was used as with bert. Same parameters with bert were used.

#### ca-bilstm-bert

We used a bidirectional LSTM to encode the parent comment, similarly to ca-bilstm-bilstm. The biLSTM representation was concatenated with the [CLS] representation before the FFNN. All other parameters were set to the same values as bert.