SlowGuess/ACL_2020_OCR huggingface dataset:
BERT-CCTK We also experimented with a BERT model that is the same as the previous one, but fine-tuned on a sample (first 100k comments) of the CCTK dataset (Table 3). We used the general toxicity labels of that dataset, and fine-tuned for a single epoch. The only difference of this model, compared to the previous one, is that it is fine-tuned on a much larger training set, which is available, however, only without context (no parent comments). The annotators of the dataset were also not provided with context (Table 3).
PERSPECTIVE The third context-insensitive classifier is a CNN-based model for toxicity detection, trained on millions of user comments from online publishers. It is publicly available through the PERSPECTIVE API. $^{8}$  The publicly available form of this model cannot be retrained, fine-tuned, or modified to include a context-awareness component. Like BERT-CCTK, this model uses an external (but now much larger) labeled training set. This training set is not publicly available, it does not include context, and was labeled by annotators who were not provided with context.
Parent: Don't they deserve some respect too?
Figure 3: Illustration of CA-BILSTM-BILSTM. Two BILSTMs, shown unidirectional for simplicity, encode the parent and target comment. The concatenation of the vector representations of the two comments is then passed to a FFNN.
Parent: Don't they deserve some respect too?
Figure 4: Illustration of CA-BILSTM-BERT. BERT encodes the target comment. BILSTM (shown unidirectional for simplicity) encodes the parent comment. The vector representations of the two comments are concatenated and passed to a FFNN.
Context Sensitive Toxicity Classifiers
CA-BILSTM-BILSTM In a context-aware extension of the context-insensitive BILSTM classifier, dubbed CA-BILSTM-BILSTM, we added a second BILSTM to encode the parent comment (Fig. 3). The vector representations of the two comments (last states from the two directions of both BILSTMs) are concatenated and passed to a FFNN, which is otherwise identical to the FFNN of the context-insensitive BILSTM.
CA-BILSTM-BERT We also used a BILSTM to encode the parent in a context-aware extension of the BERT-based classifier, called CA-BILSTM-BERT (Fig. 4). Now BERT encodes the target comment, whereas a BILSTM (the same as in CA-BILSTM-BILSTM) encodes the parent. (We could not use two BERT instances to encode both the parent and the target comment, because the resulting model did not fit in our GPU.) The concatenated representations of the two comments are passed to a FFNN,
7See an example in http://tiny.cc/m572gz.
<sup>8</sup>https://www.perspectiveapi.com/
which is otherwise the same as as in previous models. BERT is fine-tuned on the training subset, as before, and the BILSTM encoder of the parent is jointly trained (with a larger learning rate).
CA-SEP-BERT We also experimented with another context-aware version of the BERT-based classifier, dubbed CA-SEP-BERT. This model concatenates the text of the parent and target comments, separated by BERT's [SEP] token, as in BERT's next sentence prediction pre-training task (Fig. 5). Unlike CA-BILSTM-BERT, it does not use a separate encoder for the parent comment. The model is again fine-tuned on the training subset.
CA-CONC-BERT-CCTK,
CA-CONC-PERSPECTIVE These are exactly the same as BERT-CCTK and PERSPECTIVE, respectively, trained on the same data as before (no context), but at test time they are fed with the concatenation of the text of the parent and target comment, as a naive context-awareness mechanism.
Context Sensitive vs. Insensitive Classifiers
Table 5 reports ROC AUC scores, averaged over a 5-fold Monte Carlo (MC) cross-validation, i.e., using 5 different random training/development/test splits (Gorman and Bedrick, 2019); we also report the standard error of mean over the folds. The models are trained on the training subset(s) of CAT-LARGE-N (@N models) or CAT-LARGE-C (@C models), i.e., they are trained on comments with gold labels obtained without or with context shown to the annotators, respectively. All models are always evaluated (in each fold) on the test subset(s) of CAT-LARGE-C, i.e., with gold labels obtained with context shown to annotators, assuming that those labels are more reliable (the annotators had a broader view of the discussion). In each fold (split) of the MC cross-validation, the training, development, and test subsets are  $60\%$ ,  $20\%$ , and  $20\%$  of the data, respectively, preserving in each subset the toxicity ratio of the entire dataset. We always use the test (and development) subsets of CAT-LARGE-C, as always noted. We report ROC AUC, because both datasets are heavily unbalanced, with toxic comments being rare (Fig. 2).<sup>9</sup>
A first observation from Table 5 is that the best results are those of PERSPECTIVE, BERT-CCTK, and their context-aware variants (last four rows).
Figure 5: Illustration of CA-SEP-BERT. A single BERT instance encodes the parent and target comments, separated by [SEP]. The top-level representation of the [CLS] token is passed to a FFNN.
This is not surprising, since these systems were trained (fine-tuned in the case of BERT-CCTK) on much larger toxicity datasets than the other systems (upper two zones of Table 5), and BERT-CCTK was also pre-trained on even larger corpora.
What is more surprising is that any kind of information about the context does not lead to any consistent (or large) improvement in system performance. PERSPECTIVE and BERT-CCTK seem to improve slightly with the naive context-awareness mechanism of concatenating the parent and target text during testing, but the improvement is very small and we did not detect a statistically significant difference.[10] Training with gold labels obtained from annotators that had access to context (@C models) also leads to no consistent (or large) gain, compared to training with gold labels obtained out of context (@N models). This is probably due to the fact that context-sensitive comments are few (5.2% in the experiments on CAT-SMALL) and, hence, any noise introduced by using gold labels obtained out of context does not significantly affect the performance of the models.
There was also no consistent (or large) improvement when encoding the parent comments with a BILSTM (CA-BILSTM-BILSTM, CA-BILSTM-BERT) or directly as in BERT's next sentence prediction pre-training task (CA-SEP-BERT). This is again probably a consequence of the fact that context-sensitive comments are few. The small number of context-sensitive comments does not allow the BILSTM- and BERT-based classifiers to learn how to use the context encodings to cope with
<sup>10</sup>We used single-tailed stratified shuffling (Dror et al., 2018; Smucker et al., 2007),  $P < 0.01$ , 10,000 repetitions,  $50\%$  swaps in each repetition.
Recall that we also fix the bias term of the output neuron of each model (apart from PERSPECTIVE) to  $-\log \frac{T}{N}$ , to bias against the majority class. We also tried under-sampling to address class imbalance, but this technique worked best.
model @training	ROC AUC @C
BILSTM @N	56.48±1.42
BILSTM @C	56.38±1.51
CA-BILSTM-BILSTM @N	56.13±1.27
CA-BILSTM-BILSTM @C	58.00±2.70
BERT @N	75.94±2.73
BERT @C	73.49±1.49
CA-BILSTM-BERT @N	74.60 ±3.08
CA-BILSTM-BERT @C	74.46±1.84
CA-SEP-BERT @N	73.29±3.89
CA-SEP-BERT @C	73.54±3.36
PERSPECTIVE	79.27±2.87
CA-CONC-PERSPECTIVE	81.89 ± 2.79
BERT-CCTK	78.08±1.50
CA-CONC-BERT-CCTK	81.69±2.22
Table 5: ROC AUC scores (\%) averaged over five-fold MC cross-validation (and standard error of mean) for models trained on CAT-LARGE-N (@N models, gold labels obtained without showing context) or CAT-LARGE-C (@C models, gold labels obtained with context). All models evaluated on the test subset of CAT-LARGE-C (AUC @C, gold labels obtained with context). PERSPECTIVE and BERT-CCTK were trained on larger external training sets with no context, but are tested on the same test subset (in each fold) as the other models.
context-sensitive comments, and failing to cope with context-sensitive comments does not matter much during testing, again since context-sensitive comments are so few.
We conclude for our second research question (RQ2) that we found no evidence that context actually improves the performance of toxicity classifiers, having tried both simple (BILSTM) and more powerful classifiers (BERT), having experimented with several methods to make the classifiers context aware, and having also considered the effect of gold labels obtained out of context vs. gold labels obtained by showing context to annotators.
We investigated the role of context in detecting toxicity in online comments. We collected and share two datasets for investigating our research questions around the effect of context on the annotation of toxic comments (RQ1) and its detection by automated systems (RQ2). We showed that context does have a statistically significant effect on toxicity annotation, but this effect is seen in only a narrow slice  $(5.2\%)$  of the (first) dataset. We also found no evidence that context actually improves
the performance of toxicity classifiers, having tried both simple and more powerful classifiers, having experimented with several methods to make the classifiers context aware, and having also considered the effect of gold labels obtained out of context vs. gold labels obtained by showing context to the annotators. The lack of improvement in system performance seems to be related to the fact that context-sensitive comments are infrequent, at least in the data we collected.
A limitation of our work is that we considered a narrow contextual context, comprising only the previous comment and the discussion title.[11] It would be interesting to investigate in future work ways to improve the annotation quality when more comments in the discussion thread are provided, and also if our findings hold when broader context is considered (e.g., all previous comments in the thread, or the topic of the thread as represented by a topic model). Another limitation of our work is that we used randomly sampled comments. The effect of context may be more significant in conversations about particular topics, or for particular conversational tones (e.g. sarcasm), or when they reference communities that are frequently the target of online abuse. Our experiments and datasets provide an initial foundation to investigate these important directions.
We thank the anonymous reviewers for their comments. This research was funded in part by Google.
B. van Aken, J. Risch, R. Krestel, and A. Loser. 2018. Challenges for toxic comment classification: An in-depth error analysis. In 2nd Workshop on Abusive Language Online, pages 33-42, Brussels, Belgium.
D. M. Blei, A. Y Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993-1022.
D. Borkan, L. Dixon, J. Sorensen, N. Thain, and L. Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In *Companion Proceedings of The 2019 World Wide Web Conference*, WWW '19, page 491-500. Association for Computing Machinery.
11The discussion title was used only by the human annotators that examined context, not by context-aware systems, which considered only the parent comment.
T. Davidson, D. Warmsley, M. Macy, and I. Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM, pages 512-515, Montreal, Canada.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, page 4171-4186, Minneapolis, MN, USA.
N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and N. Bhamidipati. 2015. Hate speech detection with comment embeddings. In ICWWW, pages 29-30.
R. Dror, G. Baumer, S. Shlomov, and R. Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In ACL, Melbourne, Australia.
M. ElSherief, V. Kulkarni, D. Nguyen, W. Y. Wang, and E. Belding. 2018. Hate lingo: A target-based linguistic analysis of hate speech in social media. arXiv preprint.
B. Gambäck and U. K. Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In 1st Workshop on Abusive Language Online, pages 85-90, Vancouver, Canada.
L. Gao and R. Huang. 2017. Detecting online hate speech using context aware models. In RANLP, pages 260-266.
Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In ACL, pages 2786-2791.
S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran. 2017. Deceiving Google's perspective api built for detecting toxic comments. arXiv preprint.
Y. Hua, C. Danescu-Niculescu-Mizil, D. Taraborelli, N. Thain, J. Sorensen, and L. Dixon. 2018. Wikiconv: A corpus of the complete conversational history of a large online collaborative community. arXiv preprint.
R. Kumar, A. K. Ojha, S. Malmasi, and M. Zampieri. 2018. Benchmarking aggression identification in social media. In TRAC, Santa Fe, USA.
S. Malmasi and M. Zampieri. 2017. Detecting hate speech in social media. In RANLP, pages 467-472.
T. Mikolov and G. Zweig. 2012. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234-239. IEEE.
H. Mubarak, K. Darwish, and W. Magdy. 2017. Abusive language detection on arabic social media. In 1st Abusive Language Workshop, pages 52-56, Vancouver, Canada.
C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang. 2016. Abusive language detection in online user content. In ICWWW, pages 145-153.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL, pages 311-318.
J. H. Park and P. Fung. 2017. One-step and two-step classification for abusive language detection on twitter. In 1st Workshop on Abusive Language Online, pages 41-45.
J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos. 2017a. Deep learning for user comment moderation. In 1st Workshop on Abusive Language Online, pages 25-35.
J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos. 2017b. Deeper attention to abusive user content moderation. In EMNLP, pages 1125-1135, Copenhagen, Denmark.
L. Posch, A. Bleier, F. Flock, and M. Strohmaier. 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint.
Y. Ren, Y. Zhang, M. Zhang, and D. Ji. 2016. Context-sensitive twitter sentiment classification using neural network. In 30th AAAI Conference on Artificial Intelligence.
B. Ross, M. Rist, G. Carbonell, B. Cabrera, N. Kurowsky, and M. Wojatzki. 2016. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. In NLP4CMC, Bochum, Germany.
M. D. Smucker, J. Allan, and B. Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In CIKM, pages 623-632. ACM.
A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J.-Y. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint.
Z. Waseem, T. Davidson, D. Warmsley, and I. Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In 1st Workshop on Abusive Language Online, Vancouver, Canada.
Z. Waseem and D. Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In *NAACL SRW*, pages 88-93, San Diego, California.
M. Wiegand, M. Siegel, and J. Ruppenhofer. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. In Proceedings of GermEval.
E. Wulczyn, N. Thain, and L. Dixon. 2017. Ex machina: Personal attacks seen at scale. In ICWWW, pages 1391-1399.
M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In NAACL.
M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019b. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In SemEval.
Z. Zhang, D. Robinson, and J. Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In Lecture Notes in Computer Science. Springer Verlag.
Annotators were asked to judge the toxicity of each comment, given the following definitions:
- VERY TOXIC: A very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective.
- TOXIC: A rude, disrespectful, unreasonable comment or otherwise somewhat likely to make a user leave a discussion or give up on sharing their perspective.
- UNSURE: Due to polysemy, lack of context or other reasons.
- NOT TOXIC: Not containing any toxicity.
For annotation, we used the 'Figure Eight' platform and we invested 5 cents per row. For the CAT-SMALL we employed high accuracy annotators (i.e., from zone 3), selected from 7 English speaking countries (i.e., UK, Ireland, USA, Canada, New Zealand, South Africa, Australia), and only ones allowing explicit content (we also warned about the explicit content in the title). 62 quiz questions were used. For the CAT-LARGE, we invested the same amount of money but all the annotators were able to participate (again, they wore warned for the explicit content). Inter annotator agreement was measured on the quiz questions with Krippendorff's alpha and was found to be  $70\%$  and  $72\%$  for the C and N sets.
GC annotators had one more question, which was asking them to compare the toxicity of the target comment to that of the parent comment. The main scope of that question was to make it less easy for annotators to ignore the parent comment.
B Hyper parameters
All systems were trained for 100 epochs with patience of 3 epochs. We performed early stopping by monitoring the validation ROC AUC.
BILSTM
The hidden size of the LSTM cells had size 128. We used batch size 128, max length 512, and we concatenated the forward and backward last hidden states before the FFNN. We used binary cross entropy for loss and Adam optimizer was used with default parameters (learning rate 1e-03).
CA-BILSTM-BILSTM
We used the same hyper-parameters with BILSTM but included one more bidirectional LSTM to encode the parent text. The parent biLSTM had 64 hidden nodes and we concatenated the forward and backward last hidden states. The parent and the target embeddings (the ones generated by the two biLSTMS) were concatenated before being passed to the FFNN.
We used a learning rate of 2e-05 for BERT and only unfroze the top three layers during training to our data. On top of the BERT [CLS] representation, we added a FFNN of 128 hidden nodes and a sigmoid to yield the toxicity probability. 128 tokens were used as maximum sequence length.
CA-SEP-BERT
A [SEP] token separated the two texts and the [CLS] token was used as with BERT. Same parameters with BERT were used.
CA-BILSTM-BERT
We used a bidirectional LSTM to encode the parent comment, similarly to CA-BILSTM-BILSTM. The biLSTM representation was concatenated with the [CLS] representation before the FFNN. All other parameters were set to the same values as BERT.
$^{12}$ https://www.figure-eight.com/