SlowGuess/NAACL_2022_OCR huggingface dataset:
pages 859-866, Reykjavik, Iceland. European Language Resources Association (ELRA).
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10, Valencia, Spain. Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 599-605, Melbourne, Australia. Association for Computational Linguistics.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Introducing CAD: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289-2303, Online. Association for Computational Linguistics.
Kenneth D Ward. 1997. Free speech and the development of liberal virtues: An examination of the controversies involving flag-burning and hate speech. University of Miami Law Review, 52(3):733-792.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88-93, San Diego, California. Association for Computational Linguistics.
Ziqi Zhang and Lei Luo. 2019. Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web, 10(5):925-945.
Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, and Isabelle Augenstein. 2018. Discourse-aware rumour stance classification in social media using sequential classifiers. Information Processing & Management, 54(2):273-290.
We show a screenshot of the annotation interface in Figure 2.
B Detailed Results
Table 9 presents detailed results complementing Table 6 in the paper. We provide Precision, Recall and weighted F1-score using each related task for pretraining when the input is Target and Parent_Target respectively in Table 9.
Here here. I'm short and the exit row is still better. Being tall doesn't make you entitled
to it because you think I'm just as comfortable in those shit cramped seats.
Identify hate speech and counter-hate speech
Title: The ignorance and loathing is real
The comment posted by User_B is:
Instructions:
In this task, you will see a short Reddit conversation between User_A and User_B. The conversation will include the title of the discussion and one comment by each user. Your job is to label the comment by
User_B as Hate, Counter-hate or Neutral. You will follow these steps: 1. First, see the title of the discussion.
2. Second, click a button to see the comment by User_A, and then click another button to see the comment by User_B.
3. Third, label the comment by User_B as Hate, Counter-hate or Neutral.
Definitions and Examples (click HERE to see more details and examples):
[Hate] User. B attacks an individual or a group with the intention to vilify, humiliate, or incite hatred in two ways:
1. User_B may explicitly express hate in the comment
2. User_B may agree with the hate expressed by User_A
[Counter-hate] User_B challenges, condemns the hate expressed by User_A where:
1. User_A is considered as hate
2. User_B counters User_A
[Neutral] User_B neither conveys hate nor counters hate expressed by User_A
User_B:
You're an angry little guy aren't you?
Hate
Counter-hate
Read the Reddit conversation and answer the question about the comment by User_B:
Figure 2: Screenshot of the annotation interface. The left panel displays the instructions and examples. The right panel displays the Parent and the Target to be annotated.
User_A:
Neutral
Hate	Counter-hate	Neutral	Weighted Average
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Majority Baseline	0.00	0.00	0.00	0.00	0.00	0.00	0.51	1.00	0.67	0.26	0.51	0.34
Trained with ...
Target	0.56	0.55	0.56	0.41	0.36	0.38	0.67	0.71	0.69	0.58	0.59	0.58
+ Hate_Twitter	0.58	0.53	0.55	0.46	0.07	0.12	0.61	0.88	0.72	0.57	0.6	0.54
+ Hate_Redit	0.57	0.52	0.55	0.44	0.32	0.37	0.64	0.75	0.69	0.58	0.59	0.58
+ Sentiment	0.59	0.47	0.53	0.00	0.00	0.00	0.59	0.92	0.72	0.45	0.59	0.50
+ Sarcasm	0.59	0.51	0.55	0.50	0.04	0.08	0.59	0.51	0.55	0.57	0.58	0.51
+ Stance	0.56	0.55	0.56	0.51	0.41	0.45	0.68	0.74	0.71	0.61	0.61	0.61
Trained with ...
Parent_Target	0.55	0.62	0.59	0.52	0.38	0.44	0.68	0.72	0.70	0.61	0.62	0.61
+ Hate_Twitter	0.49	0.64	0.56	0.29	0.13	0.18	0.66	0.73	0.7	0.53	0.57	0.54
+ Hate_Redit	0.55	0.64	0.59	0.48	0.33	0.39	0.69	0.73	0.71	0.61	0.62	0.61
+ Sentiment	0.53	0.59	0.56	0.40	0.23	0.29	0.68	0.77	0.72	0.57	0.60	0.58
+ Sarcasm	0.56	0.54	0.55	0.45	0.09	0.15	0.62	0.86	0.72	0.56	0.60	0.54
+ Stance	0.55	0.66	0.60	0.54	0.43	0.48	0.71	0.70	0.71	0.63	0.63	0.63
Table 9: Detailed results (P, R, and F) predicting whether the Target is Hate, Neutral or Counter-hate when the input is only the Target or the Parent_Target. These results are using RoBERTa and pretrained with each related task. This table complements Table 6 in the paper.
Hate	Counter-hate	Neutral	Weighted Average
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Majority Baseline	0.00	0.00	0.00	0.00	0.00	0.00	0.51	1.00	0.67	0.26	0.51	0.34
Trained with Target + Silver + Related Task
Mean	0.56	0.54	0.55	0.48	0.46	0.47	0.67	0.71	0.70	0.60	0.60	0.60
(SD)	0.04	0.05	0.01	0.01	0.05	0.03	0.01	0.04	0.01	0.00	0.01	0.01
Trained with Parent_Target + Silver + Related Task
Mean	0.55	0.6	0.59	0.51	0.49	0.50	0.72	0.72	0.72	0.64	0.64	0.63
(SD)	0.03	0.04	0.01	0.04	0.05	0.02	0.02	0.06	0.02	0.00	0.01	0.01
Table 10: Detailed results (P, R, and F) predicting whether the Target is Hate, Neutral or Counter-hate when the input is only the Target or the Parent_Target. The results are using both Silver and pretraining on related tasks. We experiment with multiple runs using different random seeds and report the mean scores and their standard deviation.
Table 10 presents the mean scores of Precision, Recall and weighted F1-score and their standard de
viation when we use both Silver data and pretraining on related tasks with different random seeds.
Epochs	Batch size	Learning rate	Dropout
Target	5	16	1e-5	0.5
+ Silver	2	16	1e-5	0.5
+ Related task	2	8	1e-5	0.5
+ Silver + Related task	4	16	1e-5	0.5
Table 11: Hyperparameters used to fine-tune RoBERTa individually for each training setting. We accept default settings for the other hyperparameters as defined in the implementation by Pruksachatkun et al. (2020).
The results are consistent with the findings in our study: adding the Parent does improve the performance compared to the system that does not (0.63 vs. 0.60).
C Hyperparameters to Fine-tune the Systems
The neural model takes about half an hour on average to train on a single GPU of NVIDIA TITAN Xp. We use an implementation by Pruksachatkun et al. (2020) and fine-tune the RoBERTa (base architecture; 12 layers) (Liu et al., 2019) model for each of the four training settings. For each setting, we set the hyperparameters to be the same when the textual input is Target and Parent_Target respectively. Hence we only report tuned hyperparameters for each setting when the input is Target in Table 11.