Assessing Public Hate Speech Datasets: A Comprehensive Evaluation
Authors: Sardar Jaf, Basel Barakat


Reviewer paper summaries: The paper attempts to provide a comprehensive evaluation of current hate speech detection datasets. Specifically, it examines ten different datasets, conducting consistent experiments involving text normalization, label standardization, and dataset balancing. Additionally, the study includes a few statistical analyses to generate some insights. However, the paper has notable shortcomings: it lacks completeness and evaluation across different ML models / LLMs, does not experiment with newer datasets, and contains significant technical gaps and writing deficiencies. I strongly encourage the authors to review the weakness and comments section in detail and consider incorporating the provided suggestions to bolster the paper’s overall strength.
Summary of strengths: 1. The authors analyze 10 popular hate speech benchmarks and perform analysis by standardizing the text normalization/ cleaning, labels, and distribution.
2. They perform statistical analysis on these datasets, shedding light on the learning patterns of the ML models used in the experiments.
Summary of weaknesses: 1. As the paper positions itself as a position paper, it contains far fewer citations. The paper needs more coverage of relevant literature in the related works. Hate speech detection is an active research area, and numerous pertinent studies have been published in recent years. I have listed some missing references in the next section.
2. All the datasets included in the study are pre-2021. The authors should include more recent and challenging datasets like Latent Hatred (https://arxiv.org/pdf/2109.05322), ToxiGen (https://arxiv.org/pdf/2203.09509), and GotHate (https://arxiv.org/pdf/2306.01105) that are more complex and provide a more real-world distribution of online hate.
3. The authors limit their experimentation to BERT and Logistic Regression, which provides a myopic perspective on the capabilities of current language models. Other potent general-purpose LLMs and domain-specific open-sourced models should have been considered for this study.
4. The authors do not conduct extensive experiments using different seed and hyper-parameter values to assess the robustness of the model's predictions.
5. The dataset analysis lacks completeness. The authors conduct surface-level analyses such as label-based data rebalancing, label standardization, and data normalization, which provide little insight. Experiments similar to Kulkarni et al., 2023 (https://arxiv.org/pdf/2306.01105), such as inter-class similarity, adversarial data validation, analyzing the difficulty of the dataset, etc, would help improve the quality of the paper.
Comments, suggestions and typos: **Suggestions and Typos:**

1. It would be helpful to have some citations in the introduction.
2. Line 141 — banarized should be binarized

**Missing References:**

1. Fatimah Alkomah and Xiaogang Ma. 2022. A Literature Review of Textual Hate Speech Detection Methods and Datasets. Information 13, 6 (2022). https: //doi.org/10.3390/info13060273 
2. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. Pro- ceedings of the International AAAI Conference on Web and Social Media 11, 1 (May 2017),512–515. https://ojs.aaai.org/index.php/ICWSM/article/view/14955 
3. Pinkesh Badjatiya,Shashank Gupta, Manish Gupta,and Vasudeva Varma. 2017. Deep Learning for Hate Speech Detection in Tweets. In Proceedings of the 26th International Conference on World Wide Web Companion (Perth, Australia) (WWW ’17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 759–760. https://doi.org/10.1145/3041021. 3054223Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 1664–1674. https://doi.org/10.18653/v1/D19-1176
4. Atharva Kulkarni, Sarah Masud, Vikram Goyal, and Tanmoy Chakraborty. 2023. Revisiting Hate Speech Benchmarks: From Data Curation to System Deployment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '23). Association for Computing Machinery, New York, NY, USA, 4333–4345. https://doi.org/10.1145/3580305.3599896 
5. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 
6. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–3326, Dublin, Ireland. Association for Computational Linguistics. 
7. Binny Mathew, Anurag Illendula, Punyajoy Saha, Soumya Sarkar, Pawan Goyal, and Animesh Mukherjee. 2020. Hate begets Hate: A Temporal Study of Hate Speech. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 92 (October 2020), 24 pages. https://doi.org/10.1145/3415163
Overall assessment scores: 1.5