Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduces a new dataset (of about 5K tweets) aimed at training and evaluating hate speech classifiers. The tweets of the dataset were selected via a « pooling » strategy, i.e. by merging the predictions of several models trained on previously existing hate speech datasets. It was then annotated manually based on Amazon Mechanical Turk. Additionally, the paper evaluates a posteriori in what measure an active learning strategy would have been more efficient than this pooling approach. 

strengths: - the limitations of existing hate speech datasets are well explained, in particular the lack of diversity of datasets aggregated through a keyword-based approach
- the paper contains interesting methodological considerations on how to create a more diverse dataset in an efficient way. In particular, it shows the benefit of using active learning over the pooling approach.
- experiments shows that increased diversity also comes with lower performance

weaknesses: - the added value of the proposed dataset over the previous one of Founta et al. [31] is not clear. The ratio of tweets with no known hate words is admittedly higher, but in the absolute, since the data set of Founta et al. is much larger, it still contains more of such « diverse » tweets (about 478 tweets vs. 98 in the proposed dataset according to the numbers provided in Table 2 of the paper. The dataset of Founta et al. could thus be re-sampled to reach a better prevalence and relative coverage. 
- despite the fact that the authors discuss the limitation of pooling vs. active learning, the proposed dataset IS built using a pooling strategy. It can therefore theoretically not really contain more diversity than the set of datasets on which the models of the pool have been trained 
- There is insufficient documentation about several aspects of the dataset (see documentation section this review)

correctness: The claims are globally correct but somewhat overestimated
clarity: the paper is well written

relation to prior work: it is sufficiently discussed how this work differs from previous contributions
additional feedback: About subjectivity vs. ambiguity: it is true that subjectivity may partly explain the observed uncertainty but I think the annotator’s disagreement may also be due to some forms of aleatoric uncertainty, i.e. an intrinsic uncertainty that leads to a non null p(y|x) for a given tweet x.  For instance, a tweet might be intrinsically ambiguous because of some missing information about the context, the author, etc. and the conditional probability might be equal to e.g. p(y|x)=0.6. In such cases, some annotators will say y=1 while others will say y=0 not because they bring a subjective judgement but rather because they have to chose randomly between 1 and 0 . It is very important to keep such ambiguous tweets in the training set because the ambiguous region around the theoretical decision frontier is the most difficult to learn. With this in mind, the proposed approach to select samples (based on pooling or  most positives » samples) might be problematic. Indeed, they are likely to not select such ambiguous tweets and focus on the less ambiguous ones. Classifiers trained on this type of data will then tend to make « black-and-white fallacies » when they encountered ambiguous tweets.  

Minors:
«  identifies langauge » —>«  identifies language »
review_title:  Interesting methodological considerations but a somewhat disappointing dataset in the end

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: A new dataset for hate speech detection is presented. In contrast to prior works that are collected with a keyword based approach, this dataset is collected with a pooling based approach. This results in a dataset that contains hateful tweets without containing a predefined set of hateful words (in contrast to prior works), making the dataset highly relevant and closer to real-world applications. The authors exemplifies this by showing that SOTA methods perform poorly on this dataset, especially for detecting hateful tweets that does not contain hateful words (F1 < 27 %). Furthermore, the authors present an Active Learning strategy and shows that it could be very useful to create hate speech datasets in the future.
strengths: * Hate detection is a challenging, yet very relevant/important topic with positive social implications.
* Existing datasets are created from a predefined sets of words, whereas the proposed dataset, consist of hateful tweets with and without hateful words - making it more diverse and closer to real-world applications.
* The authors present and Active Learning strategy for creating new datasets for hate detections and show that it is very efficient (in terms of annotation costs). This will pave the way for researchers to create larger datasets at reasonable costs. 
weaknesses: * The dataset is rather small. It consists of only 10k tweets, whereas the largest existing dataset has 90k tweets. However, I still think the dataset is relevant as it is of higher quality (to the best of my knowledge)
* As the authors already point out the proposed pooling method is biased by existing datasets. They proposed a stratified sampling to reduce this bias.
correctness: To the best of my knowledge, the dataset is constructed in a well considered and correct way. A 20/80 test/train division is performed and sampled with stratified sampling, which ensures that each category of hate tweets are considered in both test/train set. Reasonable baseline methods are chosen.
clarity: The paper is clear and well written. There are some small typos (I would recommend pasting the text in Google docs and use their spelling control to correct these). Examples:

In the abstract: "  for hate speech To int " => "  for hate speech. To int"
In the intro: " biased dataset whose whose distribution" => " biased dataset whose distribution" 

Also, I found the following sentence unclear. Please reformulate:

"Train and test set splits contain 3,779 and 637 / 309 tweets, respectively."



relation to prior work: Yes, this is quite clear. One thing that could be improved is to train the methods on an existing dataset and see how well the performance is on the proposed dataset. As I understand, in the current experimental setup, you train/test only on the proposed dataset. It is interesting to understand if the performance of the models are improved by a bigger training set (of lower quality) or if the lower quality degenerates the performance when tested on your dataset.
additional feedback: See above comments.
review_title: An Information Retrieval Approach to Building Datasets for Hate Speech Detection

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper proposes a new dataset for hate speech detection. The authors use two different methodologies, pooling and active learning, to derive a dataset consisting of tweets annotated for hate speech. This approach builds upon datasets collected by several other scholars. This dataset is intended to serve as a more reliable benchmark for hate speech detection models.
strengths: - The paper is rigorous and it is clear that the authors have a strong grasp of the limitations of current work in this area.
- The authors cite most of the relevant literature.
- This dataset will be useful for researchers working in this area.
- The authors carefully consider the ethical implications of using crowdsourced hate speech annotations.
weaknesses: - Some of the examples used are questionable. For example, Table 1 contains the following example of "hate speech not containing hate words": “You can yell at these Libs all day. They don’t listen. But if you ridicule them nationally they can’t take it". This is clearly a derogatory comment towards a political opponent (presumably a conservative criticizing liberals), but this seems very different to other examples here. Just criticizing a political opponent using a nickname is not hate speech. This raises some concerns about the validity of the annotated data.
- I was surprised that the authors develop a binary hate speech classifier given that much of the previous work in the area has criticized such approaches. The authors mention this in the Appendix and state it is a limitation and how it caused problems in the present study (pornographic references misclassified as hate speech). I would like to see a better justification of the binary classifier approach and an analysis of how this approach performs when applying it to data with multiple labels (e.g. Founta et al's dataset).
- Like much of the research on online hate speech detection, this paper uses data from Twitter. It is not clear that this benchmark will be useful for models trained on other datasets. The contribution is therefore more limited than stated: this is a benchmark for Twitter hate speech detection models.
correctness: I am satisfied with the correctness of the work presented.
clarity: Yes
relation to prior work: Yes
additional feedback: N/A
review_title: Improving hate speech detection datasets

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The article describes the annotation of a new hate speech dataset (9,667 tweets in total) with the objective of addressing the limitations of keyword-based filtering methods for collecting hate speech data. As a solution, methods for information retrieval are employed.
The annotation process starts by searching a random sample of social media postings. The data is subsequently chosen via pooling and active learning. The data is then annotated by untrained Mechanical Turk crowdworkers. To help workers improve their annotation quality, the article applies task decomposition to hate speech and proposes to measure annotator self-consistency. A benchmarking is performed using the new data and models based on LSTM, BiLSTM, and BERT.




strengths: To address some of the limitations of prior research, an alternative and viable technique is utilized. This method is based on the Information Retrieval method, which is an approach that has been neglected in the past. Additionally, this research used task decomposition, which may assist in the annotation of hate speech messages and improve annotator agreement in future investigations.
weaknesses: Annotation guidelines could be improved as they induce some behavior in the annotators.  First, guidelines request that annotators must identify both the targets and the actions related to hate speech otherwise, their work will be rejected. Nevertheless, not all hostile texts include one or both of these components. Second, also highlighting all words in the text will yield rejection, which may also lead users to not annotate all the desired content as they would wish.

Despite previous studies warning (Arango et al. [3]), this study does not provide statistics on the data authors. It is critical to have an anonymised id for the message's author, so that data from the same user may only be utilized for training or testing purposes. Statistics such as the total number of users and the total number of messages sent by each user should be given - either as the distribution of this value within the sample or as the mean and standard deviation.

It is not clear how both pooling and active learning are combined in order to choose data for annotation.

Limited self-consistency metric: this was measured by when there's target highlight the author should mark the message as containing hate speech. This happened 94.5% of the time. However, there are hate speech messages that don’t contain direct targets, for instance: “Not all human beings deserve same rights. Some are closer to behave like animals.” Additionally, the authors discuss how self-consistency may be utilized when annotator agreement is low. Both, however, relate to distinct constructs.

It is unclear which questions were used to calculate the annotator agreement. We have a binary metrics for hate speech in this job, but we also have annotation at the word level, for which it would be beneficial to understand how agreement is calculated. Additionally, it is unclear how the responses of three annotators result in the annotation of the message for target and action.

correctness: Yes.
clarity: The paper is well written.
relation to prior work: It discusses the comparison with previous works.
additional feedback: Few corrections:

“representative coverage for cost-savings, yielding a biased dataset *whose* whose distribution diverges”

“incites *violates* against a demographic group, and identifies that target group”

In more than one case appears:
“For a tweet, if an annotator identifies *langauge* that is derogatory toward or”
review_title: A helpful new dataset on hate speech that still has room for improvement

Meta-review: This paper presents a benchmark dataset for hate speech detection, using an IR-based approach. Reviewers found the contribution compelling, though questions several aspects of novelty with respect to previous datasets, and methodology.
Decision: Accept