Target Span Detection for Implicit Harmful Content

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-26 09:12:30

Abstract: Identifying the targets of hate speech is a crucial step in grasping the nature of such speech and, ultimately, in improving the detec-tion of offensive posts on online forums. Much harmful content on online platforms uses implicit language – especially when targeting vulnerable and protected groups – such as using stereotypical characteristics instead of explicit target names, making it harder to detect and mitigate the language. In this study, we focus on identifying implied targets of hate speech, essential for recognizing subtler hate speech and enhancing the detection of harmful content on digital platforms. We define a new task aimed at identifying the targets even when they are not explicitly stated. To address that task, we collect and annotate target spans in three prominent implicit hate speech datasets: SBIC, DynaHate, and IHC. We call the resulting merged collection Implicit-Target-Span. The collection is achieved using an innovative pooling method with matching scores based on human annotations and Large Language Models (LLMs). Our experiments indicate that Implicit Target-Span provides a challenging test bed for target span detection methods.

Main paper: 
 introduction: 
 Social media provides a space where individuals can share their views openly and with a degree of anonymity. Yet, this has also led to the misuse of these platforms for disseminating harmful materials, including hate speech and toxic or offensive language. Numerous scholars are developing automated systems to identify and reduce the spread of such harmful materials. The majority of these methods rely on supervised classification methods [13, 21, 22] , which are usually black boxes [1, 2, 7] . These black box systems cause their classifications into the binary offensive or not to be less explainable for humans as to why certain content should be removed from platforms [11] . When content is classified as offensive or not, without any explanation, it leaves people unsure why the content should be removed [11] . One of the important aspects of explainable hate-speech detection is identifying the targeted groups in the given content, the focus of this work.Although the literature has explored identifying hate speech targets using clear identifiers like actual names and visible traits, the implicit targeting of protected groups has received less attention. The task is challenging due to indirectness, metaphors, and stereotypes [9] . This phenomenon may only be perceptible and recognizable by the community that is targeted due to its subtlety and target-specific nuances, thereby remaining undetected by automated systems and other individuals [15] . It is crucial to detect the targeted spans in a context due to the need for interpretable and explainable hate speech models in this area. One of the applications of target span detection is to improve the explainability and interpretability of hate speech models by providing information about the words or phrases that helped with the model decision. Moreover, annotating and training target span detection datasets and models can contribute to the limited vocabulary in this area to identify different aspects of implicit hate speech in context [1] . To expand research in this area, we introduce the implicit Target Span Identification task (iTSI), which requires a model to identify target spans in harmful content, whether explicit or implicit. An example of this task is given in Figure 1 .To study the iTSI task we need an annotated training dataset. A few earlier studies introduced hate-speech datasets that also identify the target in the content. For instance, HateXplain [12] is a hate-speech dataset with the names of the targeted groups identified. Calabrese et al. [3] published a dataset with annotated spans for target, derogation, protected characteristics, etc. Barbarestani et al. [1] provided an annotation mechanism for highlighting target spans in toxic language. However, none of these studies annotated target spans that are implicit, focusing only on those that are explicitly presented by name.In this study, we introduce a challenging dataset, Implicit-Target-Span (ITS), that captures both implicit and explicit target spans. It is built on three existing implicit hate speech datasets: Dyna-Hate [19] , IHC [8] , and SBIC [17] . To avoid the cost of exhaustive manual annotation, we explore a novel blending of human and LLM techniques. We ask annotators to tag a small portion of the dataset and then use LLMs through pooling technique [10] popularized by TREC [20] to choose the closest strategy to the human performance using a new evaluation metric. Our new ITS dataset comprises 57k annotated samples with an average number of around 1.7 target spans (targets of hate speech) per sample. Although previous IHC and SBIC datasets, identify target groups, they do not highlight any text span. ITS enhances this by marking both explicit and implicit references to targets. For instance, in Figure 1 "songwriters" is an explicit target, while "piano brains" is an implicit reference to songwriters. We discovered approximately 19,000 unique targets across IHC and SBIC datasets, a significant increase from the earlier figure of about 1,000 targets in the original dataset implying that there are nearly 20 times more implicit targets than before.We establish a performance baseline on our dataset using a BIO tagging approach [16] . We evaluate this baseline on various transformer-based encoders. Our results and analysis show the importance of iTSI and the new dataset to support significant research on target span identification in implicit hate speech. Our contributions include (1) introducing and formalizing a new target span identification task (iTSI), (2) a novel pooling-based annotation protocol to annotate both implicit and explicit target spans to create ITS, (3) establishing strong baselines on ITS, and (4) identifying challenges and suggesting improvements in building future approaches for this task. 
implicit target span identification: 
 Given content 𝐶 represented as a sequence of tokens 𝐶 = [𝑡 1 , . . . , 𝑡 𝑛 ], where 𝑡 𝑖 denotes the 𝑖-th token and 𝑛 is the total number of tokens in the content, the iTSI task is to identify token spans within 𝐶 that target a protected group. The output of this system is a set of non-overlapping tuples 𝑆, where each tuple (𝑠 𝑠𝑖 , 𝑠 𝑒𝑖 ) corresponds to the start and end indices of a detected target span within 𝐶. Thus, 𝑆 = {(𝑠 𝑠1 , 𝑠 𝑒1 ), (𝑠 𝑠2 , 𝑠 𝑒2 ), . . . , (𝑠 𝑠𝑘 , 𝑠 𝑒𝑘 )}, with 𝑠 𝑠𝑖 and 𝑠 𝑒𝑖 denotes the start and end token indices of the 𝑖-th detected target span, and 𝑘 represents the total number of detected target spans. The function 𝑓 (𝐶) → 𝑆 maps the content 𝐶 to the set of target spans 𝑆, identifying the token sequences within 𝐶 that target a protected group.For evaluating a model's ability to correctly identify target spans, the ground truth is similarly denoted as a set of non-overlapping tuples representing the actual start and end indices of target token spans within the content, 𝐺 = {(𝑔 𝑠1 , 𝑔 𝑒1 ), (𝑔 𝑠2 , 𝑔 𝑒2 ), . . . , (𝑔 𝑠𝑚 , 𝑔 𝑒𝑚 )} where 𝑚 is the number of ground truth target spans. By comparing the output 𝑆 with 𝐺, we evaluate the performance of the model 𝑓 . 
dataset construction: 
 We have developed ITS, a dataset specifically for highlighting implicit target spans (as well as explicit targets), utilizing a method inspired by pooling. Our collected dataset consists of a total of 57k annotated samples. Statistics of ITS are given in Table 1 .To reduce the reliance on prompt and the LLM for weakly labeling the dataset, we utilize a pooling method to gather annotated target spans in an implicit hate speech corpus. This involves employing an array of high-quality Large Language Models (LLMs) with a handful of prompts that were found to be effective for this task and comparing their performance with human annotators to find the best prompt and LLM. We determine the most effective LLM and prompt combination by using our 𝐹 1 𝑀 metric, an F1 score of the precision and recall of the generated targets in terms of their adherence (match) to target spans previously annotated by human reviewers.After identifying the most effective combination of LLM and prompt, we proceed to use the selected pair to annotate the entire dataset. An overview of the process is given in Figure 2 .Definition. Let 𝐶 be the content that needs to be annotated, and 𝐺 be the ground truth target spans (section 2). Output target spans from the LLM are defined as 𝑂 = {(𝑜 𝑠1 , 𝑜 𝑒1 ), . . . , (𝑜 𝑠𝑛 , 𝑜 𝑒𝑛 )} with corresponding tokens for the 𝑖th span in 𝑂 defined as 𝑇 𝑂𝑖 = {𝑡 𝑜 𝑠𝑖 , . . . , 𝑡 𝑜 𝑒𝑖 } similarly for each 𝑖th span (of 𝑚) in 𝐺 we have 𝑇 𝐺𝑖 = {𝑡 𝑔 𝑠𝑖 , . . . , 𝑡 𝑔 𝑒𝑖 } 𝐹 1 𝑀 is defined as the harmonic mean of 𝑅𝑒𝑐 𝑀 and 𝑃𝑟𝑒𝑐 𝑀 , where:𝑅𝑒𝑐 𝑀 = 1 |𝐺 | 𝑚 ∑︁ 𝑖=1 𝑃𝑀 𝑟 (𝑖) and 𝑃𝑟𝑒𝑐 𝑀 = 1 |𝑂 | 𝑛 ∑︁ 𝑖=1 𝑃𝑀 𝑝 (𝑖) (1)Recall based partial match score is the proportion of the ground truth covered by the output:EQUATIONand precision based partial match score (𝑃𝑀 𝑝 ) is defined similarly as the proportion of the output that matches ground truth. 𝑃𝑀 𝑝 and 𝑃𝑀 𝑟 thus calculate the scores for both exact and partial matches. A partial match occurs for 𝑃𝑀 𝑟 if there is an output span that is contained partially within a ground truth span -and correspondingly for 𝑃𝑀 𝑝 , a partial match occurs if a ground truth span is contained partially in the output span.Human Annotation. Hate speech, even when implicit, can be troubling to encounter, so we drew three annotators, 𝐴 1 , 𝐴 2 , and 𝐴 3 , from researchers involved in this project. They were tasked with marking up 𝑁 = 50 samples that were randomly selected across the three datasets, distributed so the sub-collections were roughly balanced and so that targets were also roughly balanced.The process began with a thorough briefing on the task, followed by each annotator receiving the same 𝑁 samples and guidelines. The annotators then highlighted explicit and implicit target spans in each of the samples. The collective responses from all annotators were then compiled into the union of the highlighted spans, where overlapping responses were extended to the largest span that encompassed them all. The level of agreement among the annotators before combining their inputs is detailed in Table 3 . We use the Longest Common Subsequence (LCS) and Dice Coefficient (DSC) as metrics for measuring the inter-annotator agreement. Similar to Calabrese et al. [3] we do not use Kohen's Kappa agreement in this study as we are evaluating the agreement on the sequence of tokens rather than individual tokens. Moreover, the task requires a partial match score as well as an exact match score which is not possible with Kappa agreement.Pooling Approach. For selecting an optimal annotation system, we opt for 𝐾 prompts and 𝐷 LLMs as potential candidates. We aim to identify which LLM and prompt combination most closely aligns with human-level performance. To achieve this, we generate a pool of 𝐾 × 𝐷 outputs from these combinations and then rank the entire pool based on 𝐹 1 𝑀 . The combination yielding the highest score is then chosen as the superior annotation system.For this task we use four LLMs (GPT-3.5 turbo checkpoint, Vicuna-13B [5] , Llama2-13B chat version [18] and phi-21 . To choose the best LLM and prompt2 , we compare the human-annotated target spans for the same samples with the LLM outputs and rank the similarity of outputs to the aggregation of all the human responses using 𝐹 1 𝑀 . The prompts used in the experiment are as follows:Prompt 1: Given the text highlight or underline parts of the text that mention or refer to the specific target. The target is sometimes not explicitly mentioned and you have to look for parts that implicitly refer to the target.Prompt 2: The task is to highlight multiple text spans from the given input hate speech content that explicitly and/or implicitly mentions, refers to a specific protected group or their representation or characteristics that have been targeted.The output ranking scores are given in Table 2 . As can be seen, GPT-3.5 achieves the highest score on Prompt 1 followed by Vicuna-13B which is an open source LLM. Overall we can see that all LLMs have closer performance to the aggregations of human annotators with prompt 1 than prompt 2. Additionally, we calculate the 𝐹 1 𝑀 of each human annotator (𝐴 𝑖 ) for aggregated annotations and observe that GPT-3.5 has a very close performance to 𝐴 1 . We choose GPT-3.5 on prompt 1 for annotating the full dataset to create ITS. 
targetdetect: baseline model: 
 We adopt an existing model architecture to train ITS and evaluate it in in-domain and out-of-domain settings. We refer to this model as TargetDetect that takes content 𝐶 and predicts the target spans 𝑆. The TargetDetect is a sequence tagging framework, based on a pre-trained transformer-based encoder(e.g., BERT, RoBERTa, etc.). Each token within 𝐶 undergoes a classification process employing the BIO (Beginning, Inside, Outside) tagging scheme. This approach marks each token as the Beginning of a target span, Inside a span, or Outside the spans of interest. By adopting this tagging strategy, we train a model and experiment with different encoders to identify all the target spans in content. The training process for TargetDetect involves minimizing the error between the model's predictions 𝑆 with the correct answers 𝐺 using cross-entropy loss defined as 𝐿 𝐶𝐸 = -𝑁 𝑖=1 𝑦 𝑖 log 𝑝 𝑖 . 𝑦 𝑖 denotes the ground truth label for the 𝑖-th token in the BIO tagging scheme, and 𝑝 𝑖 is the predicted probability that the 𝑖-th token is the beginning of, inside, or outside a target span. 
experiments: 
 In our experiments, we evaluate the performance of TargetDetect on the cross-domain test dataset from our data collection (IHC, SBIC, and DynaHate) as well as PLEAD [3] 1 ) are 1.9(0.9) and 1.6(0.7), respectively.Encoders. To train TargetDetect we use three different transformer encoders: 1) Bert-Base [6] ; 2) RoBERTa-Large [23] ; and 3) Hate-BERT [4] , a BERT model pre-trained on hate data.Evaluation Metrics. Since we have trained a sequence tagging model target span detection, we use the seqeval [14] metric to evaluate different encoders. This metric calculates F1, Precision, Recall, and Accuracy based on the predicted tagging labels and ground truth labels for each token in the content.The overall token tagging classification results based on different encoders and test datasets are given in Table 4 . The best performance is consistently achieved using RoBERTa-Large encoder. Among the encoders, Bert-Base has the worst performance and Hate-Bert performs slightly better with the average F1 score gain of 4% for test set performance in in-domain trained models and 1.5% for training on all the training sets at once. Overall we see that training in all the domains consistently improves the performance compared to training on only an in-domain dataset.We evaluate the effectiveness of training the baseline model on different training samples including our proposed dataset ITS collection and human annotated PLEAD dataset. Performance comparison of TargetDetect with RoBERTa-Large encoder on ITS and PLEAD is shown in Table 5 . The diagonal shows the F1 score results for in-domain full finetuning whereas the rest are different combinations of training on one domain and testing in another (a zero-shot approach). RoBERTa-Large is used as an encoder in this experiment. The zero-shot experiment across ITS test sets results in a higher F1 score except when TargetDetect is trained on PLEAD. Training on ITS and testing on PLEAD results in only 5% performance loss on average in F1 score. The DynaHate-trained model has the least performance loss of only 2% (i.e., 51.0% with DynaHate-trained samples vs 52.0% with PLEAD-trained samples. This indicates that training target detection with ITS and testing it on external, human-annotated datasets can achieve performance similar to that of training on in-domain, human-annotated datasets, demonstrating ITS's effectiveness across various domains.We use RoBERTa-Large and IHC (finetuned on the IHC train set) for error analysis. From our quantitative analysis, we discovered that in 26.5% of the instances where errors occurred, at least one of the predicted spans within a piece of content exhibited a "boundary" error, meaning the predicted span partially overlaps with the actual span but falls short by several characters. For instance, whereas the correct span might be "European people, " the model might only identify "European" however, in approximately 90% of such cases, the most important words or phrases to detect the targeted group were present. The boundary errors occurred in the least important words such as stop words.Additionally, we observed that one of the common model failures is due to the discrepancy between the number of predicted spans and the number of ground truth spans. Considering the failed cases, roughly a third predicted too many spans, a third predicted too few, and a third predicted the right number. Missing or overgenerating are both important sources of error, suggesting the need for improvement.Our manual observation of the failed cases also showed interesting patterns that can be useful for future research. We identify a few main failure patterns: 1) Obfuscated targets where the model fails to correctly identify the target because the original word or phrase is obfuscated -e.g., "songwriter" is written as "s0ngwr1t3r"; and 2) Implicit and subtle references to targets where the trained model fails to recognize the target because lexically and semantically the original target is replaced with its indirect referencee.g., hiding the"songwriter" by substituting it with "conservatory graduates". 
conclusion: 
 In this paper, we introduce a reliable weak labeling approach using a novel pooling-based framework for the challenging implicit target span detection task iTSI. The generated dataset using this pooling approach will be released to enhance research in this area. We also introduce a baseline model, TargetDetect, to evaluate the task on ITS. Our results show that ITS trained model can achieve comparable performance to the model trained on the human-annotated dataset. We also show that ITS is a challenging dataset that presents several potential research directions. 
 
 Appendix: https://huggingface.co/microsoft/phi-2We release the code and data at https://suppressed.for.review. 


Decision: The paper describes the development of a dataset on implicit target detection in hate speech and contains some preliminary results of some baselines.

I suggest to reject the paper for ICTIR 2024 for the following reasons:
- the reviewers raise important open questions with respect to the creation of the dataset
- ICTIR does not really have a dedicated resource track and the connection of the paper's topic to IR is not that strong
-> a more fitting venue would be some NLP-related conference with a resource focus or a resource track

Reviews: review: main_review: main_review: The paper is generally well-written. However, there are several points that lack clarity. For example, it is not clear to me what proportion of the ground-truth actually expanded due to the use of LLMs to infer weak labels. Are these weak labels merged with the strong (human-created) ones?
A high value of an evaluation metric on an "expanded dataset" doesn't mean that a downstream model is good. All this means is that the in-domain and the out-domain sets aren't comparable because they use different evaluation sets.

It's a bit surprising that prompt-1 which has got no explicit mention of "hate" works better than prompt-2.
What's the basis of selecting two prompts that are quite different in its meaning? The second one explicitly states to find entities against which/whom hate is targeted.
It wasn't clear to me how is the notion of "protected group" defined or is even relevant to this work? In the example, is "Piano brain" a protected group?
main_review: main_review: The contribution is a preliminary study that may open up the possibility for future work on the subject. Considering the release of the resource I find it acceptable at the conference although the annotation methodology could have been more robust.
main_review: main_review: Based on the objective part of the review, I don't think that this paper should be accepted.
main_review: main_review: The paper was easy to follow and understand. The main contribution of the paper is the development of a dataset on implicit target detection in hate speech. The authors also apply different baselines on the problem and evaluate their performance. Although the task is relevant and important, there are some limitations (see my previous comments) in the annotation process that need to be clarified.