STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection
Authors: Zewen Bai, Yuanyuan Sun, shengdi yin, Junyu Lu, Jingjie Zeng, Haohao Zhu, Liang Yang, Hongfei Lin
Accepted at: acl25

STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection

Zewen Bai*a, Liang Yang*a,b, Shengdi Yin*a, Junyu Lu*a, Jingjie Zenga
Haohao Zhu*a, Yuanyuan Sun*a, Hongfei Lin*a
aSchool of Computer Science and Technology,
bKey Laboratory of Social Computing and Cognitive Intelligence
Dalian University of Technology, China
dlutbzw@mail.dlut.edu.cn, (liang,hflin)@dlut.edu.cn

Abstract

The proliferation of hate speech has caused significant harm to society. The intensity and directionality of hate are closely tied to the target and argument it is associated with. However, research on hate speech detection in Chinese has lagged behind, and existing datasets lack span-level fine-grained annotations. Furthermore, the lack of research on Chinese hateful slang poses a significant challenge. In this paper, we provide two valuable fine-grained Chinese hate speech detection research resources. First, we construct a Span-level Target-Aware Toxicity Extraction dataset (STATE ToxiCN), which is the first span-level Chinese hate speech dataset. Secondly, we evaluate the span-level hate speech detection performance of existing models using STATE ToxiCN. Finally, we conduct the first study on Chinese hateful slang and evaluate the ability of LLMs to understand hate semantics. Our work contributes valuable resources and insights to advance span-level hate speech detection in Chinese. 1

Disclaimer: The samples presented by this paper may be considered offensive or vulgar.

1 Introduction

With the popularity of social media, user-generated content has experienced explosive growth, and hate speech has also flourished. Hate speech refers to harmful statements that express hatred or incite harm against specific groups or individuals based on race, religion, gender, region, sexual orientation, physical characteristics, and other factors (Bilewicz and Soral, 2020). Due to its damaging impact on individuals and society, hate language is now widely considered as a problem of increasing importance (Silva et al., 2016). Recently, researchers have been actively engaged in the study of hate speech detection, and this research has gradually shifted from post-level (Ahn et al., 2024; AlKhamissi et al., 2022b) to span-level (Pavlopoulos et al., 2021; Mathew et al., 2021; Zampieri et al., 2023).

Approximately 941 million people, or 12% of the global population, speak Mandarin Chinese as their first language (Eberhard et al., 2024). However, research on Chinese hate speech detection lags significantly behind. There are two key issues that remain unresolved. Firstly, existing research (Deng et al., 2022; Jiang et al., 2022; Zhou et al., 2022) is limited to the post-level, leaving span-level Chinese hate speech detection unexplored. The intensity and directionality of hate speech are closely tied to the target and argument it is associated with (Cowan and Hodge, 1996). In the Chinese linguistic context, challenges in span-level hate speech detection arise due to flexible word order and the absence of word delimiters like spaces. These issues complicate the identification of targets and arguments. For example, as shown in Exp.1 in Table 1, the use of inversion in Chinese disrupts the conventional subject-verb-object structure. To fill this gap, we construct a span-level Chinese hate speech dataset by annotating the Target-Argument-Hateful-Group quadruples in posts.

Secondly, although Chinese hate lexicons have been developed (Jiang et al., 2022; Lu et al., 2023), the absence of interpretable annotations hinders a deeper understanding of hate speech. As an ideographic language, Chinese has a wealth of synonyms and near-synonyms (Mair, 1991), making the forms and types of hateful slang diverse and difficult to capture. Chinese hateful slang often evades model detection through techniques such as homophonic substitution, character splitting and merging, and historical allusions (Xiao et al., 2024). For example, in Exp. 3 from Table 1, the target uses the merging technique to evade detection. To address this issue, we collect commonly used hateful slang from real-world online forums, providing
<table>
  <tr>
    <th>Exp.</th>
    <th>Post</th>
    <th>Target</th>
    <th>Argument</th>
    <th>Hateful</th>
    <th>Group</th>
  </tr>
  <tr>
    <td>1</td>
    <td>你这头蠢驴，没人会喜欢。<br><i>No one will ever like you, you idiot.</i></td>
    <td>你<br><i>you</i></td>
    <td>蠢驴<br><i>idiot</i></td>
    <td>non-hate</td>
    <td>non-hate</td>
  </tr>
  <tr>
    <td>2</td>
    <td>男同是艾滋高发群体。<br><i>Gay people are a high-risk group for HIV.</i></td>
    <td>男同<br><i>Gay people</i></td>
    <td>艾滋高发群体<br><i>a high-risk group for HIV</i></td>
    <td>hate</td>
    <td>LGBTQ. others</td>
  </tr>
  <tr>
    <td>3</td>
    <td>默我是真的很讨厌。<br><i>Silence, I really hate it.</i></td>
    <td>默(黑犬)<br><i>Silence(black dog)</i></td>
    <td>讨厌<br><i>hate</i></td>
    <td>hate</td>
    <td>Racism</td>
  </tr>
</table>

Table 1: Examples of annotated posts from STATE ToxiCN dataset with corresponding annotations of Target-Argument-Hateful-Group quadruples.

detailed annotations to construct the first annotated lexicon for Chinese hateful slang. This lexicon serves as a foundational resource for research into the understanding of Chinese hate semantics.

To address the lack of resources for span-level Chinese hate speech research, we introduce a Span-level Target-Aware Toxicity Extraction dataset (STATE ToxiCN), a novel dataset containing 8029 posts and 9533 Target-Argument-Hateful-Group quadruples addressing sexism, racism, regional bias, and anti-LGBTQ sentiments. Using this dataset, we evaluate the performance of LLMs in span-level Chinese hate speech detection. Specially, we annotate: 1) extraction of the targets and arguments from the post, 2) determination of whether each Target-Argument pair constitutes hate speech, and 3) classification of the groups for hateful Target-Argument pairs.

Moreover, we summarize and annotate commonly used hateful slang from real-world online forums to address the challenges of identifying hateful slang. We compile a comprehensive annotated lexicon, labeling each hateful slang with its frequent groups and providing explanations of its usage and contextual meaning. This is the first annotated lexicon of Chinese hateful slang with interpretative annotations. This resource not only helps to understand how hateful slang is used to disguise hate speech, but also provides valuable annotated data to evaluate and improve the ability of LLMs to detect hateful slang. The main contributions of this work are summarized as follows:

• We provide a span-level target-aware toxicity extraction dataset, containing 8k posts and 9.5k quadruples, filling the gap in span-level resources for Chinese hate speech detection.

• We construct an annotated lexicon of commonly used hateful slang in real-world online forums to evaluate LLMs’ capability in understanding Chinese hate speech semantics.

• We evaluate models on STATE ToxiCN, assessing their span-level performance and ability to detect hateful slang, highlighting key challenges and insights for improvement.

2 Related Work

Hate Speech Detection. Hate speech detection is a critical task in Natural Language Processing (NLP) that has attracted considerable attention recently. Researchers have increasingly turned to pre-trained models to address this issue (Caselli et al., 2020; Hanu and Unitary team, 2020; Zhou et al., 2021; AlKhamissi et al., 2022a; Ali et al., 2022). To facilitate progress in this field, several datasets tailored to hate speech detection have been developed (Waseem and Hovy, 2016; Davidson et al., 2017; Founta et al., 2018; Hartvigsen et al., 2022). Pavlopoulos et al. (2021) introduced span-level hate speech detection, while the TBO dataset advanced the field by pioneering the extraction of Target-Argument-Harmful triples (Zampieri et al., 2023). However, Chinese hate speech detection remains significantly underdeveloped.

Chinese Hate Speech Dataset. While some Chinese hate speech datasets exist, these efforts remain limited to the post-level. TOCP and TOCAB, derived from Taiwan’s PTT platform, focus on detecting profanity and abusive language (Chung and Lin, 2021). The Sina Weibo Sexism Review (SWSR) centers on identifying sexism (Jiang et al., 2022). The Chinese Offensive Language Dataset (COLD) categorizes sentences into types such as individual attacks and anti-bias (Deng et al., 2022). Zhou et al. (2022) introduce CDial-Bias, the first annotated dataset specifically addressing social bias in Chinese dialogues. Lu et al. (2023) present ToxiCN, a dataset encompassing both explicit and implicit toxic language samples. Xiao et al. (2024) introduce a dataset to evaluate LLMs’ robustness against cloaking perturbations.
<table>
  <tr>
    <th>Work</th>
    <th>Platforms</th>
    <th>Language</th>
    <th>#Posts</th>
    <th>Span</th>
    <th>Tar.</th>
    <th>Arg.</th>
    <th>Group</th>
    <th>Lex.</th>
  </tr>
  <tr>
    <td>Davidson et al. (2017)</td>
    <td>Twitter</td>
    <td>English</td>
    <td>24,802</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Founta et al. (2018)</td>
    <td>Twitter</td>
    <td>English</td>
    <td>80,000</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Toxic Spans (Pavlopoulos et al., 2021)</td>
    <td>Civil Comments</td>
    <td>English</td>
    <td>10,629</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HateXplain (Mathew et al., 2021)</td>
    <td>Twitter, Gab</td>
    <td>English</td>
    <td>20,148</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TBO (Zampieri et al., 2023)</td>
    <td>Twitter</td>
    <td>English</td>
    <td>4,673</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>COOLD (Deng et al., 2022)</td>
    <td>Zhihu, Weibo</td>
    <td>Chinese</td>
    <td>37,480</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>SWSR (Jiang et al., 2022)</td>
    <td>Weibo</td>
    <td>Chinese</td>
    <td>8,969</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cdial-Bias-Utt (Zhou et al., 2022)</td>
    <td>Zhihu</td>
    <td>Chinese</td>
    <td>13,394</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cdial-Bias-Ctx (Zhou et al., 2022)</td>
    <td>Zhihu</td>
    <td>Chinese</td>
    <td>15,013</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TOXiCN (Lu et al., 2023)</td>
    <td>Zhihu, Tieba</td>
    <td>Chinese</td>
    <td>12,011</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TOXiCLOAKCN (Xiao et al., 2024)</td>
    <td>Zhihu, Tieba</td>
    <td>Chinese</td>
    <td>4,582</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>STATE TOXiCN (Ours)</td>
    <td>Zhihu, Tieba</td>
    <td>Chinese</td>
    <td>8,029</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Comparison of hate speech datasets based on Platforms, Language, #Posts, span-level annotations (Span), inclusion of Target (Tar.), Argument (Arg.), Group, and Lexicon information (Lex.).

While previous works offer quality corpora, span-level research remains unexplored. STATE TOXiCN is the first such dataset, annotating Target-Argument-Hateful- Group quadruples. Unlike existing lexicons, we offer interpretative annotations and targeted group labels, creating a comprehensive Chinese hateful slang lexicon.

3 Dataset Construction

3.1 Overview

In this section, we introduce the construction process of the STATE TOXiCN dataset and the annotated lexicon of Chinese hateful slang. First, we describe the data sources and filtering procedures. Next, we detail the annotation process and the measures implemented to ensure high annotation quality. We then examine the Inter-Annotator Agreement (IAA) at different levels of granularity. Finally, we present relevant statistics for the STATE TOXiCN dataset.

3.2 Data Source and Filtering

Our dataset construction is based on the post-level Chinese hate speech dataset TOXiCN (Lu et al., 2023). We develop a high-quality span-level Chinese hate speech dataset through data filtering and annotation processes. During the data filtering phase, we extract potential samples from the original dataset and systematically clean low-quality, incomplete, or irrelevant content. For instance, we remove meaningless text such as advertisements, random character combinations, and overly short (less than 5 characters) or overly long (more than 500 characters) text fragments.

Building on this, we conduct a comprehensive review of the hate speech annotations, particularly regarding the clarity of the target and argument components of hate speech. For example, in cases describing strongly discriminatory or biased content, we further examine whether the target was specific, removing ambiguous or undefined samples. In particular, for texts lacking a clear Target-Argument structure, we opt to exclude them to ensure the dataset could accommodate the requirements of span-level analysis.

3.3 Data Annotation

3.3.1 Annotation Guidelines

During the annotation process, we establish guidelines and implement multi-stage quality control to ensure the consistency of the annotations, aiming to label Target-Argument-Hateful-Group quadruples in the posts. First, we develop detailed annotation guidelines, including standards for extracting targets and arguments (Target-Argument Pair), criteria for determining hatefulness (Hateful), and methods for classifying groups (Group):

Target-Argument Pair A span consists of both the target and its corresponding argument extracted from the post. A single post may contain more than one Target-Argument Pair.

Hateful If the Target-Argument Pair explicitly or implicitly conveys harm towards the target or other groups, it is labeled as "hateful"; otherwise, it is labeled as "non-hate."

Group Building on the Target-Argument Pair, this category identifies specific groups targeted by hateful expressions, with a single pair probably involving multiple groups.
<table>
  <tr>
    <th>Target Span</th>
    <th>Argument Span</th>
    <th>If Hateful</th>
    <th>Targeted Group</th>
  </tr>
  <tr>
    <td>0.65</td>
    <td>0.61</td>
    <td>0.68</td>
    <td>0.75</td>
  </tr>
</table>

Table 3: Fleiss’ Kappa for different granularities.

For example, in Exp. 2 from Table 1, Target-Argument Pair is “男同 (Gay people)” and “艾滋病高发群体(a high-risk group for HIV)”. This Target-Argument Pair constitutes hate against the LGBTQ community and people living with AIDS, so the Hateful label is “hate” and the Group label is “LGBTQ, others”. The quadruple is [男同|艾滋病高发群体| hate | LGBTQ, others ].

Regarding the annotation of the Chinese hateful slang lexicon, we also establish annotation guidelines for identifying, categorizing, and labeling the terms, with a focus on their frequent groups (Group) and contextual explanations (Explanation):

Group Each hateful slang term is categorized by the group it targets, such as sexism, racism, regional bias, anti-LGBTQ, or others. Some terms may target multiple groups.

Explanation An explanation of hateful slang is provided, including its literal meaning, extended meanings, the reasons for hatred towards targeted groups, and common usage patterns.

3.3.2 Mitigating Bias.
To mitigate bias, we assemble annotators with diverse backgrounds, including differences in gender, age, ethnicity, region, and educational level. The detailed information of annotators can be found in Appendix G. All annotators possess linguistic expertise and have undergone systematic training. During the annotation process, we primarily employ regular cross-validation and expert arbitration to ensure the objectivity and consistency of annotation results. Additionally, we maintain an online document to record and update Chinese hateful slang identified in the posts. This annotated lexicon serves not only as a research resource but also helps to align annotators’ standards.

3.3.3 Annotation Procedure
Cross-validation and Expert Arbitration. Each text is independently annotated by at least two annotators, who followed a unified annotation guideline to extract Target-Argument pairs, determine the hatefulness, and classify the groups. After the initial annotation phase, 20% of the samples are regularly selected for cross-validation, during which other annotators re-annotate these samples to ensure a consistent understanding of the rules and standards across annotators. This approach allows us to identify and resolve potential biases or discrepancies in the annotations in a timely manner.

For disputed samples with significant annotation differences, an arbitration team of domain experts reviews the samples, considering the textual context and annotation guidelines to determine the most appropriate annotation. We explore inter-annotator agreement (IAA) on STATE TOXICN, with Fleiss’ kappa scores (Fleiss, 1971) for each hierarchy detailed in Table 3. The detailed analysis can be found in Appendix D.

<table>
  <tr>
    <th>Category</th>
    <th>Subcategory</th>
    <th>Count</th>
    <th>Percentage (%)</th>
  </tr>
  <tr>
    <td rowspan="4">Groups</td>
    <td>Gender</td>
    <td>1663</td>
    <td>17.44</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>1232</td>
    <td>12.92</td>
  </tr>
  <tr>
    <td>Region</td>
    <td>1323</td>
    <td>13.88</td>
  </tr>
  <tr>
    <td>LGBTQ</td>
    <td>628</td>
    <td>6.59</td>
  </tr>
  <tr>
    <td>Others</td>
    <td>351</td>
    <td>3.68</td>
  </tr>
  <tr>
    <td>Multi-group</td>
    <td>866</td>
    <td>9.08</td>
  </tr>
  <tr>
    <td>Hateful</td>
    <td>Hate</td>
    <td>6063</td>
    <td>63.60</td>
  </tr>
  <tr>
    <td></td>
    <td>Non-Hate</td>
    <td>3470</td>
    <td>36.40</td>
  </tr>
  <tr>
    <td colspan="2">Total</td>
    <td>9533</td>
    <td>100.00</td>
  </tr>
</table>

Table 4: Statistics of annotated posts from the STATE TOXICN dataset, including Target, Argument, Group, and Hatefulness classifications.

Lexicon Annotation. To ensure annotation quality, we maintain a shared online document for recording and updating detailed information on Chinese hateful slang. This document includes explanations of each slang term and the specific groups they typically reference. Team members could add newly discovered slang terms, which are then evaluated and annotated by the expert team. This dynamic maintenance ensures a consistent understanding of emerging language and slang among all team members.

Additionally, the shared online document serves as a knowledge-sharing platform, providing the annotation team with consistent references and concrete examples for handling complex or ambiguous posts. This mechanism improves annotation efficiency and enhances the consistency of the annotations. This is the first Chinese hateful slang lexicon with interpretable annotations. This lexicon provides valuable research resources for span-level Chinese hate speech semantic understanding.
3.4 Data Description
STATE ToxICN dataset contains a total of 8,029 annotated posts, among which 4,942 posts include hateful content, accounting for 61.55%. A total of 9,533 quadruples are annotated, with 6,034 of them containing hateful information, making up 63.60%. We present the statistical details of STATE ToxICN in Table 4. Gender, Region, and Race are the three most common group types in the dataset. Additionally, "multi-group" refers to Target-Argument pairs that involve hatred directed at multiple target groups, with a total of 854 such instances, accounting for 8.96%. In addition, the annotated lexicon includes 830 Chinese hateful slang terms collected from real online forums. More annotated examples can be found in Appendix A.

4 Experiment

4.1 Baselines
To evaluate the performance of LLMs with varying parameter sizes and capabilities in detecting span-level Chinese hate speech, we choose twelve well-known models across three categories:
Open-source Models: mT5-base (Xue, 2020), Mistral-7B (Jiang et al., 2023), LLaMA3-8B (AI@Meta, 2024), Qwen2.5-7B (Team, 2024);
Safety-domain Models: ShieldLM-14B-Qwen (Zhang et al., 2024), ShieldGemma-9B (Zeng et al., 2024), and Closed-source LLMs: LLaMA3-70B (AI@Meta, 2024), Qwen2.5-72B (Team, 2024), Gemini-1.5-Pro (Team et al., 2023), Claude-3.5-Sonnet (Anthropic, 2024), GPT-4o (OpenAI, 2024), DeepSeek-v3 (Liu et al., 2024). Detailed information of fine-tuning is provided in Appendix F.

4.2 Evaluation metrics.
Due to the ambiguity of Chinese span boundaries, a single evaluation metric may not accurately assess the performance of models in span-level Chinese hate speech detection. To obtain more accurate evaluation results, we therefore utilize both hard and soft matching metrics. We employ the Macro-F1 scores as the main evaluation metrics.
Hard-matching: A predicted quadruplet, particularly its target and argument components, is deemed correct only if it perfectly matches its corresponding gold label.
Soft-matching: We adopt the algorithm proposed by Han et al. (2023), where a prediction is considered correct if the Target and Argument scores achieve a threshold of 0.5.

<table>
  <tr>
    <th>Category</th>
    <th>#Posts</th>
    <th>Quad.</th>
    <th>Hateful</th>
    <th>Non-hate</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>6424</td>
    <td>7631</td>
    <td>4842</td>
    <td>2789</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>1605</td>
    <td>1902</td>
    <td>1221</td>
    <td>681</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>8029</th>
    <th>9533</th>
    <th>6063</th>
    <th>3470</th>
  </tr>
</table>

Table 5: Statistics of train and test datasets, including #Posts, total quads (Quad.), hateful quads (Hateful), and non-hateful quads (Non-hate).

4.3 Experiment Settings
We conduct fine-tuning on open-source and safety-domain models, with training and testing set sizes detailed in Table 5. The fine-tuning process was performed using only a basic prompt, which included task definitions, output formats, and specific prediction requirements for all elements. We evaluate the performance of closed-source LLMs by calling their APIs. In addition to the aforementioned basic prompt, we also provided a hate speech example and a non-hate speech example. Further details regarding these requirements are provided in Appendix B.

5 Results and Analysis

5.1 RQ1: Can LLMs identify the spans of targets and arguments in Chinese text?

Finetuned Models In the tasks of identifying targets, arguments, and target-argument pairs (T-A Pair), fine-tuned models significantly outperform direct usage of LLM APIs. LLaMA3-8B, Qwen2.5-7B, and the Shield series models all achieve hard match scores of over 63%, strongly demonstrating the superior ability of fine-tuned models in identifying text span boundaries. Soft match metrics further confirm the advantage of fine-tuned models in target and argument identification, with F1 score approaching 70% or higher. These results indicate that fine-tuned models on task-specific data show a substantial advantage in span identification. LLaMA3-8B and Qwen2.5-7B perform the best. ShieldGemma-9B achieves the highest scores in soft-match metrics for two tasks.

LLM APIs Compared to fine-tuned models, LLMs access through APIs performed significantly worse in all tasks, regardless of whether hard or soft match metrics were used. Even with few-shot prompting strategies, their performance still lag far behind that of fine-tuned models. This suggests that LLMs, without being fine-tuned on Chinese hate speech data, are not adept at identifying text
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Target</th>
    <th colspan="2">Argument</th>
    <th colspan="2">T-A Pair</th>
    <th colspan="2">T-A-H Tri.</th>
    <th colspan="2">Quad.</th>
  </tr>
  <tr>
    <th>Hard</th>
    <th>Soft</th>
    <th>Hard</th>
    <th>Soft</th>
    <th>Hard</th>
    <th>Soft</th>
    <th>Hard</th>
    <th>Soft</th>
    <th>Hard</th>
    <th>Soft</th>
  </tr>
  <tr>
    <td colspan="11"><i>Finetuned Models (with Basic Prompt)</i></td>
  </tr>
  <tr>
    <td>mT5-base</td>
    <td>59.15</td>
    <td>70.55</td>
    <td>28.63</td>
    <td>67.03</td>
    <td>23.33</td>
    <td>55.90</td>
    <td>17.76</td>
    <td>43.34</td>
    <td>16.60</td>
    <td>38.61</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>62.97</td>
    <td>73.69</td>
    <td>35.58</td>
    <td>70.90</td>
    <td>30.55</td>
    <td>60.49</td>
    <td>26.15</td>
    <td>51.01</td>
    <td>23.72</td>
    <td>45.62</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td><b>64.07</b></td>
    <td>73.74</td>
    <td><b>36.72</b></td>
    <td>70.82</td>
    <td><b>31.64</b></td>
    <td>60.88</td>
    <td><b>27.04</b></td>
    <td>51.62</td>
    <td><b>24.27</b></td>
    <td>46.08</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>63.96</td>
    <td><b>74.64</b></td>
    <td>35.42</td>
    <td>70.36</td>
    <td>30.63</td>
    <td>60.52</td>
    <td>26.51</td>
    <td><b>52.86</b></td>
    <td>23.70</td>
    <td>47.03</td>
  </tr>
  <tr>
    <td>ShieldLM-i4B-Qwen</td>
    <td>63.83</td>
    <td>73.45</td>
    <td>34.80</td>
    <td>70.23</td>
    <td>30.20</td>
    <td>59.81</td>
    <td>26.18</td>
    <td>51.24</td>
    <td>23.59</td>
    <td>45.58</td>
  </tr>
  <tr>
    <td>ShieldGemma-9B</td>
    <td>63.40</td>
    <td>74.31</td>
    <td>34.40</td>
    <td><b>71.11</b></td>
    <td>29.99</td>
    <td><b>61.51</b></td>
    <td>25.64</td>
    <td><b>52.70</b></td>
    <td>23.49</td>
    <td><b>47.14</b></td>
  </tr>
  <tr>
    <td colspan="11"><i>LLM APIs (with Basic Prompt and 2 Examples)</i></td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>30.54</td>
    <td>41.03</td>
    <td>14.39</td>
    <td>47.96</td>
    <td>8.16</td>
    <td>27.34</td>
    <td>6.03</td>
    <td>20.70</td>
    <td>3.69</td>
    <td>11.93</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>40.94</td>
    <td>50.44</td>
    <td>21.10</td>
    <td>56.36</td>
    <td>15.66</td>
    <td>39.49</td>
    <td>12.48</td>
    <td>30.92</td>
    <td>8.74</td>
    <td>20.29</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>29.80</td>
    <td>37.29</td>
    <td>18.43</td>
    <td>54.96</td>
    <td>9.37</td>
    <td>26.22</td>
    <td>7.71</td>
    <td>21.88</td>
    <td>5.45</td>
    <td>14.81</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>37.61</td>
    <td>50.72</td>
    <td>15.45</td>
    <td>57.24</td>
    <td>9.72</td>
    <td>36.16</td>
    <td>7.94</td>
    <td>29.82</td>
    <td>6.29</td>
    <td>22.45</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>46.85</td>
    <td>58.19</td>
    <td>22.64</td>
    <td>62.41</td>
    <td>17.21</td>
    <td>46.41</td>
    <td>13.21</td>
    <td>35.68</td>
    <td>9.00</td>
    <td>23.34</td>
  </tr>
  <tr>
    <td>DeepSeek-v3</td>
    <td>48.16</td>
    <td>59.25</td>
    <td>22.79</td>
    <td>59.38</td>
    <td>18.68</td>
    <td>46.40</td>
    <td>14.95</td>
    <td>37.19</td>
    <td>11.48</td>
    <td>27.38</td>
  </tr>
</table>

Table 6: Performance comparison of various models across different levels of annotated tasks, including Target, Argument, Target-Argument Pair (<i>T-A Pair</i>), Target-Argument-Hateful Triple (<i>T-A-H Tri.</i>), and Target-Argument-Hateful-Group Quadruple (<i>Quad.</i>) under both Hard and Soft evaluation metrics.

span boundaries, which is consistent with the fact that their training objectives are not directly related to this task. Specifically, the hard match scores for LLMs on "Target" and "Argument" were only between 40-50%, while soft match scores were around 60%. GPT-4o and DeepSeek-v3 exhibit the best performance.

Comparison and Summary The fine-tuning technology can improve models’ abilities in identifying text span boundaries, demonstrating a clear advantage in target and argument extraction tasks. Joint extraction tasks are more challenging than single element extraction tasks, and all models perform worse in argument extraction than in target extraction. We believe this is primarily because in Chinese text, argument spans tend to be longer and structurally more complex, leading to poor hard match performance. However, the soft match scores for argument extraction were similar to those of target extraction, indicating that the models’ semantic understanding capabilities are comparable in both tasks. This suggests that despite difficulties in identifying precise boundaries, the models remain effective in semantic understanding.

5.2 <i>RQ2: Can LLMs classify Chinese target-argument pairs as hateful and identify the groups?</i>

Finetuned Models Fine-tuned models demonstrate a clear advantage in both the triplet and quadruplet tasks. In the Target-Argument-Hateful triplet task (T-A-H Tri.), the fine-tuned models achieve hard-match metrics of around 26% and soft-match metrics of around 52%. In the more complex quadruplet task, the hard-match metrics of the fine-tuned models are mostly between 25% and 27% when simultaneously identifying hateful content and target groups, while the soft-match metrics are approximately 45% - 47%. This indicates that the fine-tuned models’ performance is mainly limited by the target-argument pair identification. Overall, although the fine-tuned models perform poorly in hard-match metrics, the results in soft-match metrics reach approximately 50%. Among these, LLaMA-3-8B performs best in hard-match, while Qwen2.5-7B and ShieldGemma-9B perform better in soft-match metrics.

LLM APIs LLM APIs perform poorly on both the triplet and quadruplet tasks. In the T-A-H triplet task, LLMs achieve hard-match metrics of only 6% - 15% and soft-match metrics of 20% - 38%. In the quadruplet task, the hard-match metrics of LLMs further drop to 3% - 12%, and the soft-match metrics, though slightly better, are still only 11% - 27%. We believe that the performance limitations also primarily stem from the difficulty in identifying accurate span boundaries. These data clearly show that LLMs, without being fine-tuned on Chinese hate speech data, are significantly inadequate at performing Chinese hate speech triplet and quadru-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Target</th>
    <th colspan="2">Argument</th>
    <th colspan="2">T-A Pair</th>
    <th colspan="2">T-A-H Tri.</th>
    <th colspan="2">Quad.</th>
  </tr>
  <tr>
    <th>Hard</th><th>Soft</th>
    <th>Hard</th><th>Soft</th>
    <th>Hard</th><th>Soft</th>
    <th>Hard</th><th>Soft</th>
    <th>Hard</th><th>Soft</th>
  </tr>
  <tr>
    <td colspan="11"><i>Finetuned Models (with Basic Prompt)</i></td>
  </tr>
  <tr>
    <td>mT5-base</td><td>56.83<sub>3.2</sub></td><td>68.32<sub>2.2</sub></td><td>27.17<sub>1.46</sub></td><td>64.17<sub>1.46</sub></td><td>21.33<sub>0.00</sub></td><td>51.17<sub>1.73</sub></td><td>18.17<sub>1.46</sub></td><td>44.67<sub>1.33</sub></td><td>16.33<sub>0.27</sub></td><td>36.17<sub>2.14</sub></td>
  </tr>
  <tr>
    <td>Mistral-7B</td><td>61.03<sub>1.94</sub></td><td>72.62<sub>1.07</sub></td><td>36.71<sub>1.13</sub></td><td>70.05<sub>0.85</sub></td><td>30.27<sub>0.28</sub></td><td>58.62<sub>0.87</sub></td><td>26.25<sub>0.10</sub></td><td>51.05<sub>0.94</sub></td><td>22.22<sub>0.50</sub></td><td>43.00<sub>0.62</sub></td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td><td>61.26<sub>1.81</sub></td><td>72.12<sub>1.62</sub></td><td>35.17<sub>1.55</sub></td><td>70.83<sub>0.01</sub></td><td>28.53<sub>1.11</sub></td><td>59.00<sub>0.88</sub></td><td>24.47<sub>2.37</sub></td><td>51.38<sub>0.24</sub></td><td>19.77<sub>4.50</sub></td><td>42.46<sub>0.62</sub></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>61.79<sub>1.21</sub></td><td>73.33<sub>1.31</sub></td><td>36.26<sub>1.07</sub></td><td>69.58<sub>0.41</sub></td><td>29.92<sub>1.21</sub></td><td>57.72<sub>0.87</sub></td><td>27.64<sub>1.15</sub></td><td>53.66<sub>0.56</sub></td><td>22.76<sub>0.60</sub></td><td>43.60<sub>0.49</sub></td>
  </tr>
  <tr>
    <td>ShieldLM-F4B-Qwen</td><td>64.08<sub>0.25</sub></td><td>74.48<sub>1.03</sub></td><td>34.19<sub>1.77</sub></td><td>69.86<sub>0.37</sub></td><td>28.90<sub>3.30</sub></td><td>58.80<sub>5.51</sub></td><td>25.60<sub>1.58</sub></td><td>51.69<sub>0.45</sub></td><td>21.02<sub>0.29</sub></td><td>43.66<sub>1.98</sub></td>
  </tr>
  <tr>
    <td>ShieldGemma-9B</td><td>62.50<sub>0.30</sub></td><td>74.35<sub>0.04</sub></td><td>35.71<sub>1.31</sub></td><td>71.10<sub>1.01</sub></td><td>29.87<sub>0.12</sub></td><td>60.71<sub>1.80</sub></td><td>26.95<sub>1.31</sub></td><td>55.03<sub>2.33</sub></td><td>23.21<sub>0.28</sub></td><td>46.10<sub>1.04</sub></td>
  </tr>
  <tr>
    <td colspan="11"><i>LLM APIs (with Basic Prompt and 2 Examples)</i></td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td><td>30.87<sub>0.33</sub></td><td>41.45<sub>0.42</sub></td><td>14.80<sub>0.41</sub></td><td>46.68<sub>0.28</sub></td><td>8.29<sub>0.13</sub></td><td>25.38<sub>0.96</sub></td><td>7.40<sub>0.37</sub></td><td>22.58<sub>1.88</sub></td><td>4.72<sub>1.03</sub></td><td>13.14<sub>1.21</sub></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td><td>45.58<sub>1.64</sub></td><td>54.67<sub>4.23</sub></td><td>22.15<sub>0.41</sub></td><td>57.36<sub>0.00</sub></td><td>16.77<sub>1.11</sub></td><td>41.61<sub>1.12</sub></td><td>12.42<sub>0.06</sub></td><td>30.35<sub>0.57</sub></td><td>8.83<sub>0.99</sub></td><td>19.59<sub>0.70</sub></td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td><td>31.52<sub>1.72</sub></td><td>39.07<sub>1.78</sub></td><td>18.94<sub>0.54</sub></td><td>54.57<sub>0.39</sub></td><td>9.01<sub>0.46</sub></td><td>27.95<sub>0.73</sub></td><td>8.61<sub>0.90</sub></td><td>25.83<sub>0.95</sub></td><td>6.23<sub>0.75</sub></td><td>17.35<sub>0.54</sub></td>
  </tr>
  <tr>
    <td>Claude 3.5-Sommet</td><td>41.45<sub>1.84</sub></td><td>54.06<sub>3.34</sub></td><td>15.80<sub>0.45</sub></td><td>55.80<sub>1.11</sub></td><td>10.43<sub>0.71</sub></td><td>36.96<sub>0.80</sub></td><td>9.28<sub>0.34</sub></td><td>33.04<sub>2.22</sub></td><td>7.10<sub>1.81</sub></td><td>25.22<sub>2.77</sub></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>49.28<sub>1.43</sub></td><td>61.30<sub>1.11</sub></td><td>21.71<sub>0.93</sub></td><td>59.66<sub>0.75</sub></td><td>16.52<sub>0.69</sub></td><td>46.55<sub>1.14</sub></td><td>12.01<sub>2.00</sub></td><td>33.72<sub>1.96</sub></td><td>8.46<sub>5.54</sub></td><td>22.80<sub>5.04</sub></td>
  </tr>
  <tr>
    <td>DeepSeek-v3</td><td>52.69<sub>1.53</sub></td><td>64.25<sub>4.00</sub></td><td>23.79<sub>1.00</sub></td><td>62.10<sub>2.72</sub></td><td>19.22<sub>0.54</sub></td><td>50.40<sub>0.00</sub></td><td>16.13<sub>1.18</sub></td><td>42.07<sub>4.88</sub></td><td>11.69<sub>9.21</sub></td><td>30.11<sub>7.73</sub></td>
  </tr>
</table>

Table 7: Performance comparison of various models on a hateful-slang-containing subset of the test set. Green indicates a decrease in F1 score on the subset compared to the full dataset, while red indicates an increase.

plet predictions, even with few-shot prompting failing to bridge the performance gap. DeepSeek-v3 performs significantly better than other APIs, particularly in the quadruplet prediction task.

Comparison and Summary The fine-tuned model significantly outperforms the LLM APIs, with the performance gap primarily stemming from T-A pair extraction. The fine-tuned model outperforms the LLM APIs in both hate speech the triplet and quadruplet prediction. Specifically, the fine-tuned models show better performance in tasks requiring the simultaneous identification of hate speech and its target group. Although there is still room for improvement in terms of precise matching, the application of fine-tuning technology undoubtedly provides a feasible path for handling such complex tasks. At the same time, the performance of the LLM APIs in these tasks is disappointing, indicating that they need targeted fine-tuning to be applied to these types of scenarios.

5.3 RQ3: Can LLMs understand Chinese hateful slang?

5.3.1 Impact of Hateful Slang on Chinese Hate Speech Detection

To evaluate the impact of hate slang on span-level Chinese hate speech detection models, we remove the posts without hateful slang from the test set and obtain a subset of 502 posts containing hateful slang for focused testing. By comparing the model performance on this subset of data with its performance on the full test set, we can gain a clearer understanding of how hate slurs affect model performance. The experimental results are presented in Table 7. The superscript numbers in the table indicate the difference in F1 score between the model’s performance on the selected subset and its performance on the full test set. Green indicates a decrease in F1 score on the subset compared to the full dataset, while red indicates an increase.

Finetuned Models Across most tasks, the hard and soft matching metrics for the models show a decreasing trend. However, the metric for the hate speech triplet prediction task (T-A-H Tri.) is an exception, exhibiting an increase. We attribute this primarily to two factors: Firstly, posts containing hateful slang generally demonstrate a more explicit hateful intent, while the posts we remove include many types that are difficult to identify, such as implicit hate expressions. This makes it easier for the T-A-H Tri. task to capture the relationships within hate speech triplets, thereby leading to the observed increase in the metric. Secondly, the experimental results also confirm that the main challenges posed by hateful slang lie in extracting the target and argument, as well as accurately classifying the targeted group. When the training data is not sufficiently comprehensive, fine-tuned models struggle to effectively handle the complexities introduced by hateful slang.

LLM APIs LLM APIs demonstrate superior performance on the subset of the test set containing hateful slang. With the exception of a slight decrease in the soft matching metric for argument identification, nearly all other metrics show an improvement. We believe that this is primarily due to two factors: First, hateful slang often appears as the target, and the explicit target terms effectively compensate for the limitations of LLM APIs in span boundary recognition. Second, the identifica-
<table>
  <tr>
    <th>Example 1</th>
    <th>Example 2</th>
  </tr>
  <tr>
    <td><b>Term:</b> “默” Text: “默我是真的很讨厌。”<br><br><i>ShieldGemma-9B:</i> I speculate that ‘默’ (mò) is used in this context to express hatred towards a group associated with silence or stillness. Without more context regarding the speaker’s background, it’s difficult to determine what they mean by ‘默’.<br><br><i>DeepSeek-v3:</i> The term ‘默’ (mò) could be a name, nickname, or designation. More context is needed to determine if it has hateful connotations. If ‘默’ refers to a specific group and expresses animosity, it could have hateful undertones.<br><br><i>Human Annotator:</i> The word ‘默’ (mò), meaning 'silence', when spelled out as '黑犬' (hēiquǎn), meaning 'black dog,' is a merge-word used to insult Black people.</td>
    <td><b>Term:</b> “冉闵” Text: “神州不止一个冉闵。”<br><br><i>ShieldGemma-9B:</i> ‘冉闵’ (Ran Min) could be coded hostility toward: minority ethnic or racial groups (as historically used to express prejudice); political opponents (for demonization); or religious groups (as a veiled expression of intolerance).<br><br><i>DeepSeek-v3:</i> Ran Min of the Sixteen Kingdoms is known for the 'Kill the Hu' order and extreme actions against northern minorities, resulting in numerous deaths. He's seen by some as a national hero, but by others as a tyrant or nationalist.<br><br><i>Human Annotator:</i> Ran Min was the founder of the Ran Wei regime. He led a revolt against the 'Hu' people and issued the 'Kill the Hu' order. He is often used as a symbol of nationalism, and has been used to justify racial discrimination.</td>
  </tr>
</table>

Figure 1: Examples of Chinese hateful slang understanding analysis with LLM. The hateful slang terms and texts appear in black. ShieldGemma-v3 explanations are in green, DeepSeek-v3 explanations are in red, and human annotator explanations are in blue.

tion of hateful slang often requires rich background knowledge; for example, Example 2 in the figure requires knowledge of Chinese history and related common usages, which is precisely where LLM APIs excel.

Comparison and Summary Fine-tuned models and LLM APIs exhibit distinct characteristics when processing hateful slang. Fine-tuned models, when lacking sufficient training and understanding of specific domain knowledge, struggle to effectively address the challenges posed by hateful slang, particularly in areas such as fine-grained entity recognition and complex language pattern understanding. In contrast, LLM APIs leverage their robust background knowledge and contextual understanding to better comprehend and handle these complex situations, thus achieving superior performance. However, the performance gap between LLM APIs and fine-tuned models on this task remains significant. Infusing the background knowledge of large models into fine-tuned models may offer a promising avenue for improvement.

5.3.2 Understanding Capabilities of LLMs on Chinese Hateful Slang

To investigate the understanding capabilities of large language models regarding Chinese hateful slang, we conducted experiments using two relatively high-performing models, ShieldGemma-9B and DeepSeek-v3. In these experiments, we provided the models with hateful terms and their surrounding sentence context, asking them to generate explanations of the hateful slang and identify the potentially affected groups. The experimental results are detailed in Figure 1, with specific experimental details provided in Appendix C.

For a more nuanced analysis of the models’ understanding abilities, we selected two terms with distinct Chinese characteristics as case studies. In Case 1, "默" (mò) is a typical "merging word," whose literal meaning is 'silence,' but it is composed of the characters "黑犬" (hēiquǎn), meaning "black dog." This type of character combination is a unique linguistic phenomenon in Chinese. The experimental results showed that neither model could accurately understand the hateful information contained within it. In Case 2, "冉闵" (Ràn Mín) is a hateful slang term rooted in Chinese history and culture, referring to an ancient emperor who massacred ethnic minorities and is often used for racial discrimination. In comparison, while ShieldGemma-9B could understand the background information, it failed to generate an accurate explanation of the hateful information and clearly identify the potentially affected groups. DeepSeek-v3’s generated information, however, closely matched the results given by human annotators. This demonstrates that even relatively high-performing models face significant challenges in understanding Chinese hateful slang that possesses cultural specificity and subtle connotations.

6 Conclusions

With the advancement of hate speech detection, recent research has shifted from post-level detection to more fine-grained span-level detection. In this work, we focus on building resources for fine-grained Chinese hate speech detection. Firstly, we present the first span-level target-aware toxicity extraction dataset (STATE TOXIcn). Based on
this dataset, we evaluate the performance of LLMs in span-level Chinese hate speech detection. Secondly, we provide the first Chinese hateful slang lexicon with interpretable annotations. All the hateful slang terms are sourced from real-world online platforms. Using this lexicon, we evaluate the impact of hateful slang on model detection capabilities and assess LLMs’ understanding of hateful slang. Experimental results show that current LLMs still face challenges in effectively addressing both span-level hate speech detection and the understanding of hate semantics. We hope that our resources and benchmarks will be valuable for researchers in this field.

Limitation

Despite implementing rigorous quality control measures in constructing the span-level Chinese hate speech detection dataset and the annotated lexicon, we acknowledge several limitations. First, although we have taken steps to minimize labeling bias, subjective differences in annotators’ understanding of toxic language may still result in mislabeled data. Additionally, while we strived to ensure accuracy and consistency in annotating target-argument pairs, the inherent characteristics of the Chinese language, such as flexible grammar and ambiguous boundaries, pose challenges for completely precise annotations.

Our Annotated Lexicon covers a wide range of commonly used Chinese hateful slang; however, due to the dynamic nature and rapid evolution of the Chinese internet language, certain emerging or less widely recognized hateful slang (e.g., more complex word transformations or domain-specific slang) might not be fully captured. Finally, this study focuses primarily on textual features and does not consider non-textual elements, such as images, videos, or metadata about the authors, which may limit the model’s ability to comprehensively detect multimodal hate speech.

In the future, we aim to expand the scope of our dataset to include more diverse contexts and hateful slang, while exploring multimodal and cross-domain hate speech detection methods to improve overall performance and applicability.

Ethics Statement

We adhere strictly to the data usage agreements of all public online social platforms and conduct thorough reviews to ensure that no user privacy information is included in our dataset. The opinions and findings reflected in the samples of our dataset do not represent the views of the authors, either explicitly or implicitly. We aim to ensure that the benefits of our proposed resources outweigh any potential risks. All resources are provided exclusively for scientific research purposes.

To minimize the psychological impact of evaluating harmful content, we have implemented a multi-faceted approach. This includes obtaining informed consent after thoroughly explaining the nature of the content they may encounter, carefully managing exposure by limiting weekly evaluation volumes, and empowering annotators to immediately cease work should they experience any discomfort. Furthermore, we proactively monitor their mental health through regular check-ins, ensuring a supportive and responsible working environment.

Acknowledgment

This research is supported by the Natural Science Foundation of China (No. 62376051, 61702080, 62366040), the Key R&D Projects in Liaoning Province award numbers (2023JH26/10200015), the Fundamental Research Funds for the Central Universities (DUT24LAB123).

References

Hyeseon Ahn, Youngwook Kim, Jungin Kim, and Yo-Sub Han. 2024. Sharedcon: Implicit hate speech detection using shared semantics. In Findings of the Association for Computational Linguistics ACL 2024, pages 10444–10455.

AI@Meta. 2024. Llama 3 model card.

Raza Ali, Umar Farooq, Umair Arshad, Waseem Shahzad, and Mirza Omer Beg. 2022. Hate speech detection on twitter using transfer learning. Computer Speech & Language, 74:101365.

Badr AlKhamissi, Faisal Ladhak, Srinivasan Iyer, Ves Stoyanov, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, and Mona Diab. 2022a. Token: Task decomposition and knowledge infusion for few-shot hate speech detection. arXiv preprint arXiv:2205.12495.

Badr AlKhamissi, Faisal Ladhak, Srinivasan Iyer, Veselin Stoyanov, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, and Mona Diab. 2022b. ToKen: Task decomposition and knowledge infusion for few-shot hate speech detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2109–2120, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/clause-3-5-sonnet.

Michal Bilewicz and Wiktor Soral. 2020. Hate speech epidemic: the dynamic effects of derogatory language on intergroup relations and political radicalization. Political Psychology, 41:3–33.

Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2020. Hatebert: Retraining bert for abusive language detection in english. arXiv preprint arXiv:2010.12472.

I Chung and Chuan-Jie Lin. 2021. Tocab: A dataset for chinese abusive language processing. In 2021 IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI), pages 445–452. IEEE.

Gloria Cowan and Cyndi Hodge. 1996. Judgments of hate speech: The effects of target group, publicness; and behavioral responses of the target. Journal of Applied Social Psychology, 26(4):355–374.

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512–515.

Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. 2022. Cold: A benchmark for chinese offensive language detection. arXiv preprint arXiv:2201.06025.

David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. Ethnologue: Languages of the World, twenty-seventh edition. SIL International, Dallas, Texas. Online version available at http://www.ethnologue.com.

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.

Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of the international AAAI conference on web and social media, volume 12.

Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450.

Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify.

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509.

Sanne Hoeken, Sophie Splithoff, Silke Schwandt, Sina Zarriß, and Özge Alacam. 2023. Towards detecting lexical change of hate speech in historical data. In Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change, pages 100–111, Singapore. Association for Computational Linguistics.

Robin Jeshion. 2013. Slurs and stereotypes. Analytic Philosophy, 54(3).

Aiqi Jiang, Xiaohan Yang, Yang Liu, and Arkaitz Zubiaga. 2022. Swsr: A chinese dataset and lexicon for online sexism detection. Online Social Networks and Media, 27:100182.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Junyu Lu, Bo Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin. 2023. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16235–16250, Toronto, Canada. Association for Computational Linguistics.

Victor H Mair. 1991. What is a chinese" dialect/topiclect"?: Reflections on some key sino-english linguistic terms.

Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hateexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867–14875.

OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/.

John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. SemEval-2021 task 5: Toxic spans detection. In Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021), pages 59–69.

Leandro Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Proceedings of the International AAAI Conference on Web and Social Media, volume 10, pages 687–690.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jahui Yu, Radu Soriciut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie
Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.

Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2024. ToxiCloakCN: Evaluating robustness of offensive language detection in Chinese with cloaking perturbations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6012–6025, Miami, Florida, USA. Association for Computational Linguistics.

L Xue. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.

Marcos Zampieri, Skye Morgan, Kai North, Tharindu Ranasinghe, Austin Simmons, Paridhi Khandelwal, Sara Rosenthal, and Preslav Nakov. 2023. Target-based offensive language identification.

Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharupu, Olivia Sturman, and Oscar Wahltimez. 2024. Shieldgemma: Generative ai content moderation based on gemma.

Zhixin Zhang, Yida Lu, Jingyuan Ma, Di Zhang, Rui Li, Pei Ke, Hao Sun, Lei Sha, Zhifang Sui, Hongning Wang, et al. 2024. Shieldlm: Empowering llms as aligned, customizable and explainable safety detectors. arXiv preprint arXiv:2402.16444.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. 2022. Towards identifying social bias in dialog systems: Framework, dataset, and benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3576–3591.

Xianbing Zhou, Yang Yong, Xiaochao Fan, Ge Ren, Yunfeng Song, Yufeng Diao, Liang Yang, and Hongfei Lin. 2021. Hate speech detection based on sentiment knowledge sharing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7158–7166.

A  Details of the Annotated Lexicon

We use JSON files to store the annotated lexicon, which is a mainstream coding specification to facilitate machine-readability. The data structure is defined as Sample = (Term, [Groups], Definition), where Groups indicates the groups that are commonly targeted by the hateful slang. If a term does not have a specific targeted group, then Groups is set to “others”.

• Term: 默(Mò)
Meaning: Silence
Groups: Racism
Definition: “默”是“黑犬”的拼字，指黑色的狗，对黑人群体的侮辱性称呼。
Definition in English: The term “默” (Mò, meaning silence) is a spelling variant of “黑犬” (black dog), an insulting term for the Black community.

• Term: 金针菇
Meaning: Enoki mushroom (metaphor for male genitalia)
Groups: Sexism
Definition: 用于对男性生殖器官的比喻，常含贬义。
Definition in English: “Enoki mushroom” is a derogatory metaphor for male genitalia.

• Term: 小仙女
Meaning: Little fairy (refers to selfish, hypocritical, or annoying women)
Groups: Sexism
Definition: 在某些情况下被用来指代那些自私自利、双标、让人厌恶的女性。
Definition in English: “Little fairy” is used in some contexts to refer to women perceived as selfish, hypocritical, or annoying.

• Term: 4000+
Meaning: Homophone for “死全家” (may your whole family die)
Groups: Others
Definition: “4000+”是“死全家”中文谐音，常用于恶意辱骂。
Definition in English: “4000+” is a homophone for “死全家” (may your whole family die), often used in malicious insults.

B  Details of the Prompt

The basic prompt and the version with 2 examples are shown in Figure 2.
<table>
  <tr>
    <th>Basic Prompt</th>
  </tr>
  <tr>
    <td><b>Chinese:</b><br>
你是一个内容审查专家，请你分析我的句子并且从中提取出一个或者多个四元组。请从下面的文本抽取一个或多个四元组，每一个四元组输出格式为评论对象|对象观点|是否仇恨|仇恨群体。评论对象可以为 'NULL', 对象观点尽量简洁, 仇恨群体只包括 (LGBTQ、Region、Sexism、Racism、others, non-hate)，同一四元组可能涉及多个仇恨群体，是否仇恨标签为 (hate, non-hate), 多个四元组之间用 [SEP] 分隔, 最后一个四元组后面加 [END]。提取出句子中包含的所有四元组:<br>
<b>English:</b><br>
You are a content review expert, please analyze my sentences and extract one or more quadruples from them. Please extract one or more quadruples from the following text, with the output format of each quadruple being ‘Target|Argument|Whether Hateful|Group’. The target can be ‘NULL’, the argument should be as concise as possible, targeted groups only include (LGBTQ, Region, Sexism, Racism, others, non-hate), the same quadruple may involve multiple groups, the ‘whether hate’ tag is (hate, non-hate), multiple quadruples are separated by [SEP], and the last quadruple is followed by [END]. Extract all quadruples contained in the sentences:</td>
  </tr>
</table>

<table>
  <tr>
    <th>Basic Prompt with 2 Examples</th>
  </tr>
  <tr>
    <td><b>Chinese:</b><br>
你是一个内容审查专家，请你分析我的句子并且从中提取出一个或者多个四元组。请从下面的文本抽取一个或多个四元组，每一个四元组输出格式为评论对象|对象观点|是否仇恨|仇恨群体。评论对象可以为 'NULL', 对象观点尽量简洁, 仇恨群体只包括 (LGBTQ、Region、Sexism、Racism、others, non-hate)，同一四元组可能涉及多个仇恨群体，是否仇恨标签为 (hate, non-hate), 多个四元组之间用 [SEP] 分隔, 最后一个四元组后面加 [END]。<br>
仇恨样本输入：谁喜欢小黑啊，黑不拉几的看一眼都嫌脏，也就某些连汽车排气口都不放过的男人会喜欢黑妞<br>
仇恨样本输出：小黑|黑不拉几的看一眼都嫌脏|Racism|hate [SEP] 某些连汽车排气口都不放过的男人|喜欢黑妞|Sexism, Racism|hate [END]<br>
非仇恨样本输入：由同性伴侣抚养长大的孩子，在学校表现比异性伴侣抚养的孩子更好，并且比异性伴侣的孩子高中毕业率高出4.8%。<br>
非仇恨样本输出：由同性伴侣抚养长大的孩子|在学校表现比异性伴侣抚养的孩子更好|non–hate|non–hate [END]<br>
提取出句子中包含的所有四元组：<br>
<b>English:</b><br>
You are a content review expert, please analyze my sentences and extract one or more quadruples from them. Please extract one or more quadruples from the following text, with the output format of each quadruple being ‘Target|Argument|Whether Hateful|Group’. The target can be ‘NULL’, the argument should be as concise as possible, targeted groups only include (LGBTQ, Region, Sexism, Racism, others, non-hate), the same quadruple may involve multiple groups, the ‘whether hate’ tag is (hate, non-hate), multiple quadruples are separated by [SEP], and the last quadruple is followed by [END].<br>
Hate sample input: Who likes black people? They are so dirty that even a glance makes you feel unclean. Only some men who don’t even mind the exhaust pipes of cars would like black girls.<br>
Hate sample output: black people | so dirty that even a glance makes you feel unclean | Sexism, Racism | hate [SEP] some men who don’t even mind the exhaust pipes of cars | like black girls | Sexism, Racism | hate [END]<br>
Non-hate sample input: Children raised by same-sex couples perform better at school than those raised by opposite-sex couples, and have a 4.8% higher high school graduation rate than children raised by opposite-sex couples.<br>
Non-hate sample output: Children raised by same-sex couples | perform better at school than those raised by opposite-sex couples | non-hate | non-hate [END]<br>
Extract all quadruples contained in the sentences:</td>
  </tr>
</table>

Figure 2: Illustration of the basic prompt and the version with 2 examples.
C Experimental Details of Hateful Slang Understanding Generation

We prompt the non-fine-tuned ShieldGemma-9B and DeepSeek-v3 to generate hateful slang understanding. The prompt used is as follows.

Prompt: [Text]中的[Term]是否包含特定群体的仇恨信息，如果是，请说明对哪些群体构成仇恨并说明原因。

Prompt in English: In [Text], does the [Term] constitute hate speech directed at specific groups? If yes, please identify the targeted groups and the reasons behind the hatred.

D Analysis of Inter-annotator Agreement

To explore inter-annotator agreement (IAA), we calculate Fleiss’ kappa scores (Fleiss, 1971) for the Target, Argument, Hateful, and Group labels separately. Due to the direct relationship between “Hateful” and “Group”, if the “Hateful” annotation is inconsistent, the “Group” label will be discarded. The experimental results are shown in Table 3. The score for argument span is the lowest, mainly due to the ambiguity of span boundaries in Chinese and the added complexity of argument spans. Despite having established relevant guidelines, it is not possible to fully standardize annotators’ labeling practices. The Target span boundaries are clearer, resulting in a higher score. The Kappa score for the Group label is 0.75, with "Region" and "Racism" being the most easily confused.

E Derivative Rules of Chinese Hateful Slang

Lu et al. (2023) propose seven types of derivative rules: Deformation (separating and combining individual characters), Homophonic (similar pronunciation), Irony (positive words used ironically to insult), Abbreviation (shortening and contracting sensitive words), Metaphor (degrading targets into something sarcastic), Code Mixing (use of non-Chinese codes to emphasize tone), and Borrowed Words (foreign phonetic words with specific toxic cultural connotations). We add two more derivative rules, Historical Allusions and Stereotypes.

Historical Allusions: China has a rich history of allusions, and events or figures from these allusions are sometimes used as insults in certain contexts (Hoeken et al., 2023). The name “冉闵” (an ancient Chinese emperor) is often used as a symbol of racial discrimination, as he once ordered the massacre of other races.

Stereotypes: Internet users often resort to stereotypes to insult their targets, degrading them into negative or prejudiced representations (Jeshion, 2013). Due to an incident in the 20th century where some people from Henan province were involved in stealing manhole covers to sell for money, the term "井盖" (manhole cover) has become a stereotype for people from Henan, often used in regional discrimination to label them as thieves or poor.

F Detailed Information of the Fine-tuning

We use LLaMA-Factory² (Zheng et al., 2024) for fine-tuning, which is a widely used framework for fine-tuning LLMs. To avoid overfitting, we monitor the trends of training and test losses and confirm through preliminary experiments that the performance stabilizes after 10 epochs. Therefore, all models are trained for 10 epochs. We adopt the LoRA method and evaluate the results.

To reduce hyperparameter sensitivity, we train models for each learning rate and select the result with the highest F1 score on the test set, then calculate the final performance by weighted averaging. The hyperparameters are presented in Table 8.

<table>
  <tr>
    <th>Hyperparameters</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e-5, 2e-5, 3e-5, 4e-5, 5e-5</td>
  </tr>
  <tr>
    <td>Cutoff length</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Compute type</td>
    <td>fp16</td>
  </tr>
  <tr>
    <td>Gradient accumulation</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Maximum gradient norm</td>
    <td>1.0</td>
  </tr>
</table>

Table 8: Annotators Demographics.

G Detailed Information of the Annotators

To mitigate bias, we assemble a diverse group of annotators, encompassing a variety of genders, ages, ethnicities, regions, and educational backgrounds. The statistical information is presented in Table 9.

<table>
  <tr>
    <th>Characteristic</th>
    <th>Demographics</th>
  </tr>
  <tr>
    <td>Gender</td>
    <td>8 male, 8 female</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>7 age < 25, 9 age ≥ 25</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>10 Asian, 6 others</td>
  </tr>
  <tr>
    <td>Region</td>
    <td>From 9 different provinces</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>5 BD, 6 MD, 5 Ph.D.</td>
  </tr>
</table>

Table 9: Annotators Demographics.

²https://github.com/hiyouga/LLaMA-Factory
H  License

We confirm that the dataset is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

Reviewer paper summaries: This paper proposes a fine-grained Span-level Target-Aware Toxicity Extraction Chinese hate speech benchmark (STATE TOXICN). They develop a benchmark that tags hate targets and arguments associated with hate speech, while detecting hate/non-hate classification and categorizing the type of hate speech. Previous studies on Chinese hate speech detection have primarily focused on post-level analysis rather than span-level identification. However, Chinese lacks delimiters (such as spaces) and allows for flexible word order, making it particularly challenging to define spans. Therefore, they use a span-level target aware method.
Additionally, the authors construct an annotated lexicon for Chinese hate speech. The benchmark is used to evaluate various language models assessing their ability to detect targets, arguments, hate classification, and categories. API-based language models exhibit lower performance compared to fine-tuned models. 
Furthermore, they evaluate Chinese slang detection performance using the annotated lexicon. API-based models (DeepSeek) outperform fine-tuned models because API-base models have rich background knowledge.
While API-based LLMs excel at understanding cultural context, they struggle with the detection of specific hate speech expressions (targets, arguments). Fine-tuned models are effective at precise span-level detection but difficult to understand contextual slang expressions. The authors emphasize that if you want better hate speech detection in Chinese, you need finer labels, better slang handling, and models that understand more than just surface-level toxicity.

This paper presents STATE ToxiCN, a new benchmark for Chinese hate speech detection with fine-grained, span-level annotations. It introduces the first Chinese hate speech dataset where each post is labeled with Target-Argument-Hateful-Group (TAHG) quadruples at the span level​. In addition, the authors compile a curated lexicon of Chinese hateful slang terms, each annotated with its meaning and targeted group, to aid in understanding how covert hate expressions function​. Using these resources, the paper provides a comprehensive evaluation of models: comparing fine-tuned models against closed-source large language model (LLM) APIs on multiple tasks, including extracting TAHG spans and interpreting hateful slang​. The study offers valuable contributions by filling the gap in Chinese span-level hate speech resources and by analyzing model performance on both explicit and slang-based toxic content.

This paper introduces the first span-level Chinese hate speech dataset, and evaluates the performance of multiple LLMs in fine-tuned and few-shot scenarios on this dataset. Their findings show the limitations of current LLM APIs without fine-tuning in the Chinese hate speech detection and span-level hate-speech identification tasks.
Summary of strengths: First-ever span-level, target-aware Chinese hate speech dataset, STATE ToxiCN with over 9.5k annotated quadruples, and the dataset reflects real-world hate speech across multiple forms—sexism, racism, regional bias, anti-LGBTQ—making it practically useful and comprehensive.

The Target-Argument-Hateful Group annotation structure is grounded in linguistic theory and allows for fine-grained toxic content analysis, not just surface-level detection.

First annotated Chinese hateful slang lexicon that includes usage context and group targets, which fills a gap in the field and enables slang-aware model evaluation.

Ensure reliable data annotation through systematic validation methods and a well-structured annotation process.

Conducts experiments with various models presenting a thorough comparison between fine-tuned models and API-based models.

Evaluates the detection performance of models and derives a reasonable analysis based on the experimental results.

- The dataset is the first span-level Chinese hate speech corpus with TAHG annotations, providing fine-grained labels for the target and argument spans, their hatefulness, and the targeted group​. This is a notable contribution, as prior Chinese hate speech datasets were only at the post level, so the work addresses an important resource gap.
- The dataset annotation was conducted with high standards and quality control. All posts were double-annotated by trained linguists following detailed guidelines, with regular cross-validation checks and expert arbitration to resolve disagreements​.
- The experiments clearly show that fine-tuned models outperform closed-source LLM APIs on span-level hate speech tasks.
- The paper offers an interesting analysis of hateful slang understanding of LLMs.  It finds that closed-source LLMs perform better on slang-heavy posts compared to their overall performance, attributing this to the rich background and cultural knowledge acquired during their broad pretraining."

The paper introduces a new dataset with detailed explanations of pre-processing and annotation. The dataset was constructed with ethical consideration and with the goal of mitigating bias. In addition, the dataset was used to fine-tune state-of-the-art LLMs, which provide valuable insights with descriptive empirical results on span-level Chinese hate-speech detection. The motivation and contributions are clearly described. The paper is well-crafted, and the novelty is clearly observed.
Summary of weaknesses: During the benchmark creation process, filtering was conducted to ensure objective tagging by removing ambiguous data. However, this may have led to the exclusion of challenging cases, potentially leaving only relatively easy to distinguish data.

Recent hate speech is often implicit. Therefore, if ambiguous target data is removed, it becomes impossible to evaluate a model’s ability to detect implicit hate speech.

Table 6 does not seem particularly insightful. What is the takeaway for those two models performing better than the others? Additionally, in some cases, model performance on the benchmark shows an F1-score of less than 10%. It raises concerns about the practicality and effectiveness of the benchmark.

The data in each category is not balanced. It can cause category bias.

The main contributions of this are resource-based (dataset + lexicon). The paper does not seem to introduce a novel model architecture or detection technique. Which is fine, but I think limits its impact overall, especially because it shows that LLMs still struggle with this, which is already known. They’re just re-asserting a known fact.

- The data filtering and selection process for building the dataset is not described in sufficient detail. The paper mentions removing low-quality or irrelevant content​, but it does not provide the exact procedure used for filtering. Important details are missing, like what instructions were given to filter out ambiguous cases and who performed this filtering. This makes it difficult to reproduce or verify the dataset construction process.


- The evaluation methodology for span-level extraction is not fully clear. It remains ambiguous how span evaluation is handled, specifically, whether it is computed at the character level, token level, or using some other method.

Details of the fine-tuning process are not provided, which hinders replicability. (Is there a validation set? How many epochs were each individual model trained? If there’s no validation set, how can we compare the performance of the fine-tuned models: since each has different numbers of parameters, over/under-fitting may occasionally have occurred? Details could have been provided in the Appendix to mitigate these concerns)

The availability of the dataset is not clear. (In the Ethics Statement, the authors state: “All resources are provided exclusively for scientific research purposes.” Will the dataset be made public after the paper is published?)
Comments, suggestions and typos: In line 28, ‘flourished’ generally has a positive connotation, I wouldn’t use it with the spread of hate speech.

In the paragraph starting with line 224 - I don’t see how that’s a hateful thing to say. AIDS is a big problem in the LGBTQ community, especially for gay men. This negates a lot of confidence in the integrity of the annotations.

It would be better if Section 4.3 also provided the percentage ratio for each figure.

It would be helpful if the paper suggested how this benchmark could be used in practice.

Utilizing the linguistic characteristic of Chinese for span-level target annotation was a good approach, but it would be better if the explanation included the characteristics of hate speech as well.

The reference to Figure 1 appears broken in the current version (Line 554)

In Section 5.1 (RQ1), results in Table 6 are elaborated, but there’s no explicit reference to this Table, which makes the reader search for these results manually. A reference to this table may be added.

As mentioned in the previous section, details of the fine-tuning process (no. epochs, presence of validation set, hyperparameters) can be provided in the Appendix to increase soundness and enable replicability. 

In Appendix A, Figure 2 contains some terms and definitions, which are only provided in Chinese. Optionally, an English translation may be added for the readers to better comprehend the contents of the annotated lexicon.
Overall assessment scores: 3

3

4