Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph
Authors: Yibo Zhao, Jiapeng Zhu, Can Xu, Yao Liu, Xiang Li
Accepted at: acl25

Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph

Yibo Zhao, Jiapeng Zhu, Can Xu, Yao Liu 1*, Xiang Li 1*
East China Normal University, Shanghai, China

Abstract

The rapid growth of social media platforms has raised significant concerns regarding online content toxicity. When Large Language Models (LLMs) are used for toxicity detection, two key challenges emerge: 1) the absence of domain-specific toxicity knowledge leads to false negatives; 2) the excessive sensitivity of LLMs to toxic speech results in false positives, limiting freedom of speech. To address these issues, we propose a novel method called MetaTox, leveraging graph search on a meta-toxic knowledge graph to enhance hatred and toxicity detection. First, we construct a comprehensive meta-toxic knowledge graph by utilizing LLMs to extract toxic information through a three-step pipeline. Second, we query the graph via retrieval and ranking processes to supplement accurate, relevant toxicity knowledge. Extensive experiments and case studies across multiple datasets demonstrate that our MetaTox boosts overall toxicity detection performance, particularly in out-of-domain settings. In addition, under in-domain scenarios, we surprisingly find that small language models are more competent. Our code is available at https://github.com/YiboZhao624/MetaTox.

Disclaimer: This paper describes toxic and discriminatory content that may be disturbing to some readers.

1 Introduction

Online social media platforms have become a major source of information for people worldwide. Meanwhile, they also provide a communication tool for spreading toxic content including harassment, trolling, cyberbullying, and hate speech, which poses a serious and continual threat to the harmony of society (Simpson, 2013) and harms children’s mental health (Simpson, 2019). It thus becomes a critical task to detect the toxic content , while cautiously avoiding hindering freedom of expression.

To address the problem, previous work can be mainly categorized into three types: (1) rule-based methods (Chen et al., 2012; Gittari et al., 2015) relying on the pre-defined rules; (2) embedding-based methods (Warner and Hirschberg, 2012; Djuric et al., 2015; Park and Fung, 2017) leveraging text representations for classification; (3) transformer-based methods (D’Sa et al., 2020; Luu and Nguyen, 2021) employing transformer architectures with small-scale parameters. The first two types of approaches struggle with semantic understanding, limiting their ability to detect implicit toxicity. While methods of the third type can capture more complex semantics, their reliance on domain-specific training data results in significant performance degradation on out-of-domain data. However, due to their lightweight nature and ease of training, they are widely applied.

The rapid development of large language models (LLMs) has provided new insights into the detection of toxic contents. Therefore, we try to explore whether LLM-based methods can outperform small

![A showcase of our method MetaTox, which enhances the LLM to detect toxicity by injecting speech-related toxic knowledge in form of triplets.](page_1012_384_384_384.png)

Figure 1: A showcase of our method MetaTox, which enhances the LLM to detect toxicity by injecting speech-related toxic knowledge in form of triplets.
models in this domain. Some studies (Yang et al., 2023; Zhang et al., 2024a) leverage LLMs as data augmentation tools to enhance toxicity detection. However, due to LLMs’ lack of domain-specific knowledge in hatred and toxicity, the performance of these methods is still limited. Further, recent research (Zhang et al., 2024b) highlights another issue: LLMs are extremely sensitive to groups or topics that may raise fairness concerns, such as race, gender, and religion, often leading to false positive misjudgment, which impairs the freedom of speech. For example, the benign post “forget this white nationalist mess. i’m america first...” is misjudged as toxic by LLM possibly because the model is too sensitive to the “white nationalist”.

To supplement external domain knowledge for LLMs, Retrieval Augmented Generation (RAG) (Lewis et al., 2020) has been widely adopted. However, naïve RAG-based methods heavily rely on the semantic similarity between the embeddings of queries and documents. When query contents are implicitly toxic with vague semantics, retrieved documents could be irrelevant and impair the model performance. Inspired by the recent success of GraphRAG (Edge et al., 2024), as well as the inherent nature of toxic content targeting specific groups, regions, and other entities, we organize external knowledge into a meta-toxic knowledge graph, and dig out toxicity knowledge highly relevant to speech via graph retrieval. The retrieved triplets or paths, which represent concentrated toxicity knowledge derived from the originally fragmented toxic corpus, capture rich meta-toxic information. This process facilitates more accurate detection of toxicity, thereby alleviating the issue of false positive misjudgments.

In this paper, we propose MetaTox, which utilizes a meta-toxic knowledge graph and LLMs for hate and toxic speech detection. We first construct a meta-toxic knowledge graph from existing toxic benchmark datasets by designing a three-step pipeline: rationale reasoning, triplet extraction, and entity resolution. Specifically, rationale reasoning aims to reason about what contents trigger toxicity in the speech; triplet extraction involves extracting toxic entities and relations, with a self-checking strategy to ensure the quality of these triplets; and entity resolution merges nodes and relations with similar semantics respectively. Subsequently, given a downstream potentially toxic speech, we query the knowledge graph by retrieval and ranking for toxicity detection. Retrieval returns relevant paths to the speech, while ranking further filters noise and refines the triplets in paths. We then leverage the extracted triplets that serve as toxic prompts to boost the LLMs’ capability in toxicity detection. **The entire process is training-free**. A showcase of our method is depicted in Figure 1. Our contributions are summarized as follows.
• We construct a novel meta-toxic knowledge graph. To our best knowledge, it is the first domain-specific knowledge graph for hate and toxic speech detection. We will open-source the knowledge graph upon paper acceptance.
• We propose an effective **training-free** method MetaTox for hate and toxic speech detection. In particular, MetaTox addresses the notorious false positive misjudgment issue, decreasing the ethical risk of hurting the freedom of speech.
• We conduct extensive experiments to evaluate the performance of MetaTox. Surprisingly, we find that small language models are more suitable for in-domain toxicity detection, while our method is superior in out-of-domain settings. This further sheds light on model selection in various toxicity detection scenarios.

2 Related Work

Previous methods can mainly be classified into rule-based, embedding-based, and transformer-based, and LLM-based approaches.

Rule-based methods (Chen et al., 2012; Gittari et al., 2015) rely on predefined rules that match specific patterns within the text. (Liu and Fors, 2015) uses the insults and swears words to form a dictionary for hatred and toxicity detection. Embedding-based approaches, on the other hand, combine text representations obtained by methods like word2vec (Djuric et al., 2015; Park and Fung, 2017). Transformer-based methods (D’Sa et al., 2020; Luu and Nguyen, 2021), typically perform fine-tuning on pre-trained models like BERT. Recent studies have explored the use of LLMs, which typically serve as a powerful data augmentation tool. Some methods (Lee et al., 2024) directly prompt LLMs to generate explanatory information about the text, which is then combined with the original text to train a detection model. Furthermore, some approaches (Yang et al., 2023; Zhang et al., 2024a) employ structured reasoning methods such as Chain-of-Thought (CoT) (Wei et al., 2024) and Tree-of-Thought (ToT) (Yao et al., 2024) to infer hidden semantics of input text. Some studies
aim to enhance the robustness of hatred and toxicity detection. For example, TextAttack (Morris et al., 2020) improves robustness through data augmentation, while Bespalov et al. (2023) proposes the ToxicTrap modular, leveraging adversarial training to strengthen model resilience.

However, previous approaches have either overlooked related real-world knowledge or relied solely on distilling the limited internal knowledge of LLMs. In contrast, our method is the first to build a hate-related KG and introduce it into hatred and toxicity detection, offering interpretive knowledge injection and multi-hop inference ability.

3 Methodology

In this section, we present MetaTox, our proposed method for enhancing hatred and toxicity detection using a meta-toxic knowledge graph. MetaTox consists of two stages: (1) construction of the meta-toxic knowledge graph through a three-step pipeline, and (2) querying the graph to enhance the downstream task of binary classification for toxicity detection. We begin with the data collection process, which lays the foundation for constructing the knowledge graph.

3.1 Data Collection

For meta-toxic knowledge graph construction, we leverage three well-established English datasets: HateXplain (Mathew et al., 2021), ToxicSpans (Pavlopoulos et al., 2021), and IHC (EISherief et al., 2021). Since our goal is to enhance toxicity detection by supplementing the LLM with accurate and curated toxicity knowledge, the knowledge graph should serve as a domain-specific corpus that reflects only toxic content. Therefore, we exclusively retain toxic samples labeled as “toxic”, “hate” or “offensive” from the training sets of these datasets to construct the knowledge graph.

For toxicity detection, we use the test sets from the three datasets, aligning each sample’s label with either “toxic” or “non-toxic” to formalize a binary classification task. This enables us to evaluate the LLM’s performance improvement in detecting potentially toxic speech with the support of our meta-toxic knowledge graph.

3.2 Graph Construction

We propose a three-step process to build the meta-toxic knowledge graph based on solely toxic samples: (1) rationale reasoning, which involves identifying the contents that trigger toxicity in speech, thereby uncovering implicit toxic meanings; (2) triplet extraction, where toxic entities and relations are extracted using a self-checking mechanism to ensure the triplets are correctly formatted and toxic; (3) entity resolution, which merges semantically similar entities and relations to reduce noise and shrink the graph. The three steps are illustrated in Figure 2 and described in detail below.

![The pipeline of meta-toxic knowledge graph construction, including rationale reasoning, triplet extraction, and entity resolution.](page_749_186_654_496.png)

Figure 2: The pipeline of meta-toxic knowledge graph construction, including rationale reasoning, triplet extraction, and entity resolution.

3.2.1 Rationale Reasoning

To construct the meta-toxic knowledge graph from initial toxic speech, it is essential to first identify the core toxic elements, i.e. entities and relations, within the speech. However, toxic semantics are often implicit and abstract, involving concepts such as race, gender and religion, rather than specific named entities. This introduces challenges in directly extracting these elements in a single step.

To address this, we employ a rationale reasoning step prior to triplet extraction, which applies LLMs to articulate why the speech is considered toxic, with relevant elements leading to this conclusion naturally incorporated into the rationale. For example, by informing the LLM that “white lives matter event” belongs to hate speech, the LLM explains: “a counter-movement to the Black Lives Matter movement, shows potential harm to the Black Lives Matter movement”. This explanation not only provides the context that triggers toxicity but also highlights the toxic entities in-
Figure 3: Graph query pipeline of MetaTox. We propose five steps including entity extraction, node mapping, path retrieval, formatting, and ranking and filtering to inject accurate knowledge to the LLM.

volved. Hence, this rationale reasoning step can be interpreted as a form of data augmentation, which draws out implicit toxic elements from the speech and makes them more explicit to better guide the LLM’s reasoning logic towards toxicity, thereby reducing difficulties for subsequent triplets extracting. We leverage in-context learning to enhance LLM’s ability to generate high-quality rationales, using carefully designed prompts. A detailed description of the prompt can be found in Appendix A.1.

It is important to emphasize that while LLMs exhibit excessive sensitivity and struggle to correctly judge when used for toxic content detection, our approach explicitly informs the model that the given text is toxic and leverages the model for reasoning rather than direct judgment or generation. This equates to providing the model with prior knowledge, where the model only needs to deduce the reasons for the toxicity from the text. A similar approach has also been adopted and validated in studies such as STaR (Zelikman et al., 2022), which guide models in generating new rationales based on correct answers, thereby helping them learn to solve increasingly difficult problems using the generated rationales.

3.2.2 Triplet Extraction
In this step, we take the toxic speech and the corresponding rationales as input and prompt the LLM to extract toxic triplets, represented in the Subject-Predicate-Object (SPO) format, such as “(white lives matter, is against, black lives matter)”. Specifically, we implement a template that instructs the LLM to extract triplets triggering hatred, given two exemplars to guide the LLM in understanding the extraction process and outputting triplets in a unifying SPO format. It is worth noting that the extracted relations are not constrained to a predefined set but are determined by the LLM. Specifically, we include a few illustrative examples of potential relations in the prompt template and instruct the LLM to generate triplets using relations that best reflect the underlying semantics, rather than selecting from a fixed list. The details of the prompts are shown in Appendix A.2.

However, we observed two main issues with the extracted triplets. First, due to occasional imprecision in following instructions, the LLM may generate triplets that deviate from the standard SPO format, like omitting a subject or object. Second, the triplets may fail to capture toxic semantics, possibly due to inherent ambiguity in hate speech.

Thus, we propose a self-checking mechanism to refine the extracted triplets, as illustrated with pictorial cases in Figure 2. For triplets that are incorrectly formatted, we discard them using regular expressions. For triplets that fail to capture toxic semantics, we urge the LLM to filter out non-toxic triplets with few-shot prompting, ensuring that only those triplets capable of evoking hatred and toxicity are retained. Details of the filtering prompt are provided in Appendix A.3.

After applying the self-checking mechanism, we obtain a curated set of toxic triplets that explicitly reflect toxic semantics, effectively leveraging external domain knowledge.

3.2.3 Entity Resolution
In this step, we take the extracted triplet elements (entities and relations) as input, applying a clus-
tering algorithm to merge these elements and ultimately generate a meta-toxic knowledge graph with entities resolved. Unlike traditional entity resolution methods for knowledge graphs, our approach simplifies the standardization process by assigning identical names to entities within the same cluster, rather than relying on specific Internationalized Resource Identifiers (IRIs).

To perform clustering on the triplet elements, we first use a pre-trained BERT model (Devlin et al., 2019) to encode the textual attributes of entities and relations into embeddings. We then apply a clustering algorithm to group similar entities and relations respectively, based on their textual embeddings. For deduplication, we determine the name that appears most frequently within each cluster and assign it as the unified name for all elements in that cluster. In other words, we use clustering combined with a voting scheme to merge similar elements. This method not only helps resolve slight spelling variations, such as “Jew” and “jew”, but also standardizes elements with different names referring to the same concept, such as “LGBT” and “LGBTQ+”. To prevent over-merging, we set a relatively high similarity threshold.

The benefits of entity resolution are twofold. On the one hand, it reduces noise by eliminating unnecessary spelling differences and retaining representative expressions supported by most elements. On the other hand, it decreases the overall size of the graph, which in turn enhances the efficiency of graph retrieval.

3.3 Graph Query

Given a potentially toxic speech, we devote to querying on the meta-toxic knowledge graph to supplement speech-related toxic triplets, thereby providing the LLM with accurate, domain-specific guidance when judging the hatred and toxicity of the given speech. Briefly, we divide the query process into Retrieval (Entity Extraction, Node Mapping, Path Retrieval) and Ranking (Formatting, Ranking and Filtering). The overall query procedure is illustrated in Figure 3.

Entity Extraction aims to identify various entities in the given speech, treating them as candidate toxic entities for further validation. Given that explicit toxic entities may be absent, we employ the LLM to extract as many relevant entities as possible, including both specific named entities and broader concepts such as race, gender, and religion. The detailed instruction is shown in Appendix A.4.

Node Mapping faces the challenge that entities extracted from the speech may not exactly match the names of nodes in the meta-toxic knowledge graph. To resolve this, we formulate it as a dense retrieval task, where the most semantically similar node is mapped to each extracted entity based on textual embeddings generated by a pre-trained BERT model. At this point, candidate toxic entities are identified within the knowledge graph. The Faiss library (Douze et al., 2024) is applied to optimize retrieval efficiency.

Path Retrieval extracts coherent toxic knowledge contained from the graph based on the paths that connect previously mapped nodes through one or multi-hop relations. While a straightforward retrieval approach would be to extract all neighbors for each mapped node, this may result in recalling excessive unrelated triplets that could degrade toxicity detection performance, particularly when sensitive entities such as “Jew”, “Nigger”, “Nazi”. To mitigate the noise introduced by unnecessary entities or relations, we enumerate pairwise combinations of the mapped nodes and retrieve the shortest path for each node pair. After splitting all the paths into SPO triplets, we take the union of them as the candidate retrieved triplets, reflecting toxicity knowledge potentially related to the text.

Formatting transforms the retrieved triplets into candidate knowledge with natural language by concatenating the SPO elements of each triplet, like “white lives matter is against black lives matter”. After ranking and filtering each remaining knowledge will ultimately be sent to the LLM following this format as part of the prompts.

Ranking and Filtering is introduced to rank the candidate knowledge and discard irrelevant ones for denoising, as LLMs are sensitive to the prompt content. Specifically, we rank the candidate knowledge by sorting the cosine similarity between each candidate’s knowledge and the input speech in descending order. This prioritizes toxicity knowledge that is more relevant to the speech. Additionally, we filter out less related knowledge based on similarity, which are considered as adverse noise.

After querying the meta-toxic knowledge graph, the ultimate toxicity knowledge is retrieved as supplementary information. We then insert the retrieved knowledge into the prompt template utilized for enhancing LLM’s ability to detect hatred and toxicity. Two exemplars are designed to inform the LLM on utilizing external toxicity knowledge. The prompting template is presented in Appendix A.5.
4 Experiments

As outlined in Section 3.1, we construct three meta-toxic knowledge graphs based on the HateXplain, ToxicSpans, and IHC datasets, respectively, with Qwen2.5-14B-Instruct (Qwen) (Qwen Team, 2024). Then Qwen and Llama3.1-8B-Instruct (Llama) (AI@Meta, 2024) are employed to generate predictions. We evaluate MetaTox from three perspectives: (1) Graph Evaluation, which provides qualitative and quantitative analysis on the constructed graphs; (2) Toxicity Prediction, where MetaTox is compared to baseline methods under both in-domain and cross-domain settings to evaluate effectiveness and robustness of MetaTox; (3) Case Studies, which offer in-depth analyses by guiding LLMs to output reasoning paths, providing insights into how MetaTox enhances interpretability and reasoning abilities. To validate the rationale behind our pipeline design, we have further devised an ablation study, the details of which can be found in Appendix B.

4.1 Graph Evaluation

For the constructed KG, to verify whether the triplets contain hate-related semantics and adhere to the required format, we conduct a manual evaluation. To conserve human resources, we sampled 100 triplets from each of the three constructed KGs. Additionally, to mitigate annotators’ potential bias towards classifying all triplets as hate-related, we introduce an equal proportion of triplets generated by LLMs that do not contain hate-related information, blending them in a 1:1 ratio.

We assembled a team of 5 annotators who are master’s and PhD students, proficient in English, and have some relevant background knowledge. We paid the annotators a wage higher than the minimum wage in their local area. Before annotating, we informed them about the task and ensured that they fully understood and consented to annotate hate speech triplets.

Given the complexity of annotating hate speech triplets, we specifically emphasized three rules: 1) A triplet is considered correct only if it meets the required format and contains hate-related content. 2) When the correctness of a data point was unclear, annotators were instructed to search online first. 3) After completing the annotation, the five annotators should discuss to reach an agreement. The final accuracy of HateXplain, IHC, ToxicSpans KGs is 89%, 87% and 82%, respectively. Most of the triplets are correct, while the main types of errors are format issues and the use of pronouns without specifying the entities they referred to.

To illustrate the scale information of the KGs, we also conducted statistical analysis on the graph. We analyze the graph properties, particularly the number of nodes and relations, to assess the effectiveness of our data mining approach. Additionally, we examine the merging ratio by integrating meta-toxic knowledge graphs derived from different datasets, demonstrating how the meta-toxic knowledge graph expands. After merging three datasets, we observe a 21.52% reduction in the number of entities and a 3.25% reduction in the number of triplets in the merged graph. This suggests that the targets of toxic speech exhibit significant overlap across datasets, supporting the transferability of our approach. The detailed quantitative results are presented in Table 1. To facilitate the understanding of the knowledge graph structure, we illustrate sampled triplets in Appendix C.

<table>
  <tr>
    <th></th>
    <th>Toxic Samples</th>
    <th>Entities</th>
    <th>Triplets</th>
  </tr>
  <tr>
    <th>IHC</th>
    <td>7,373</td>
    <td>20,043</td>
    <td>25,534</td>
  </tr>
  <tr>
    <th>HateXplain</th>
    <td>10,273</td>
    <td>24,442</td>
    <td>33,276</td>
  </tr>
  <tr>
    <th>ToxicSpans</th>
    <td>9,905</td>
    <td>21,667</td>
    <td>30,347</td>
  </tr>
  <tr>
    <th>Merged</th>
    <td>27,551</td>
    <td>51,917<sup>†</sup></td>
    <td>86,350<sup>‡</sup></td>
  </tr>
</table>

Table 1: Data Summary. † means that it has been reduced by 21.52% compared to before the merge; ‡ means 3.25% off.

4.2 Toxicity Prediction

We conduct experiments in both in-domain and cross-domain scenarios, comparing our method with vanilla LLM and the naive RAG-enhanced LLM for toxicity detection. Further, to assess whether LLMs can outperform fine-tuned small language models , we also include HateBERT (Caselli et al., 2021) and BERT (Devlin et al., 2019) as baselines. It’s worth noting that HateBERT requires an initial retraining phase of 18 days and 2 million steps, followed by supervised fine-tuning (SFT) on downstream tasks. For the vanilla LLM, we directly input the test speech into the LLM and prompt it to provide the prediction. For the naive RAG method, we retrieve the top-2 most similar speeches from the training set as additional knowledge. To ensure a fair comparison, the training set used for the RAG method is the same as the data
Table 2: Results on using meta-toxic knowledge graph built from the same dataset, best results are highlighted.

<table>
  <tr>
    <th rowspan="2">Backbone Model</th>
    <th colspan="3">Qwen2.5-14B-Instruct</th>
    <th colspan="3">Llama3.1-8B-Instruct</th>
    <th colspan="2">HateBERT</th>
    <th colspan="2">BERT</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Vanilla LLM</th>
    <th>RAG</th>
    <th>MetaTox</th>
    <th>Vanilla LLM</th>
    <th>RAG</th>
    <th>MetaTox</th>
    <th>SFT</th>
    <th>SFT</th>
  </tr>
  <tr>
    <td rowspan="4"><i>HateXplain</i></td>
    <td>Acc.↑</td>
    <td>70.95</td>
    <td>73.13</td>
    <td><b>73.39</b></td>
    <td>63.36</td>
    <td>69.59</td>
    <td>68.87</td>
    <td>77.96</td>
    <td><b>78.17</b></td>
  </tr>
  <tr>
    <td>F1↑</td>
    <td>64.04</td>
    <td>70.18</td>
    <td>72.48</td>
    <td>48.11</td>
    <td>62.02</td>
    <td>62.50</td>
    <td><b>81.96</b></td>
    <td>81.34</td>
  </tr>
  <tr>
    <td>AUC↑</td>
    <td>76.36</td>
    <td>83.70</td>
    <td>84.02</td>
    <td>79.77</td>
    <td>80.50</td>
    <td>82.27</td>
    <td>88.82</td>
    <td><b>90.44</b></td>
  </tr>
  <tr>
    <td>FPR↓</td>
    <td>66.62</td>
    <td>48.72</td>
    <td>32.10</td>
    <td>88.75</td>
    <td>40.69</td>
    <td>38.74</td>
    <td>31.33</td>
    <td><b>24.94</b></td>
  </tr>
  <tr>
    <td rowspan="4"><i>IHC</i></td>
    <td>Acc. ↑</td>
    <td>66.34</td>
    <td>66.71</td>
    <td>73.65</td>
    <td>50.79</td>
    <td>64.29</td>
    <td>69.04</td>
    <td>78.35</td>
    <td><b>78.72</b></td>
  </tr>
  <tr>
    <td>F1↑</td>
    <td>66.32</td>
    <td>66.67</td>
    <td>69.95</td>
    <td>48.03</td>
    <td>64.23</td>
    <td>68.55</td>
    <td><b>71.52</b></td>
    <td>70.57</td>
  </tr>
  <tr>
    <td>AUC↑</td>
    <td>64.38</td>
    <td>66.79</td>
    <td>70.10</td>
    <td>62.97</td>
    <td>66.74</td>
    <td>63.75</td>
    <td>79.70</td>
    <td><b>79.74</b></td>
  </tr>
  <tr>
    <td>FPR↓</td>
    <td>48.42</td>
    <td>43.24</td>
    <td><b>12.32</b></td>
    <td>77.63</td>
    <td>51.58</td>
    <td>34.23</td>
    <td>17.49</td>
    <td>14.19</td>
  </tr>
</table>

Table 3: Results on using meta-toxic knowledge graph built from another dataset, best results are highlighted.

<table>
  <tr>
    <th rowspan="2">Backbone Model</th>
    <th colspan="3">Qwen2.5-14B-Instruct</th>
    <th colspan="3">Llama3.1-8B-Instruct</th>
    <th colspan="2">HateBERT</th>
    <th colspan="2">BERT</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Vanilla LLM</th>
    <th>RAG</th>
    <th>MetaTox</th>
    <th>Vanilla LLM</th>
    <th>RAG</th>
    <th>MetaTox</th>
    <th>SFT</th>
    <th>SFT</th>
  </tr>
  <tr>
    <td rowspan="4"><i>HateXplain</i></td>
    <td>Acc.↑</td>
    <td>70.95</td>
    <td>71.21</td>
    <td><b>73.28</b></td>
    <td>63.36</td>
    <td>65.96</td>
    <td>68.24</td>
    <td>59.30</td>
    <td>57.38</td>
  </tr>
  <tr>
    <td>F1↑</td>
    <td>64.04</td>
    <td>64.97</td>
    <td><b>72.38</b></td>
    <td>48.11</td>
    <td>54.32</td>
    <td>61.63</td>
    <td>69.11</td>
    <td>45.78</td>
  </tr>
  <tr>
    <td>AUC↑</td>
    <td>76.36</td>
    <td>81.29</td>
    <td><b>83.80</b></td>
    <td>79.77</td>
    <td>81.50</td>
    <td>81.67</td>
    <td>72.63</td>
    <td>58.57</td>
  </tr>
  <tr>
    <td>FPR↓</td>
    <td>66.62</td>
    <td>64.32</td>
    <td>32.10</td>
    <td>88.75</td>
    <td>80.95</td>
    <td>67.13</td>
    <td>66.11</td>
    <td>86.32</td>
  </tr>
  <tr>
    <td rowspan="4"><i>IHC</i></td>
    <td>Acc. ↑</td>
    <td>66.34</td>
    <td>63.36</td>
    <td><b>73.42</b></td>
    <td>50.79</td>
    <td>52.84</td>
    <td>69.04</td>
    <td>45.86</td>
    <td>40.36</td>
  </tr>
  <tr>
    <td>F1↑</td>
    <td>66.32</td>
    <td>63.34</td>
    <td><b>69.42</b></td>
    <td>48.03</td>
    <td>51.01</td>
    <td>68.85</td>
    <td>45.06</td>
    <td>37.66</td>
  </tr>
  <tr>
    <td>AUC↑</td>
    <td>64.38</td>
    <td>67.21</td>
    <td><b>69.78</b></td>
    <td>62.97</td>
    <td>61.15</td>
    <td>63.63</td>
    <td>39.88</td>
    <td>33.27</td>
  </tr>
  <tr>
    <td>FPR↓</td>
    <td>48.42</td>
    <td>43.24</td>
    <td><b>11.64</b></td>
    <td>77.63</td>
    <td>72.97</td>
    <td>34.23</td>
    <td>61.86</td>
    <td>84.23</td>
  </tr>
</table>

used to construct our meta-toxic knowledge graph.

To apply LLMs for classification, we follow the evaluation approach from MMLU (Hendrycks et al., 2021), where classification is determined by comparing the logit values of candidate options “a” (toxic) and “b” (non-toxic). Our evaluation metrics include classification accuracy (Acc.), F1 score (F1), and precision-recall area under the curve (AUC). Additionally, we calculate the false positive rate (FPR) to assess how effectively our method reduces false positives.

4.2.1 In-domain Setting
For the in-domain setting, the graph construction and toxicity detection are performed on the training and test sets of the same dataset, respectively.

As illustrated in Table 2, MetaTox demonstrates performance improvements across the HateXplain and IHC datasets with two backbone models, consistently outperforming baseline methods. The results reveal key insights into the effectiveness of our approach. For HateXplain, where toxic semantics are relatively explicit, the related knowledge is easier to retrieve by naive RAG, improving the performance to some extent. However, MetaTox outperforms naive RAG, primarily due to the more concise and relevant knowledge provided by KG.

In contrast, for IHC, where toxic semantics are more implicit, overall performance is lower compared to HateXplain. Using Qwen, while naive RAG yields only marginal improvement, MetaTox achieves a remarkable performance boost. This discrepancy may stem from the fact that documents retrieved by naive RAG often contain excessive unrelated information. In contrast, MetaTox, through the combined contributions of knowledge graph construction and query on the graph, not only retrieves the most related knowledge but also organizes it into concise triplets, significantly improving the model’s understanding of the provided knowledge. When using the less powerful Llama for detection, MetaTox achieves performance slightly better than naive RAG, reflecting the advantage of its more effective knowledge integration, even with limited model capacity. In brief, our method effectively enhances the performance for in-domain scenario.

Compared to HateBERT and fine-tuned BERT, surprisingly, we find that a simple BERT-based model, after training, can outperform a 14B LLM under in-domain scenarios. The key distinction lies in the way of capturing toxicity: our method explicitly presents hate-related information with KG triplets, whereas HateBERT implicitly recognizes toxicity patterns by SFT. We speculate that small models’ performance advantage attributes to supervised learning, which enables them to memorize toxicity knowledge within the
Lack of Domain-Specific Knowledge

Ground Truth: Toxic
blacks are 1/6 white population . shoot someone at random 5/6 chance they are white .

Prediction: Non-toxic
Explanation: The statement does not contain any hate speech or derogatory language towards any group of people. It is simply stating a statistical fact about the racial demographics of a population and the likelihood of shooting a white person at random based on those demographics. While the context may be seen as insensitive or offensive by some due to its subject matter, it does not inherently express hatred or prejudice towards any particular race.

Related Speech:
1. you should see sm. towns in arkansas; majority were white 70% blacks 25% other 5%. now. white 45%; black 20% & mexican (maj illegals) 35% Label: Toxic
2. blacks are 32.6 times more likely to attack a white person than a white person to attack a black. Label: Non-toxic

Prediction: Non-toxic
Explanation: The statement, while potentially controversial, does not contain hate speech or harmful stereotypes. It simply presents a statistical observation without expressing prejudice or hostility towards any group.

Related Toxic Triplets:
(Shoot, promote, Murder)
(individuals, demean, savage blacks),
(Whites, promote, racial hierarchy)
(racial hierarchy, promote, discrimination)

Prediction: Toxic
Explanation: The context suggests shooting someone randomly and mentions the racial demographics, implying a bias against non-whites and promoting violence based on race, which aligns with the provided triplets that include demeaning and discriminatory sentiments towards different racial groups. This reflects a hateful stance.

False Positive Misjudgment

Ground Truth: Non-toxic
retweeted elder lansing ( ): forget this white nationalist mess . I'm america first . so is our great ...

Prediction: Toxic
Explanation: The tweet contains phrases that suggest a negative sentiment towards a specific group "white nationalist mess" and implies exclusionary nationalism "I'm america first". This can be perceived as promoting division and hostility towards those who do not align with their nationalistic views. Therefore, I would classify this as hateful content.

Related Speech:
1. retweeted elder lansing ( ): it's called the white house not the white supremacist ... Label: Non-toxic
2. retweeted elder lansing ( ): demokkrats stop your racist deceit / obama sat underneath a racist ... Label: Toxic

Prediction: Toxic
Explanation: The phrase "white nationalist" promotes racial superiority and hatred. "America First" can imply exclusionary nationalism, often targeting specific racial or ethnic groups. Therefore, this statement includes elements of hate speech.

Related Toxic Triplets:
no related triplets, it is more likely to be non-toxic speech.

Prediction: Non-toxic
Explanation: The context does not contain any explicit hate speech or content that aligns with the provided triplets. Although the phrase "white nationalist mess" can sometimes be associated with nationalist ideologies, the text itself does not express hate towards any specific group based on the information given. Without additional context indicating hate, we cannot classify this as toxic.

Figure 4: Example of false negative misjudgment caused by lack of domain-specific knowledge. From top to bottom, each block shows the ground truth label and context, followed by the predictions and reasoning of the vanilla LLM, naive RAG, and MetaTox.

Figure 5: Example of false positive misjudgment caused by LLMs’ extreme sensitivity towards certain phrases. The blocks are distributed the same as Figure 4.

training data (Chu et al., 2025). While HateBERT performs well on static datasets, toxic detection requires rapid updates in fast-evolving social media environments. This makes our approach, which constructs a knowledge graph that is easier to edit and expand, more suitable for such tasks. Therefore, we conduct experiments in a cross-domain setting as discussed below.

4.2.2 Cross-domain Setting
For the cross-domain setting, the knowledge graph is constructed using the training set of ToxicSpans, while detection is performed on test sets of HateXplain and IHC. Correspondingly, naive RAG is only allowed to retrieve documents from ToxicSpans.

The experimental results are presented in Table 3. Notably, we observe a performance drop with naive RAG on IHC and negligible improvement on HateXplain. This can be attributed to the different facets of toxic semantics between ToxicSpans and test datasets. As analyzed in Section 4.2.1, the text representation-based method, HateBERT and BERT, struggles in OOD scenarios, resulting in a significant performance drop. HateBERT performs slightly better than BERT, likely due to its extensive retraining on a relatively large toxicity corpora, RAL-E dataset. In contrast, MetaTox still maintains a notable improvement in performance, because by incorporating external accurate toxicity knowledge, LLMs can uncover the underlying, invariant toxicity essence concealed within the diverse expressions of toxic content, therefore enhancing generalization.

It is worth noting that our method outperforms other LLM-based approaches across all scenarios, achieving the lowest or near-lowest false positive rate, effectively mitigating the risk of infringing on freedom of speech caused by false positives.
4.3 Case Study
To further analyze how MetaTox incorporates domain-specific knowledge and mitigates false positive errors, we present two case studies with the reasoning explanations provided by the LLM.

As shown in Figure 4, the original speech implicitly promotes toxicity by stirring up antagonism between blacks and whites. The word “shoot” also emphasizes the racial disparities. The vanilla LLM incorrectly focuses on numbers like “1/6” and “5/6”, which misleads the LLM to interpret the speech as a statistical analysis rather than a toxic trigger. When enhancing with naive RAG, the retrieved relevant texts contain statistics like “70%”, and “32.6 times”, indicating that the retriever fails to understand the intended meaning of the text. This further deepened the model’s misunderstanding, leading to misjudgments. On the contrary, MetaTox correctly identifies pivotal elements like “shoot”, and “racial hierarchy”, guiding the model to correctly interpret the speech as promoting race-based violence and predict correctly.

As shown in Figure 5, the original context emphasizes unity by stating “I’m America first,” while suggesting ignoring toxic content presented by “white nationalist mess”. However, the vanilla LLM misclassifies it as toxic because it overly focuses on phrases like “America First” and “white nationalist mess”, incorrectly interpreting them as indicators of toxicity. When enhanced with naive RAG, the LLM retrieves two related retweets for the same post. However, since both retweets focus on racial issues, even though they express different opinions, they fail to shift the LLM’s focus away from racial attributes, resulting in a false positive misjudgment. In contrast, MetaTox effectively filters the retrieved triplets, directing the model to follow the prompt instructions that “when there are no related triplets, the text is more likely to be non-toxic”, which significantly reduces the false positive misjudgment.

5 Conclusion
In this paper, we proposed a novel method called MetaTox to address both false positive and false negative misjudgments caused by the lack of domain-specific knowledge and LLMs’ extreme sensitivity. First, we leverage LLMs to extract toxic content through a three-step pipeline, which builds the meta-toxic knowledge graph. Next, we query the graph with retrieval and ranking processes to provide additional, relevant toxic knowledge. Extensive experiments and detailed case studies across various datasets show that MetaTox significantly lowers the false positive rate while improving overall toxicity detection performance in OOD scenarios. In addition, under in-domain scenarios, we surprisingly find that small language models are more competent.

Acknowledgement
This work was supported by Shanghai “Science and Technology Innovation Action Plan” Project under Grant No.23511100700 and National Natural Science Foundation of China under Grant 42375146.

Limitation
Our study still has several limitations. First, due to the computational constraint, we did not conduct experiments with larger LLMs, suggesting that the full potential of our method remains to be further explored. Future research could leverage more powerful LLMs to construct even more comprehensive meta-toxic knowledge graphs. Additionally, our method is currently limited to binary classification in English-language scenarios. Future work can extend our approach to multi-modal, multilingual, and multi-cultural tasks, thereby broadening its applicability across diverse contexts.

Ethical Statement
Our research focuses on toxic speech detection, primarily addressing two key issues faced by LLMs in this task: false negatives caused by a lack of domain-specific knowledge, and false positives resulting from excessive sensitivity to certain content. Our primary goal is to facilitate more accurate detection of toxicity, thereby alleviating the issue of false positive misjudgments and protecting freedom of speech, which aims to contribute to the creation of a more harmonious online environment. We construct a meta-toxic knowledge graph filled with toxic content. However, there is also a risk of potential misuse of our proposed method. Specifically, our methods might be misused to generate toxic speech. It is essential to emphasize that our work focuses on detecting toxic speech while safeguarding freedom of speech, instead of censorship. The datasets we used are all existing open-source datasets, aligning with their intention for scientific research. We also adhered to the MIT license for HateXplain and IHC datasets, and the CC0-1.0 license for the ToxicSpans dataset.
References

AI@Meta. 2024. Llama 3 model card.

Dmitriy Bespalov, Sourav Bhabesh, Yi Xiang, Liutong Zhou, and Yanjun Qi. 2023. Towards building a robust toxicity predictor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 581–598, Toronto, Canada. Association for Computational Linguistics.

Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17–25, Online. Association for Computational Linguistics.

Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. In 2012 international conference on privacy, security, risk and trust and 2012 international confernece on social computing, pages 71–80. IEEE.

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. Preprint, arXiv:2501.17161.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vlada Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the 24th international conference on world wide web, pages 29–30.

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazare, Maria Loneli, Lucas Hosseini, and Hervé Jégou. 2024. The faiss library.

Ashwin Geet D’Sa, Irina Illina, and Dominique Fohr. 2020. Bert and fasttext embeddings for automatic detection of toxic speech. In 2020 International Multi-Conference on: “Organization of Knowledge and Advanced Technologies” (OCTA), pages 1–5.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130.

Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Mummun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Njagi Dennis Gitari, Zhang Zuping, Hanyurwimfura Damien, and Jun Long. 2015. A lexicon-based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4):215–230.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).

Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4205–4224, Mexico City, Mexico. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Shuhua Liu and Thomas Forss. 2015. New classification models for detecting hate and violence web content. In 2015 7th international joint conference on knowledge discovery, knowledge engineering and knowledge management (IC3K), volume 1, pages 487–495. IEEE.

Son T Luu and Ngan Luu-Thuy Nguyen. 2021. Uit-ise-nlp at semeval-2021 task 5: Toxic spans detection with bilstm-crf and toxicbert comment classification. arXiv preprint arXiv:2104.10100.

Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hateexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867–14875.

John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the
2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126, Online. Association for Computational Linguistics.

Ji Ho Park and Pascale Fung. 2017. One-step and two-step classification for abusive language detection on twitter. arXiv preprint arXiv:1706.01206.

John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. SemEval-2021 task 5: Toxic spans detection. In Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021), pages 59–69.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Robert Mark Simpson. 2013. Dignity, harm, and hate speech. Law and Philosophy, 32(6):701–728.

Robert Mark Simpson. 2019. ‘won’t somebody please think of the children?’ hate speech, harm, and childhood. Law and Philosophy, 38(1):79–108.

William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the second workshop on language in social media, pages 19–26.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.

Yongjin Yang, Joonkee Kim, Yujin Kim, Namgyu Ho, James Thorne, and Se-Young Yun. 2023. HARE: Explainable hate speech detection with step-by-step reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5490–5505, Singapore. Association for Computational Linguistics.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc.

Jiang Zhang, Qiong Wu, Yiming Xu, Cheng Cao, Zheng Du, and Konstantinos Psounis. 2024a. Efficient toxic content detection by bootstrapping and distilling large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19):21779–21787.

Min Zhang, Jianfeng He, Taoran Ji, and Chang-Tien Lu. 2024b. Don’t go to extremes: Revealing the excessive sensitivity and calibration limitations of LLMs in implicit hate speech detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12073–12086, Bangkok, Thailand. Association for Computational Linguistics.
A  Prompts

A.1  Rationale Reasoning Prompt

Rationale Reasoning Prompt

<USER>: You are a helpful assistant aiming at hateful speech detection. For each message, give the reason why it is hateful. Here are 2 examples:
Example 1:
<USER>: Context 1: The Democrat party is like a dog sniffing its own tail. It stinks.
<ASSISTANT>: Analysis 1: The context is considered hateful because it uses a derogatory comparison to dehumanize a political group (the Democrat party) by likening it to a dog, which implies foolishness or absurdity. Additionally, the phrase 'it stinks' conveys a strong negative judgment, suggesting disdain and contempt. Such language can foster hostility and division, contributing to a culture of hate rather than constructive discourse.
Example 2:
<USER>: Context 2: The working middle class is an easy target! Have fun bleeding us dry with more taxes & Sisyphean social programs, elite upper class robber barons!
<ASSISTANT>: Analysis 2: The context is considered hateful because it employs inflammatory language by labeling the upper class as 'robber barons', which evokes historical connotations of greed and exploitation. This framing dehumanizes individuals within that socioeconomic group and fosters animosity. Additionally, the phrase 'bleeding us dry' expresses a strong sense of victimization and resentment, promoting division between classes rather than encouraging constructive dialogue. Such rhetoric can perpetuate class hatred and animosity.
<USER>: New Context: context
<ASSISTANT>: Analysis:

A.2  Triplet Extraction Prompt

Triplet Extraction Prompt

<USER>: You are a helpful assistant skilled in inductive reasoning and summarization. For each message, you will receive an original context and an analysis. Your task is to extract triplets that trigger hate.
In general, the context is more important and the analysis is just for assisting. The output should start with 'Output' and be with the format (head object A, predicate, tail object B) like (King, is, man). Also, there is a special condition that if the two phrase or word co-appear, they will lead to hate, you can use the "co-appear" as the relation and the two phrases or words as the head object and the tail object. There are some recommended relations, you can use them if they are suitable: is, demean, promote, refer to, imply, link, co-appear, insult, downplay, etc. For entities, it is better to use the original words instead of the conclusion of the analysis. Here are two examples:
Example 1:
<USER>: Input 1: Context: the persecution of christians isn’t as bad as the persecution of muslims.
Analysis: The context is considered hateful because it downplays the suffering of one group (Christians) by comparing it to the suffering of another (Muslims).
This minimizes the severity of the persecution faced by Muslims and can be seen as a form of victim-blaming.
Furthermore, the implication that Christians are being persecuted is not universally recognized or acknowledged, and this statement can be seen as perpetuating a narrative that is divisive and can be hurtful to those who experience persecution.
<ASSISTANT>: Output 1:
(persecution of Christians, is not as bad as, persecution of Muslims),
(comparing to suffering of Muslims, downplay, suffering of Christians)
Example 2:
<USER>: Input 2: Context: just shows how many retarded jerks we have living here buying everything he has. yea, that means your kids you stupid clueless parents, wise up.
Analysis: The context is considered hateful because it uses derogatory language to insult individuals based on their socioeconomic status (retarded jerks) and their ability to make informed decisions (clueless parents).
The use of the word retarded* is particularly egregious as it’s a term that has been historically used to demean people with intellectual disabilities.
This language is not only hurtful but also discriminatory, perpetuating stigma and hate towards those who may be vulnerable.
The final sentence, wise up,is a condescending and aggressive statement that further escalates the hate speech.
<ASSISTANT>: Output 2:
(retarded jerk, co-appear, parents),
(retarded jerk, insult, individuals),
(wise up, condescend, aggressive statement)
<USER>: New Input: context
<ASSISTANT>: Output:
A.3 Self-checking Prompt

Self-checking Prompt

<USER>: Please select the triplet evoking the hate from the following triplets began with 'Input:' and within the []. In general, the triplet should be in form like (Object A, relation, Object B), where A is a noun phrase or word, relation is a predicate(verb or a linking verb). If the input triplets don't fit the format, you should rewrite the input triplets first. The number of triplet is not fixed, can be one or several, you are required to select all hate triggered triplets. Finally, output the hate triggered triplets in [] only.
Here are two examples:
Example 1:
<USER>: Input 1: [(downplays suffering of Christians, by comparing to, suffering of Muslims),(persecution of Christians, is not as bad as, persecution of Muslims),(minimizes severity of persecution, faced by, Muslims),(implication that Christians are being persecuted, is not universally recognized, acknowledgment)]
<ASSISTANT>: Output 1: [(persecution of Christians, is not as bad as, persecution of Muslims),(comparing to suffering of Muslims, downplay, suffering of Christians).]
Example 2:
<USER>: Input 2: [(retarded jerks, shows, socioeconomic status),(clueless parents, buying everything, he has),(derogatory language, insult, individuals),(wise up, condescend, aggressive statement)]
<ASSISTANT>: Output 2: [(retarded jerk, insult, individuals), (wise up, condescend, aggressive statement)]
<USER>: Input: context
<ASSISTANT>: Output:

A.4 NER Prompt

Named Entity Recognition Prompt

Extract and list all distinct entities from the following text. Only return the entities as a comma-separated list like following template:
Input: <sentence>
Output: <entity1>,<entity2>,<entity3> ...
Example:
Text: u are the best gift for Muslim as you r a pig.
Entities: gift,Muslim,pig
Text: text
Entities:

A.5 Query Prompt

Query Prompt

<Instructions>: You are a helpful assistant skilled in inductive reasoning. For each message, you will receive a context and related triplets as reference. All the triplets will lead to hate. If the context contains the same meaning with one of the triplets, it is hateful. Otherwise, it is more possible to be benign, but you should leverage your knowledge to judge it. Your task is to determine whether the context is hateful or not with a single alphabet letter among given answer choices a and b. You should only output a or b.
a: { [Hateful] }
b: { [Non-hateful] }
Here are 2 examples:
Example 1:
<USER>: TRIPLETS: (niggers, is described as, stupid),
CONTEXT: Niggers are stupid.
<ASSISTANT>: a
Example 2:
<USER>: TRIPLETS: (white woman, shows, racism), (racism, aiming at, black man)
CONTEXT: That old white lady condemns any type of racism against black men.
<ASSISTANT>: b
New Input:
<USER>: TRIPLETS: {triplets}
CONTEXT: {context}
<ASSISTANT>:
B  Ablation Study

To validate the effectiveness and rationality of each step in our retrieval pipeline, we designed an ablation study. We selected Qwen-14B-Instruct as the backbone model and conducted the experiment on the HateXplain dataset. Specifically, we compared the settings without rerank & filter and those replacing shortest path search with 1-hop neighbor search. The results are presented in Table 4.

By comparing the results of the first and second rows, we find that rank & filter significantly improves performance. Similarly, comparing the second and third rows shows that shortest path search outperforms 1-hop neighbor search. This may be because the elements involved in shortest path search are more relevant, reducing input noise.

Table 4: The ablation results on the HateXplain dataset based on Qwen2.5-14B-Instruct.

<table>
  <tr>
    <th>method</th>
    <th>Acc.</th>
    <th>F1</th>
  </tr>
  <tr>
    <th>MetaTox(w/ rank & filter, shortest path)</th>
    <td>73.39</td>
    <td>72.48</td>
  </tr>
  <tr>
    <td>w/o rank & filter, w/ shortest path</td>
    <td>70.17</td>
    <td>71.73</td>
  </tr>
  <tr>
    <td>w/o rank & filter, w/ 1-hop subgraph</td>
    <td>65.38</td>
    <td>69.18</td>
  </tr>
</table>

C  Examples in Knowledge Graph

```json
{
  'nodes': [
    'nigger', 'stupid','Jews',
    'Muslims', 'terrorists',
    'intellectually inferior',
    'immigrants', 'demean country',
    'violence', 'culture',
    'feminist', 'man-hating',
    'useless', 'shoplifter'
  ],
  'triplets' : [
    ['nigger','is','stupid'],
    ['stupid','coappear','Muslims'],
    ['Muslims','are','terrorists'],
    ['nigger','implies','intellectually inferior'],
    ['immigrants','demean','country'],
    ['nigger','is responsible for','violence'],
    ['Muslims','demean','sincerity'],
    ['feminist','are','man-hating'],
    ['immigrants','are','useless'],
    ['immigrants','ruin','culture'],
    ['nigger','is','shoplifter']
  ]
}
```

Listing 1: Examples in Knowledge Graph.

Reviewer paper summaries: The paper introduces MetaTox, a method that leverages a knowledge graph to improve toxicity detection. The approach involves constructing a meta-toxic knowledge graph by using LLMs to extract toxic triplets. The KG is then queried through retrieval and ranking to provide supplementary information, which enhances the performance of LLMs in toxicity detection tasks. Experimental results show that MetaTox improves toxicity detection, especially in out-of-domain scenarios.

This paper proposes MetaTox, a method for augmenting large language models (LLMs) with domain-specific “meta-toxic” knowledge to improve toxicity detection. The approach first constructs a knowledge graph from existing labeled toxic data by

1. Rationale Reasoning: LLM prompts that explain why text is labeled toxic, surfacing implicit hateful references.
2. Triplet Extraction: Converting these rationales into (subject, predicate, object) triplets.
3. Entity Resolution: Clustering near-duplicate or semantically similar elements.

At inference time, given a potential toxic input, the system retrieves relevant toxic triplets from the graph to guide the LLM’s final verdict.
Summary of strengths: 1. The authors provide a compelling motivation for their research, clearly outlining the limitations of existing LLM-based toxicity detection methods, especially the shortcomings of RAG. They clearly identify both false negative scenarios (due to lack of domain-specific knowledge) and false positive scenarios (due to oversensitivity), providing solid grounding for the proposed approach.

2. MetaTox shows substantial improvements in cross-domain settings compared to other baseline methods, demonstrating its robustness and generalizability.

3. The work includes human evaluation to verify the quality and accuracy of the generated knowledge graph. This additional evaluation greatly enhances confidence in the reliability and usability of the KG.

4. The case studies are interesting and helpful in demonstrating how MetaTox corrects false positive and false negative errors by injecting knowledge into LLM predictions.

1. The authors motivate the difficulty LLMs have in both subtle hate detection and correct handling of benign mentions of sensitive topics. Injecting external “toxic knowledge” is a sensible idea.
2. The triplet-based pipeline offers a structured way to incorporate explicit hate relationships. “Self-checking” steps ensure the triplets maintain consistent formatting and encode hateful semantics.
Summary of weaknesses: 1. Constructing and querying the meta-toxic KG introduces considerable computational overhead. The authors do not sufficiently address scalability concerns or compare the runtime and complexity against simpler methods.

2. The definition and categorization of “relations” within the meta-toxic KG are not clearly explained. Given the abstract and nuanced nature of toxicity, readers would benefit from explicit explanations or examples of relations and their semantic significance.

3. Although human evaluation for knowledge graph triplets is mentioned, there is insufficient evaluation of whether the rationale reasoning and extracted triplets consistently reflect the intended toxicity meanings. This could lead to subtle errors propagating through the knowledge graph.

4. Toxicity can significantly vary depending on context. While the KG method helps provide explicit toxicity signals, it remains unclear how well the approach handles highly context-sensitive or implicit toxic content that may not align neatly with predefined or explicit relations.

1. The paper’s scope is fairly narrow (binary classification on English data), and it leans heavily on large LLMs for both creation of the knowledge graph and final judgments. Additionally, the improvement over simpler baselines in in-domain scenarios is limited.
2. Building the knowledge graph and rationale extraction heavily depends on a large, possibly proprietary model. Which showing little improvment yet larger cost than small model (eg. HateBert).
Comments, suggestions and typos: 1. It would be helpful if the authors could provide a clearer explanation regarding the specific domains and the differing focuses of the three datasets used. This would help readers better understand and interpret the cross-domain experimental setting.

2. The repository provided appears to have issues. None of the files, except for the README.md, are currently accessible.

A potential area for improvement would be to move beyond strictly binary labeling (“toxic” vs. “non-toxic”) and evaluate the system in a more fine-grained or multi-label setting (e.g., racism, sexism, homophobia, cyberbullying). Real-world online moderation often requires identifying specific types of harm (harassment, threats, hateful stereotyping, etc.).
Overall assessment scores: 3.5

2
Metareview score: 3
Metareview: The paper covers a topic of binary toxicity classification (monolingual) with LLMs. The work addresses the issue of not so great LLMs performance on the task with the domain meta-toxic knowledge graph that should make the predictions more precise, at least, to reduce false positives. Then, the authors tested two opensource models -- Qwen and LLaMa -- with such graph incorporation on the binary classification task comparing them to Transformer-encoder based approaches like BERT, HateBERT.
Summary of reasons to publish: * The work addresses an important problem of improving LLMs performance on toxicity classification task;

* The novel approach is proposed -- meta-toxic graphs -- which incorporation does not require any additional fine-tuning;

* The approach showed some improvements in the LLMs performances.