Implanting LLM’s Knowledge via Reading Comprehension Tree for Toxicity Detection (2024)

Abstract: Warning: This paper contains several toxic and offensive statements. Toxicity detection plays a crucial role in maintaining the peace of the society. Existing methods can be roughly categorized as small language model (SLM) based and large language model (LLM) based. However, due to the limitation of SLMs on general knowledge and the potential embedded bias in LLMs despite their large amount of knowledge, it is not a good idea to detect toxicity only with either SLM or LLM based method. In this work, we propose to implant LLM’s knowledge into SLM based methods such that we can stick to both types of models’ strengths. To this end, we develop a reading comprehension (RC) tree to transfer knowledge between two models. Specifically, we first construct the RC tree, from an extensive to intensive reading perspective, to capture the local and global information in the text. We then model samples encoded by SLM and knowledge extracted from LLM as two distributions using the constructed RT tree. We finally transfer knowledge via optimal transportation between two distributions. Extensive experiments prove the effectiveness of our method on real-world and machine-generated datasets. Our code and data are available at https://github.com/ khk-abc/toxic-detection.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 947–962 August 11-16, 2024 ©2024 Association for Computational Linguistics
Toxicity detection plays a crucial role in maintaining the peace of the society. Existing methods can be roughly categorized as small language model (SLM) based and large language model (LLM) based. However, due to the limitation of SLMs on general knowledge and the potential embedded bias in LLMs despite their large amount of knowledge, it is not a good idea to detect toxicity only with either SLM or LLM based method.
In this work, we propose to implant LLM’s knowledge into SLM based methods such that we can stick to both types of models’ strengths. To this end, we develop a reading comprehension (RC) tree to transfer knowledge between two models. Specifically, we first construct the RC tree, from an extensive to intensive reading perspective, to capture the local and global information in the text. We then model samples encoded by SLM and knowledge extracted from LLM as two distributions using the constructed RT tree. We finally transfer knowledge via optimal transportation between two distributions. Extensive experiments prove the effectiveness of our method on real-world and machine-generated datasets. Our code and data are available at https://github.com/ khk-abc/toxic-detection.

heading: 1 Introduction
text: With the prevailing of online communication, toxic content has been growing in recent years on the websites. In addition, the malicious use of large language models makes toxic language more common and difficult to detect. Toxic language may result in serious hazards such as the promotion of violent crimes and discrimination against marginalized groups. Due to its harmfulness to individuals
∗Corresponding author.
and society (Shakespeare, 2013), toxic language has become a serious concern.
The harm of toxic content can be prevented either by pre-detection before online release or by post-detection before spreading to a broader audience, and thus the detection of toxic content plays a crucial role in controlling toxicity. Existing methods for this purpose can be roughly categorized into small language model (SLM) based and large language model (LLM) based types. The finetuned SLM based methods (Antypas and CamachoCollados, 2023; He et al., 2023) are the mainstream. Due to the limited knowledge, the performance of these methods is not very satisfying. Recently, a LLM based method (Zhang et al., 2023a) is presented for this task. However, the potential of LLMs has not been fully exploited. More importantly, LLMs have their own bias which may lead to new safety issues (Liyanage and Ranaweera, 2023; Leong et al., 2023; Zhang et al., 2024).
As Fig. 1 (a) and (b) show, neither the RoBERTa (Liu et al., 2019) based method nor ChatGPT (OpenAI, 2022) can make a correct prediction about the given sample. To have a close look, we let two language models paraphrase ‘cockroaches’ and ’scorpions’ by masking these two words for
947
RoBERTa and directly asking ChatGPT. It can be seen that RoBERTa associates ‘cockroaches’ with ‘everywhere’ and ‘scorpions’ with ‘poisonous’ while ChatGPT actually knows they are both ‘insects’. We hence infer that the wrong prediction of RoBERTa based method is caused by the limited knowledge and that of ChatGPT might be due to the potential bias embedded in the model (Lee et al., 2023; Felkner et al., 2023).
In view of the shortcomings of SLM and LLM based methods, a natural idea is if we can combine both types of language models to leverage their strengthes. However, this is not a trivial task since toxicity detection requires the deep understanding of the text. A simple combination by treating LLM’s knowledge as the context of SLM or concatenating embeddings from two models does not improve SLM much, as shown in Fig. 1 (c). This drives us to solicit an effective way to comprehend the text. Fortunately, humans have well established the theory for enhancing reading skills (Saricoban, 2002; Toprak and Almacioğlu, 2009; Pressley and Afflerbach, 2012), and a pioneering work (Luo et al., 2019; Zhang et al., 2021) has solved the summarization problem by simulating the human reading process.
In this paper, we propose to implant LLM’s knowledge into SLM based method via a reading comprehension tree for toxicity detection. (1) We first construct the reading-comprehension (RC) tree to gain a deep understanding of the text, i.e., we first form local rough understanding via scanning words and phrases in the text and then incorporate the global comprehensive semantics of the entire text. (2) We then use the RC tree to model the sample encoded by SLM and knowledge from LLM as two distributions, i.e., we add task-specific features to the root of the tree to get multi-grained information distributions in a top-to-down manner. (3) We finally introduce the optimal transport method to fulfill knowledge implanting task, i.e., we apply optimal transport to LLM’s knowledge distribution and the sample’s distribution.
Our main contributions are as follows:
• We propose a novel framework which can effectively implant LLM’s knowledge into the SLM based method for toxicity detection.
• We construct a reading comprehension (RC) tree by simulating the human cognitive process for getting a deep understanding of the
text, thereby facilitating the multi-grained knowledge transfer from LLM to SLM.
• Extensive experimental results on three widely used toxic detection datasets across different languages and sources prove the superiority of our proposed framework.

heading: 2 Related Work
text: Toxic Language Detection. Toxicity means language contains profanity and is rude or disrespectful, leading to disharmony and conflicts in society (Feldman et al., 2015; Dixon et al., 2018). There are many types of toxicity, e.g., cyberbully (Slonje et al., 2013; Kowalski, 2018), hate speech (Del Vigna12 et al., 2017; Fortuna and Nunes, 2018), offensive language (Chen et al., 2012; Risch et al., 2020), and social bias (Garb, 1997; Liang et al., 2021). Several datasets have been collected (Zampieri et al., 2019; Sap et al., 2020; Haber et al., 2023; Mathew et al., 2021) across different types of toxicity. Various methods are proposed to tackle toxicity detection (Davidson et al., 2019; Ali et al., 2022; Gupta et al., 2023; Antypas and Camacho-Collados, 2023), e.g., multi-task learning based (Kapil and Ekbal, 2020) and prompt tuning based (He et al., 2023). However, most of existing methods utilize SLMs to capture the semantics of samples. Hence their performance is limited by SLMs’ knowledge.
Large Language Models (LLMs) have shown impressive performance on many tasks (Wei et al., 2022; Singhal et al., 2023; Madani et al., 2023; MacNeil et al., 2023). Recently, a few LLMs based methods are also proposed for toxicity detection (Zhang et al., 2023b) or fairness and factchecking (Zhang et al., 2023a). However, due to the factors like unsafe training data and policy decisions, it has been observed that LLMs contain toxicity like societal biases (Abid et al., 2021; Ferrara, 2023; Ray, 2023). Consequently, there are potential risks when deploying LLMs in various tasks.
Reading Comprehension is a cognitive and constructive process about text (Woolley and Woolley, 2011). It is necessary to get a comprehensive understanding of the text for further applications (Luo et al., 2019; Zhang et al., 2021). In the field of natural language processing, reading comprehension is usually modeled as question answering (Rajpurkar et al., 2018; Joshi et al., 2017) and applied to summarization tasks (Song et al., 2020; Jia et al., 2021; Gu et al., 2021). Existing methods mainly
focus on evaluating the importance of text spans to specific questions or summarizations (Ye et al., 2020; Xu et al., 2021; Bao et al., 2022; Raina and Gales, 2022; Luo et al., 2022). In contrast, our method aims at comprehending text spans themselves and captures the multi-granularity semantics into a reading-comprehension tree, which is suitable for diversified downstream tasks including the knowledge transfer in our work.

heading: 3 Methods
text: This section illustrates the details of our reading comprehension tree (RCT) based method.

heading: 3.1 Problem Definition
text: The task of toxicity detection is to recognize the multiple aspects of toxic text (Lu et al., 2023), including toxicity, toxic type, expression, and targeted groups. In this work, we aim to leverage both the fine-tuned representations from a small language model (SLM) and the knowledge from a large language model (LLM) to solve this problem.
Formally, given a set of training data T S for fine-tuning a SLM, and a text sample S = {S1, S2, · · · , SN} (N is the text length) in the test data T T , we first get a fine-tuned representation Ss. We also extract the knowledge Ks 1 from a LLM for the sample S. We finally obtain a toxicity detector F by implanting Ks into Ss via a reading comprehension tree Trc.
S[ytox,ytype,yexp,ytar] = F(S, Ss,Ks, Trc), (1)
where ytox, ytype, yexp, ytar denote the labels for toxicity, toxic type, expression, and targeted groups, respectively.

heading: 3.2 Reading Comprehension (RC) Tree Construction
text: This section presents the construction of the reading comprehension (RC) tree, which is inspired by the cognitive process and can be used to facilitate the knowledge transfer.
During human’s reading process, it is important to first extensively scan and then intensively read the text. The extensive scanning forms the local rough understanding of the text while the intensive reading obtains the global comprehensive semantics. To mimic this process, we first store the words into the leaf nodes. We then form the local rough
1It is easy to get Ss and Ks using existing methods and the detail will be presented later.
understanding by convoluting over several local nodes at the lower layer into a node at the upper layer, and we also get the global comprehensive semantics by summarizing the lower layer into the upper layer via a cross-attention mechanism. Recursively, we obtain the comprehensive semantics of the whole text in the tree.

heading: 3.2.1 Construction Process
text: We mimic the cognitive process in a local to global way during the construction process of the RC tree.
Forming the local rough understanding Given a sequence of words in a text t = {t1, t2, · · · , tN}, we first treat the representation of each word as a leaf node in the bottom layer, i.e., the D-th layer TDrc in the RC tree, where D is the depth of the tree. We then build an upper layer to get an aggregated local information by applying a convolutional operation to nodes at the lower layer as follows:
T i−1rc = CNN(T i rc), i = 1, 2, · · · , D, (2)
where T i−1rc ∈ RLi−1×d and T irc ∈ RLi×d denote the i-1th and ith layer of the tree, and Li−1 and Li denote the number of nodes at the i-1th and ith layer, respectively. d is the dimension of the representation of the node feature. Note that a small i denotes that the layer is close to the root, e.g., T 1rc is the root of the tree. In addition, the number of nodes in each layer is determined by the following formula, which is controlled by the window of CNN :
Li−1 = ⌊(Li − 1)/2⌋+ 1 (3)
Getting the global comprehensive semantics We get the global sentence-level semantics by applying cross-attention to two neighboring layers T i−1rc and T irc of the tree. Specifically, we compute the global semantic relevance scores si−1,i ∈ RLi−1×Li between all node pairs in two neighboring layers as:
si−1,i = Attention(T i−1rc , T i rc), (4)
Further, we incorporate the global features of nodes at the lower T irc layer into the features of nodes at the upper T i−1rc layer using the following formula:
T i−1rc = T i−1 rc + si−1,i · T irc, (5)
In this way, we recursively summarize the local and global information in the text, corresponding to the process of extensive scanning and intensive reading.

heading: 3.2.2 RC Tree for Sample and Knowledge
text: Fine-tuning sample’s representation by SLM We utilize a SLM as the encoder for fine-tuning samples’ representations. We then employ the sample’s representations Ss as the nodes in the leaf layer of the RC tree. In this study, we use BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the representative of SLMs.
Obtaining knowledge from LLM To acquire the knowledge towards each sample from LLMs, we first employ a LLM (gpt-3.5-turbo as the representative of LLMs here) as the content moderator.
M = R(LLM,OR), (6) whereRmeans that we employ a LLM as a content moderator role, and OR is the instruction for the role. Then we instruct the moderatorM to extract the knowledge K about the given sample S. K =M(S,OK), (7) where OK 2 denotes the instruction for extracting the knowledge K. The knowledge K is further encoded as Ks using the same encoder for Ss.
Building RC tree for sample and knowledge After getting representations for the sample Ss and knowledge Ks, we construct their reading comprehension trees, i.e., Trc,S and Trc,K , by applying the RC tree construction process to Ss and Ks as the input text span.

heading: 3.3 Obtaining Task-related Features and
text: Associated Information Distributions
Since the RC tree encapsulates the comprehensive semantics of the sample and knowledge, we employ
2The instructions OR and OK are given in Appendix A.2
them for obtaining their task-related features and associated information distributions. This is for further implanting the task-related semantics of knowledge into samples.
Starting from the root of the tree, we recursively obtain task-related multi-grained features in the constructed trees Trc,S and Trc,K in a top-to-down manner. Specifically, we first get the task representation by combining prompting, task related question like ‘Is it toxic?’, and the definition of a specific task (Details are in Appendix) as:
Qtask = E(S, [M ]task) + Eq + Ed, (8)
where E(S, [M ]task) denotes the task-specific prompt, and Eq and Ed denote the task related question and the definitions of a task, respectively.
Then we add the task-specific representation to the root of the tree (taking Trc,S as an illustration example):
R1S = T 1 rc,S +Qtask, (9)
Recursively, we obtain the task-related multigrained features of nodes at the lower layer based on the cross-attention operation.
rsi,i−1 = Attention(T irc,S , R i−1 S +Qtask), (10)
where rsi,i−1 ∈ RLi×Li−1 denotes the relevance scores between the features of nodes at the T irc,S layer and the combined feature of Ri−1S and Qtask. Further, we can obtain the task-related multigrained node features at the layer T irc,S as:
RiS = T i rc,S + rsi,i−1 · (Ri−1S +Qtask), (11)
where Ri−1S ∈ RLi−1×d and RiS ∈ RLi×d denote the task-related features at i-1th and ith layer. The relative importance of each node at the current layer can be computed by:
gj = σ(FFN(R i Sj , Qtask)) (12)
The features of nodes at the ith layer are as follows:
P iS =
Li∑
j=1
gj ·RiSj , (13)
where P iS is the i-th element of PS ∈ RD×d, denoting the task-related features of nodes at the ith layer. And d denotes the dimension of the encoded features. Finally, we obtain the associated task-related information distribution by applying a forward feed network.
P̂ iS = FFN(P i S), (14)
where P̂ iS denotes i-th element of the task-related information distribution P̂S ∈ RD for the sample S.
By now, we have task-related features P iS and the associated information distribution P̂ iS (i = 1, 2, · · · , DS) for the sample S. Similarly, we have P iK and P̂ i K (i = 1, 2, · · · , DK) for the knowledge K.

heading: 3.4 Implanting Knowledge via Optimal Transport
text: Optimal transport (OT) (Villani et al., 2009) denotes transporting a distribution A ∈ Rn to another distribution B ∈ Rm. The original target in OT problem is hard to optimize and Sinkhorn and Knopp (1967); Knight (2008) propose an equivalent optimization target and provide the solution as:
P∗ij = uivje −λCij , (15)
where ui and vj are the i and j elements of u ∈ Rn and v ∈ Rm intermediate variables for iterative optimization. P ∈ Rn×m+ is the transport matrix and C ∈ Rn×m is the cost matrix. λ is the adjust factor.
We employ OT to implant the knowledge to the sample. Formally, we first get the cost matrix C by computing the Euclid distance between the taskrelated features PK for the knowledge and features PS for the sample as follows:
Cij = ∥P iK − P jS∥2, (16)
where Cij means the transport cost from P iK into P jS , and a high relevance between P i K and P j S denotes a small cost Cij .
Further, we treat the knowledge distribution P̂K as the source and the sample distribution P̂S as the target. We then optimize the information transport matrix P in Eq. 15 using the Sinkhorn-Knopp algorithm (The detail is in Appendix). We further transport the task-related features entailed in the knowledge into the sample as:
P̃S = PS + α ·PTPK , (17) where α is an adjusted factor updated during the training process.
Finally, we classify the sample S with the help of implanted knowledge as:
[ytox, ytype, yexp, ytar] = FFN(P̃S) (18)

heading: 4 Experiments
text: This section presents experimental evaluation.

heading: 4.1 Experimental Setup
text: Datasets. We use three public toxic datasets, i.e., ToxicCN (Lu et al., 2023), Hatexplain (Mathew et al., 2021), and ToxiGEN.
Baselines. We choose three types of methods as baselines.
(1) Detection tools: PERSPECTIVEAPI, PAIBERT (PAI, 2023), TOXICBERT, and UNROBERTA (Hanu and Unitary team, 2020),
(2) LLM based methods: GPTZERO, GPTCOT, and GPTUNILC (Zhang et al., 2023a),
(3) SLM based methods: TKEBERT/ROBERTA (Lu et al., 2023), PROMPTBERT/ROBERTA (He et al., 2023).
Metrics and Implementation. We employ the weighted precision (P), recall (R), and F1-score (F1) as metrics (Lu et al., 2023). We train our model on a NVIDIA A800 80GB GPU and apply the AdamW optimizer to optimize parameters 3.

heading: 4.2 Results and Analysis

heading: 4.2.1 Main Results
text: We evaluate our RCT method by comparing it with the baselines. To ensure the fairness, we also utilize BERT and RoBERTa as the backbone and employ the same GPT (gpt-3.5-turbo) as LLM based methods to extract the knowledge. The main comparison results on ToxicCN, Hatexplain, and ToxiGEN are shown in Table 1, 2, and 3, respectively. We have the following important observations.
3The details of datasets, baselines, and implementations are given in Appendix A.4, A.5, and A.6, respectively.
Firstly, our RCT method consistently and significantly outperforms all baselines across three types of datasets. This clearly demonstrates the superiority of our method by combining SLM and LLM to leverage their strengths. Specifically, our RCT is the best on the Chinese ToxicCN and English Hatexplain datasets, showing its flexibility for different toxic language detection. Moreover, the best performance of our RCT on ToxiGEN indicates that it is good at recognizing both real-world and LLM-generated toxic contents.
Secondly, the SLM based methods are better than detection tools and LLM based methods, due to their specific task adaptability through finetuning. However, their performance is still worse than our RCT. Given the settings except the additional knowledge from the LLM, the reason is clear that the inferior performance of these SLM based methods must attribute to the lack of the knowledge. More evidences can be obtained from Table 5 by comparing the ‘w/o tree’ variant which is better
than TKE and Prompt. Thirdly, among three LLM based methods, GPTZero outperforms the detection tools, and the performance of GPTCoT is further improved by employing knowledge as rationales. At the same time, despite its careful demonstrations and problem modeling, GPTUnilc shows an unstable performance on three datasets. It is also worth noting that these LLM based methods achieves better performance on ToxiGEN than other two datasets. The reason might be that toxic texts in ToxiGEN are generated by inputting manually crafted prompts into LLMs, which are more familiar to LLMs and easier for these LLM based methods to recognize.
Cross dataset analysis. In order to investigate the generalization ability of fine-tuned models, we conduct the cross-dataset evaluation on Hatexplain and ToxiGEN. Note we cannot do this for ToxicCN because its language is different from those of Hatexplain and ToxiGEN, but the backbones and the fine-tuning procedure are language dependent.
In contrast, Hatexplain and ToxiGEN are from English, the model is applicable to both datasets for monolingual toxic detection. Hence we test models trained on ToxiGEN with samples in Hatexplain, and vice versa. The results are shown in Table 4.
We can find that our proposed RCT method performs significantly better than TKEBERT/ROBERTA and PROMPTBERT/ROBERTA, showing that our method has a better generalization ability on cross datasets. Moreover, the improvements on Hatexplain ⇒ ToxiGEN are much more obvious than those on ToxiGEN⇒Hatexplain. For example, the F1 score of our RCTRoBERTa reaches 77.7, while that of PromptRoBERTa is only 50.1, showing an 27.6 absolute increase. This suggests that the model trained on the real-world dataset can be well adapted to the LLM-generated dataset. The rea-
son might be that the real-world dataset contains diversified toxic contents.
On the other hand, the spurious associations between words and labels in the dataset potentially lead to unintended biases, e.g., the term ’gay’ frequently appears in toxic texts, causing the model to overlook the context of samples containing ’gay’ and treat them directly as toxic (Zhang et al., 2023c; Sen et al., 2022; Garg et al., 2023). If a model relies on spurious correlations, it may perform well within the same dataset, but cannot generalize to test samples with different spurious correlations. To verify this, we compute the top 100, 500, and 1000 spurious associated words in Hatexplain and ToxiGEN datasets (Zhang et al., 2023c). The overlapping rate of spurious associated words is 0.00%, 0.60%, and 1.37%, respectively. Our method achieves the best performance in crossdataset validation between Hatexplain and ToxiGEN, demonstrating its robustness to spurious associated words, which mitigates unintended biases.

heading: 4.2.2 Deep Analysis
text: Ablation analysis. To evaluate the impacts of different components in our framework, we conduct an ablation analysis and show results in Table 5.
• w/o tree: we remove the RC tree and sum sentence-level representation of samples and knowledge for transporting information. • w/o knowledge: we remove the knowledge from LLM and utilize the RC tree to capture the multi-grained features of samples. • w/o OT: we remove the optimal ransport (OT) and combine the multi-grained features of samples and knowledge via the simple addition.
From the ‘w/o tree’ ablation, we find that most results drop a lot, inferring the RC tree is crucial for capturing the local and global information in the text. The only exception is on the expression task with a slight increase. The reason might be due to the ignorance of syntactic structure information when we build the RC tree.
From the ‘w/o knowledge’ ablation, it is clear the performance of the model drops in all tasks without the help of knowledge from the LLM, which supplies more supporting information related to the toxic text and is helpful to recognize the toxic text.
From the ‘w/o OT’ ablation, we find the performance mainly decreases in the expression and target tasks. We analyze this is because optimal transport can capture the relevance of multi-grained features between samples and knowledge according to associated information distribution, which improves the transferring of multi-grained features.
In general, the performance of all degraded variants decreases on different tasks, proving the necessity and helpfulness of our designs in the proposed RCT framework.
Case analysis. We choose two toxic and one non-toxic samples from the Toxigen dataset for case study and show results in Table 6. In all three cases, we let the SLM and LLM to paraphrase the specific terms in the text by masking the corresponding terms for SLM and directly ask LLM about the terms.
In Case 1, we ask SLM to paraphrase ‘abortion’ using ‘What’s the abortion? It is <mask>.’. Due to limited knowledge, SLM cannot fully comprehend the text and associates ‘abortion’ with ‘rape’, thus finally makes wrong prediction. In contrast, LLM
owns the knowledge that women have the right of ‘abortion’ 4. Consequently, both LLM and our method, which borrows the knowledge from the LLM, can predict the text as ‘Non-toxic’.
In Case 2, the text states that ‘people with mental illness cannot work in the public sector’. Owing to similar samples in the training data and the finetuning procedure, SLM associates ‘mental illness’ with ‘schizophrenia’. As a result, both SLM and our method, which uses the SLM based method as the backbone, can gain the positive cognition about mental illness and successfully detect the toxicity correctly. In contrast, due to the bias (Gadiraju et al., 2023; Yeh et al., 2023) against the disability, LLM assigns wrong labels for the text.
In Case 3, the text implies a toxic opinion that ‘if the impairment is noticeable, it could be called disability’. LLM ignores the toxicity due to the bias against disability. Moreover, due to the implicitness of toxicity, the SLM based method focuses on the surface positive expression ‘shouldn’t call it disability’ but fails to capture the hidden toxic opinion and thus also ignores the toxicity. However, our method gives the true label. We believe this is because our method has an ability to grasp the subtle meaning of the text via the human-like cognitive process and the general knowledge from the LLM. Indeed, a separate statistic shows that our model successfully correct about 22% (105/485) percent of samples for which both SLM and LLM based methods make wrong predictions.
In summary, our method implants the knowledge from the LLM into the SLM encoded sam-
4The detailed knowledge from the LLM are provided in Table 13 in the Appendix.
ples. Meanwhile, with the advantage of acquiring task-specific information from training data via fine-tuning, as well as the deep understanding of the text via the constructed RC tree, our method further improves SLM based methods and achieves a more superior performance in toxicity detection.

heading: 5 Conclusion
text: In view of the limitation of knowledge in SLM based methods and the potential bias and harmfulness in LLM based methods, we propose a novel framework to implant knowledge from LLMs into SLM based methods to leverage their strengthes for toxicity detection. Specifically, we design a reading comprehension (RC) tree to mimic the cognitive process, which gets a deep understanding of texts in the sample and knowledge and also facilitates the transportation of the knowledge. Experiments on three widely used datasets prove that our method achieves significantly better performance than existing detection tools and methods.

heading: Limitations
text: We propose a method to detect the toxicity of a content, where we utilize an optimal transport technique to implant knowledge from LLMs into SLM based methods. However, the Sinkhorn-Knopp algorithm to solve the optimal transport is an iterative method, which may take more time to achieve convergence. Meanwhile, we propose a reading comprehension (RC) tree structure to get a deep understanding of the texts for recognizing the toxic content. It is worth exploring our proposed RC tree on other tasks, e.g., text classification. Finally, we separately conduct toxic detection in a single language, and the multi-linguistic toxic detection has not been studied in this paper, which can be exploited in the future.

heading: Ethics Statement
text: We strictly adhere to usage agreements, and the statement in the paper is only for research purposes. We suggest to ensure reliability via re-checks after auto-detection, considering the importance of controlling toxicity.

heading: Acknowledgments
text: This work was supported by a grant from the National Natural Science Foundation of China (NSFC) project (No. 62276193).

heading: A Appendix

heading: A.1 Sinkhorn-Knopp Algorithm for Optimal Transport
text: This section presents the definition of optimal transport and the solution, i.e., Sinkhorn-Knopp algorithm, for optimal transport.
Definition. The optimal transport (OT, Villani et al., 2009; Flamary et al., 2021; Wei et al., 2023) problem means transporting a distribution A ∈ Rn to another distribution B ∈ Rm. Specifically, transporting the i-th element Ai of A to the j-th element Bj of B with transmission amount Pij and cost Cij, where Pij and Cij are the elements of the i-th row and j-th column of transport matrix P ∈ Rn×m+ and cost matrix C ∈ Rn×m, respectively. The OT problem minimizes the weighted cost as the optimized target to obtain the optimal transmission matrix P∗, which is denoted as:
LWD = min P∈U(A,B)
nm∑
ij
PijCij, (19)
where LWD is called Wasserstein distance and U(A,B) = {P|P1m = A,PT1n = B} is the joint distribution of distributions A and B.
Solution. It is hard to optimize LWD. Sinkhorn and Knopp (1967); Knight (2008) propose an equivalent optimization target as:

heading: LλWD = min
text: P∈U(A,B)
nm∑
ij
PijCij − 1
λ
nm∑
ij
Pij logPij,
(20) where λ is the adjustment factor. The optimal transport matrix is then computed iteratively as follows:
P∗ij = uivje −λCij , (21)
where ui and vj are the i-th and j-th elements of u ∈ Rn and v ∈ Rm intermediate variables for iterative optimization.
A.2 Instruction for Knowledge Extraction
As depicted in Tab. 8, we first request GPT-3.5turbo to serve as an online content moderator with ‘Role Play’ instruction, and then we instruct the ‘moderator’ to analyze the knowledge according to the given claim via ‘Analyze Knowledge’ instruction.

heading: A.3 Tasks-specific Prompts, Questions and
text: Definitions
Firstly, we introduce the tasks of toxicity, toxic type, expression, and target:
• Toxicity task aims to determine whether the text is toxic.
• Toxic type task aims to judge the type of toxic text, including ‘hateful’ and ‘offensive’ in our employed datasets.
• Expression task aims to classify the expressive methods when the text embodies toxicity, including ‘explicit’ and ‘implicit’ and ‘paraphrase’ in our employed datasets.
• Target task aims to identify the targeted groups of toxic text, including ‘race’, ‘gender’, ‘religion’ and so on.
In order to get a task-specific representation, we consider three aspects: task-specific semantics in samples, task questions, and definitions. This subsection will show the task-specific prompts, questions, and definitions.
Firstly, we obtain the task-related semantics in samples as follows:
prompt = [S,Mtox,Mtype,Mexp,Mtar], (22)
where S is the placeholder for samples. Mtox, Mtype, Mexp and Mtar mean soft prompt tokens for toxicity, toxic type, expression and target tasks, respectively. Then we obtain the specific taskrelated features in the samples via selecting the representation of associated soft prompt tokens.
Next, the questions and definitions are designed based on the annotation principles and definitions of the datasets, as show in Tab. 9 and 10.

heading: A.4 Datasets
text: We choose three widely used toxic datasets to verify the effectiveness of our proposed method.
ToxicCN (Lu et al., 2023) is a Chinese dataset for fine-grained toxic detection, which includes
hierarchical taxonomy annotations (i.e., toxicity, toxic type, expression, and targets) and collects online contents from social medias. This dataset allows us to research joint toxic Chinese recognition.
Hatexplain (Mathew et al., 2021) is an English dataset for toxic detection, which is collected from online contents. This dataset includes binary toxicity labels, ternary toxicity labels, toxic spans in text, and targets of toxic text. Following the setting of ToxicCN, we reorganize the ternary toxicity labels in Hatexplain into labels of ‘toxic type’ task, i.e., ‘hate speech’, and ‘offensive’. And Hatexplain is finally reorganized to include the toxicity, toxic type, and target tasks.
ToxiGEN (Hartvigsen et al., 2022) is also an
English toxic identify dataset, and the text is generated through manually designed prompts inputted into LLM. And the dataset comprises the binary toxicity labels and targets of the generated text.
In ToxicCN, Lu et al., 2023 annotates the multiple aspects of toxic text for fine-grained detection of toxic text, including toxicity, toxic type, expression and target. Following the settings of ToxicCN, we reorganized the Hatexplain to include toxicity, toxic type and target tasks, and reorganized the ToxiGEN to include toxicity and target tasks. The above three datasets across different languages and sources help us extensively analyze toxicity recognition across languages and sources. In addition, we strictly abide by the licenses for using the datasets (e.g., CC BY-NC-ND 4.0) and use them only for scientific research. And personal information has been anonymized in these datasets.

heading: A.5 Baselines
text: We compare our method with three types of baselines:
(1) Detection tools: Tools employed for toxicity binary detection.
• PERSPECTIVEAPI is a detection tool for toxic text from multiple languages.
• PAIBERT (PAI, 2023) is a pretrained detection tool that only can detect toxicity in Chinese text.
• TOXICBERT and UNROBERTA (Hanu and Unitary team, 2020) are pretrained detection tools only for English toxicity detection.
(2) LLM based methods: Methods based on GPT3.5-turbo for toxicity recognition.
• GPTZERO utilizes GPT-3.5-turbo (OpenAI, 2022) to answer whether the text is toxic by a yes/no question.
• GPTCOT utilizes the knowledge from GPT3.5-turbo as the rationale and requests GPT3.5-turbo to answer yes/no.
• GPTUNILC (Zhang et al., 2023a) models detection as a fairness checking problem and provides some demonstrations for LLM to recognize the toxicity.
Algorithm 1 Algorithm of our method Require: Sample S Ensure: Y = [ytox, ytype, yexp, ytar]
1: K ← ChatGPT(S) 2: Trc,S ← ConstrctTree(S) 3: Trc,K ← ConstrctTree(K) 4: for Qtask in [Qtox, Qtype, Qexp, Qtar] do 5: RS ← QueryTree(Qtask, Trc,S) 6: RK ← QueryTree(Qtask, Trc,K) 7: RS ← OptimalTransport(RK → RS) 8: ytask ← classifier(RS) 9: end for
(3) SLM methods: Methods based on fine-tuning pretrained models for detection toxic text.
• PROMPTBERT and PROMPTROBERTA (He et al., 2023) introduce prompt tuning for toxicity binary detection based on pretrained models.
• TKEBERT and TKEROBERTA (Lu et al., 2023) recognize multi aspects of toxic text (i.e., toxicity, toxic type, expression and targets), but fine-tune pretrained models only via ‘CLS’ token without extra designs.
In addition, we strictly abide by the licenses (e.g., apache-2.0 and CC-BY-SA-4.0 license) for using the tools and use them only for scientific research.

heading: A.6 Training Settings
text: We show the training hyperparameter settings in Tab. 7. The training hyperparameter settings are mainly based on (Lu et al., 2023). Specifically, the ‘kernel size’, ‘stride’ and ‘padding’ mean the size of the convolutional blocks, scan step and padding length, respectively. In addition, we employ different pretrained models as backbones for adapting different languages (i.e., English and Chinese). For example, we use bert-base-chinese ( 103M params) and chinese-roberta-wwm-ext (102M params) as Chinese text encoder and apply bert-base-uncased (110M params) and roberta-base (125M params) as English text encoder. We utilize the offical open source codes for reproducing the results of Prompt and TKE. We utilize the LLM with model version GPT-3.5-turbo and employ PerspectiveAPI by calling api interface. Following the setting of TKE (Lu et al., 2023), we run all models with 5 different seeds and report the average performance. And our method takes about 3.5 hours in each run.

heading: A.7 Time Cost Comparison
text: As Tab. 12 shows, we compare the time cost of our method with baselines when they classify a text. Our method takes a little more time for extra knowledge processing and the iterative solution to the optimal transport problem.

heading: A.8 Pseudocode illustration
text: In order to better follow our idea, we illustrate the brief pseudocode to help reading as Algorithm 1 shows. Please reference our code for more details.

heading: Future Work
text: Finally, we separately conduct toxic detection in a single language, and the multi-linguistic toxic detection has not been studied in this paper, which can be exploited in the future