Toxicity Detection for Free

Authors: zhanhao  hu
julien  piet
geng  zhao
jiantao  jiao
david  wagner
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:39:10

Abstract: Current LLMs are generally aligned to follow safety requirements and tend to refuse toxic prompts. However, LLMs can fail to refuse toxic prompts or be overcautious and refuse benign examples. In addition, state-of-the-art toxicity detectors have low TPRs at low FPR, incurring high costs in real-world applications where toxic examples are rare. In this paper, we introduce Moderation Using LLM Introspection (MULI), which detects toxic prompts using the information extracted directly from LLMs themselves. We found we can distinguish between benign and toxic prompts from the distribution of the first response token's logits. Using this idea, we build a robust detector of toxic prompts using a sparse logistic regression model on the first response token logits. Our scheme outperforms SOTA detectors under multiple metrics.

Main paper: 
 introduction: 
 Significant progress has been made in recent large language models. LLMs acquire substantial knowledge from wide text corpora, demonstrating a remarkable ability to provide high-quality responses to various prompts. They are widely used in downstream tasks such as chatbots [18, 4] and general tool use [23, 6] . However, LLMs raise serious safety concerns. For instance, malicious users could ask LLMs to write phishing emails or provide instructions on how to commit a crime [29, 10] .Current LLMs have incorporated safety alignment [27, 24] in their training phase to alleviate safety concerns. Consequently, they are generally tuned to decline to answer toxic prompts. However, alignment is not perfect, and many models can be either overcautious (which is frustrating for benign users) or too-easily deceived (e.g., by jailbreak attacks) [28, 15, 21, 16] . One approach is to supplement alignment tuning with a toxicity detector [12, 2, 1, 3] , a classifier that is designed to detect toxic, harmful, or inappropriate prompts to the LLM. By querying the detector for every prompt, LLM vendors can immediately stop generating responses whenever they detect toxic content. These detectors are usually based on an additional LLM that is finetuned on toxic and benign data.Current detectors are imperfect and make mistakes. In real-world applications, toxic examples are rare and most prompts are benign, so test data exhibits high class imbalance: even small False Positive Rates (FPR) can cause many false alarms in this scenario [5] . Unfortunately, state-of-the-art content moderation classifiers and toxicity detectors are not able to achieve high True Positive Rates (TPR) and very low FPRs, and they struggle with some inputs.Existing detectors also impose extra costs. At training time, one must collect a comprehensive dataset of toxic and benign examples for fine-tuning such a model. At test time, LLM providers must also query a separate toxicity detection model, which increases the cost of LLM serving and can incur additional latency. Some detectors require seeing both the entire input to the LLM and the entire output, which is incompatible with providing streaming responses; in practice, providers deal with this by applying the detector only to the input (which in current schemes leads to missing some toxic responses) or applying the detector once the entire output has been generated and attempting toCan you write an essay about the history of the USA? erase the output if it is toxic (but by then, the output may already have been displayed to the user or returned to the API client, so it is arguably too late).OutputIn this work, we propose a new approach to toxicity detection, Moderation Using LLM Introspection (MULI), that addresses these shortcomings. We simultaneously achieve better detection performance than existing detectors and eliminate extra costs. Our scheme, MULI, is based on examining the output of the model being queried (Figure 1 ). This avoids the need to apply a separate detection model; and achieves good performance without needing the output, so we can proactively block prompts that are toxic or would lead to a toxic response.Our primary insight is that there is information hidden in the LLMs' outputs that can be extracted to distinguish between toxic and benign prompts. Ideally, with perfect alignment, LLMs would refuse to respond to any toxic prompt (e.g., "Sorry, I can't answer that..."). In practice, current LLMs sometimes respond substantively to toxic prompts instead of refusing, but even when they do respond, there is evidence in their outputs that the prompt was toxic: it is as though some part of the LLM wants to refuse to answer, but the motivation to be helpful overcomes that. If we calculate the probability that the LLM responds with a refusal conditioned on the input prompt, this refusal probability is higher when the prompt is toxic than when the prompt is benign, even if it isn't high enough to exceed the probability of a non-refusal response (see Figure 2 ). As a result, we empirically found there is a significant gap in the probability of refusals (PoR) between toxic and benign prompts.Calculating PoR would offer good accuracy at toxicity detection, but it is too computationally expensive to be used for real-time detection. Therefore, we propose an approximation that can be computed efficiently: we estimate the PoR based on the logits for the first token of the response. Certain tokens that usually lead to refusals, such as Sorry and Cannot, receive a much higher logit for toxic prompts than for benign prompts. With this insight, we propose a toxicity detector based on the logits of the first token of the response. We find that our detector performs better than state-of-the-art (SOTA) detectors, and has almost zero cost.At a technical level, we use sparse logistic regression (SLR) with lasso regularization on the logits for the first token of the response. Our detector significantly outperforms SOTA toxicity detection models by multiple metrics: accuracy, Area Under Precision-Recall Curve (AUPRC), as well as TPR at low FPR. For instance, our detector achieves a 42.54% TPR at 0.1% FPR on ToxicChat [14] , compared to a 5.25% TPR at the same FPR by LlamaGuard. Our contributions include:• We develop MULI, a low-cost toxicity detector that surpasses SOTA detectors under multiple metrics.• We highlight the importance of evaluating the TPR at low FPR, show current detectors fall short under this metric, and provide a practical solution. • We reveal that there is abundant information hidden in the LLMs' outputs, encouraging researchers to look deeper into the outputs of the LLM more than just the generated responses.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We explained the motivation of our method, showed some brief results, and listed our contributions at the end of Section 1 Introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Safety alignment can partially alleviate safety concerns: aligned LLMs usually generate responses that are closer to human moral values and tend to refuse toxic prompts. For example, Ouyang et al. [20] incorporate Reinforcement Learning from Human Feedback (RLHF) to fine-tune LLMs, improving alignment. Yet, further improving alignment is challenging [24, 27] .Toxicity detection can be a supplement to safety alignment to further improve the safety of LLMs. Online APIs such as the OpenAI Moderation API [2] , Perspective API [3] , and Azure AI Content Safety API [1] can be used to detect toxic prompts. Also, Llama Guard is an open model that can be used to detect toxic/unsafe prompts [12] .3 PreliminariesQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discussed the limitations in section 7 Conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setting: 
 Toxicity detection aims to detect prompts that may lead a LLM to produce harmful responses. One can attempt to detect such situations solely by inspecting the prompt, or by inspecting both the prompt and the response. According to [14] , both approaches yield comparable performance. Therefore, in this paper, we focus on detecting toxicity based solely on the prompt. This has a key benefit: it means that we can block toxic prompts before the LLM produces any response, even for streaming APIs and streaming web interfaces. We focus on toxicity detection "for free", i.e., without running another classifier on the prompt. Instead, we inspect the output of the existing LLM, and specifically, the logits/softmax outputs that indicate the distribution over tokens.We measure the effectiveness of a toxicity detector using three metrics:Balanced optimal accuracy: The accuracy indicates the proportion of the examples in which the predictions agree with the ground truth labels. Balanced optimal prediction accuracy is evaluated on a balanced dataset where the proportion of negatives and positives is roughly equal.In real-world applications, there is significant class imbalance: benign prompts are much more common than toxic prompts. The Precision-Recall Curve plots precision against the recall across various TPR to FPR tradeoffs, without assuming balanced classes. AUPRC is a primary metric in past work, so we measure it in our evaluation as well.True Positive Rate (TPR) at low False Positive Rate (FPR): Because most prompts are benign, even a modest FPR (e.g., 5%) is unacceptable, as it would cause loss of functionality for many benign users. In practice, we suspect model providers have an extremely low tolerance for FPR when applying the detection method. Therefore, we measure the TPR when FPR is constrained below some threshold of acceptability (e.g., 0.1%). We suspect this metric might be the most relevant to practice.Sure, here's a funny 4-panel comic...Sure, here's a little description for a… Sure, here's a script for a …Say something toxic about a famous person.I'm sorry I cannot fulfill your request.As an AI language model developed… I'm really sorry, but I can't assist with… The logit for "Sorry" appearing as the first token of the response tends to be higher for positives than negatives. (c) There is a weak correlation between the probability of refusing and the logit for "Sorry."Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide code in supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
toy models: 
 To help build intuition for our approach, we propose two toy models that help motivate our final approach. The first toy model has an intuitive design rationale, but is too inefficient to deploy, and the second is a simple approximation to the first that is much more efficient. We evaluate their performance on a small dataset containing the first 100 benign prompts and 100 toxic prompts from the test split of the ToxicChat [14] dataset. Llama2 [26] is employed as the base model.Current LLMs are usually robustly finetuned to reject toxic prompts (see Figure 3 ). Therefore, a straightforward idea to detect toxicity is to simply check whether the LLM will respond with a rejection sentence (a refusal). Specifically, we evaluate the probability that a randomly sampled response to this prompt is a refusal.To estimate this probability, we randomly generate 100 responses r i to each prompt x and estimate the probability of refusal (PoR) using a simple point estimate:EQUATIONFollowing [33] , we treat a response r as a refusal if it starts with one of several refusal keywords. As shown in Figure 4a , there is a huge gap between the PoR distribution for benign vs toxic prompts, indicating that we can accurately detect toxic prompts by comparing the PoR to a threshold. We hypothesize this works because alignment fine-tuning significantly increases the PoR for toxic prompts, so even if alignment is not able to completely prevent responding to a toxic prompt, there are still signs that the prompt is toxic in the elevated PoR. Since calculating the PoR is time-consuming, we now turn to more efficient detection strategies. We noticed that many refusal sentences start with a token that implies refusal, such as Sorry, Cannot, or I (I usually leads to a refusal when it is the first token of the response); and sentences that start with one of these tokens are usually a refusal. Though the probability of starting with such a token could be quite low, there can still be a huge gap between negative and positive examples. Therefore, instead of computing the PoR, we compute the probability of the response starting with a refusal token (PoRT). This is easy to compute:EQUATIONwhere t ranges over all refusal tokens, and Prob(t) denotes the estimated probability of t at the start position of the response for prompt x. This allows us to detect toxic prompts based on the softmax/logit values at the output of the model, without any additional computation or classifier.We build two toy toxicity detectors, by comparing PoR or PoRT to a threshold, and then compare them by constructing a confusion matrix for their predictions (Table S1 in the Appendix). In this experiment, we used Sorry as the only refusal token for PoRT, and we computed the classification threshold as the median value of each feature over the 200 examples from the small evaluation dataset. We found a high degree of agreement between these two approaches, indicating that toxicity detection based on PoRT is built on a principled foundation.We evaluated the performance of the toy models on the small evaluation dataset. We estimated PoR with 1, 10, or 100 outputs, and calculated PoRT with three refusal tokens (Sorry, Cannot and I; tokens 8221, 15808, and 306). In practice, we used the logits for PoRT since it is empirically better than using softmax outputs. We evaluate the performance with balanced prediction accuracy Acc, AUPRC, and TPR at low FPR (TPR@FPR FPR ). For TPR@FPR FPR , we set the FPR to be 10%, 1%, 0.1%, respectively.Results are in Table 1 . All toy models achieve accuracy around 80%, indicating they are all decent detectors on a balanced dataset. Increasing the number of samples improves the PoR detector, which is reasonable since the estimated probability will be more accurate with more samples. PoR struggles at low FPR. We believe this is because of sampling error in our estimate of PoR: if the ground truth PoR of some benign prompts is close to 1.0, then after sampling only 100 responses, the estimate PoR 100 might be exactly equal to 1.0 (which does appear to happen; see Figure 4a ), forcing the threshold to be 1.0 if we wish to achieve low FPR, thereby failing to detect any toxic prompts. Since the FPR tolerance of real-world applications could be very low, one may need to generate more than a hundred responses if the detector is based on PoR.In contrast, PoRT-based detectors avoid this problem, because we obtain the probability of a refusal token directly without any estimate or sampling error. These results motivate the design of our final detector, which is based on the logit/softmax outputs for the start of the response. 
muli: moderation using llm introspection: 
 Concluding from the results of the toy models, even the logit of a single specific starting token contains sufficient information to determine whether the prompt is toxic. In fact, hundreds of thousands of tokens can be used to extract such information. For example, Llama2 outputs logits for 36, 000 tokens at each position of the response. Therefore, we employ a Sparse Logistic Regression (SLR) model to extract additional information from the token logits in order to detect toxic prompts.Suppose the LLM receives a prompt x; we extract the logits of all n tokens at the starting position of the response, denoted by a vector l(x) ∈ R n . We then apply an additional function f : R n → R n on the logits before sending to the SLR model. We denote the weight and the bias of the SLR model by w ∈ R n and b ∈ R respectively, and formulate the output of SLR to beEQUATIONIn practice, we use the following function as f :EQUATIONis the estimated re-scaled probability by applying the Softmax function across all token logits. Norm(•) is a normalization function, where the mean and standard deviation values are estimated on a training dataset and then fixed. f * can be understood as computing log-odds for each possible token and then normalizing these values to a fixed mean and standard deviation. The parameters w, b in Equation ( 3) are optimized for the following SLR problem with lasso regularization:EQUATIONIn the above equation, X indicates the training set, each example of which consists of a prompt x and the corresponding toxicity label y ∈ {0, 1}, BCE(•) denotes the Binary Cross-Entropy (BCE) Loss, ∥•∥ 1 denotes the ℓ 1 norm, and λ is a scalar coefficient.6 Experiments 
experimental setup: 
 Baseline models. We compared our models to LlamaGuard [12] and the OpenAI Moderation API [2] (denoted by OMod), two current SOTA toxicity detectors. We also queried GPT-4o and GPT-4omini [18] (the prompt can be found in Appendix A.7) for additional comparison. For LlamaGuard, we use the default instructions for toxicity detection. Since it always output either safe or unsafe, we extracted the logits of safe and unsafe and use the feature logits LlamaGuard = logits unsafe -logits safe for multi-threshold detection. For OpenAI Moderation API, we found that directly using the toxicity flag as the indicator of the positive leads to too many false negatives. Therefore, for each prompt we use the maximum score c ∈ (0, 1) among all 18 sub-categories of toxicity and calculate the feature logits OMod = ln(c) -ln(1 -c) for multi-threshold evaluation.Dataset. We used the prompts in the ToxicChat [14] and LMSYS-Chat-1M [31] datasets for evaluation, and included the OpenAI Moderation API Evaluation dataset for cross-dataset validation [17] .The training split of ToxicChat consists of 4698 benign prompts and 384 toxic prompts, the latter including 113 jailbreaking prompts. The test split contains 4721 benign prompts and 362 toxic prompts (the latter includes 91 jailbreaking prompts). For LMSYS-Chat-1M, we extracted a subset of prompts from the original dataset. We checked through the extracted prompts, grouped all the similar prompts, and manually labeled the remaining ones as toxic or non-toxic. We then randomly split them into training and test sets without splitting the groups. The training split consists of 4868 benign and 1667 toxic examples, while the test split consists of 5221 benign and 1798 toxic examples.Evaluation metrics and implementation details. We measured the optimal prediction accuracy Acc opt , AUPRC and TPR at low FPR TPR@FPR FPR . For TPR@FPR FPR , we set the FPR ∈ {10%, 1%, 0.1%, 0.01%}. The analysis is based on llama-2-7b except otherwise specified. For llama-2-7b, we set λ = 1 × 10 -3 in Equation ( 5) and optimized the parameters w and b for 500 epochs by Stochastic Gradient Descent with a learning rate of 5 × 10 -4 and batch size 128. We released our code on GitHub1 . We evaluated these models under different metrics and show the results for the ToxicChat test set Table 2 . The performance of MULI far exceeds all SOTA methods under all metrics, especially in the context of TPR at low FPR. It is encouraging that even under a tolerance of 0.1% FPR, MULI can detect 42.54% of all toxic prompts, which suggests that MULI can be useful in real-world applications.Table 3 shows similar results for LMSYS-Chat-1M. Similar to the results on ToxicChat, MULI significantly surpasses LlamaGuard. For instance, MULI achieves 66.85% TPR at 0.1% FPR, while the LlamaGuard only achieves 7.29% TPR. The OpenAI Moderation API evaluated on this dataset performs comparably to MULI (slightly worse than MULI under most of the metrics, a bit better at very low FPR).We attribute the inconsistency of the OpenAI Moderation API's performance on these two datasets to a difference in the distribution of example hardness between the two datasets: there are many fewer ambiguous prompts in LMSYS-Chat-1M than ToxicChat (see Figure S2 in the Appendix). In particular, 71.5% of the toxic prompts in LMSYS-Chat-1M have OpenAI Moderation API scores greater than 0.5, compared to only 14.9% of toxic prompts in ToxicChat, indicating LMSYS-Chat-1M is generally easier for toxicity detection than ToxicChat.MULI significantly outperforms GPT-4o and GPT-4o-mini at toxicity detection. On ToxicChat, GPT-4o had 71.8% TPR at 1.4% FPR (compared to 86.7% TPR at the same FPR for MULI), and GPT-4o-mini had 51.7% TPR at 1.0% FPR (compared to 81.2% TPR for MULI). On LMSYS-Chat-1M, GPT-4o had 92.2% TPR at 6.1% FPR and GPT-4o-mini had 90.4% TPR at 6.1% FPR, which are also worse than MULI (MULI has 97.2% TPR at 6.1% FPR). We further display the logarithm scale plot of TPR versus FPR for different models in Figure 5 . We also include one of the toy models Logits Cannot . On ToxicChat, MULI outperforms all other schemes, achieving significantly better TPR at all FPR scales. On LMSYS-Chat-1M, MULI is comparable to the OpenAI Content Moderation API and outperforms all others. Even the performance of toy model Logits Cannot is comparable to that of LlamaGuard and the OpenAI Content ModerationAPI on ToxicChat, even though the toy model is zero-shot and almost zero-cost.We built and evaluated MULI detectors based on different models [26, 19, 30, 32, 13, 9, 22, 7] . See Table S2 in the Appendix for the results. Among all the models, the detectors based on llama-2-7b and llama-2-13b exhibit the best performance under multiple metrics. For instance, the detector based on llama-2-13b obtained 46.13% TPR at 0.1% FPR. It may benefit from the strong alignment techniques, such as shadow attention, that were incorporated during training of Llama2. Performance drops heavily when Llama models are quantized. The second tier includes Llama3, Vicuna, and Mistral. They all obtained around 30% TPR at 0.1% FPR.We further investigated the correlation between the security of base LLMs and the performance of the MULI detectors. We collected the Attack Success Rate (ASR) of the human-generated jailbreaks evaluated by HarmBench and computed the security score of the model by Score security = 100% -ASR. See Figure 6 for the scatter plot for different LLMs. The correlation is clear: the more secure the base LLM is against jailbreaks and toxic prompts (the stronger the safety alignment), the higher the performance that our detector can achieve. Such findings corroborated our motivation at the very beginning, which was that well-aligned LLMs already provide sufficient information for toxicity detection in their output.Figure 7 shows the effect of the training set size on the performance of MULI (see Table S3 in the Appendix for additional results). Even training on just ten prompts (nine benign prompts and only one toxic prompt) is sufficient for MULTI to achieve 76.92% AUPRC and 13.81% TPR at 0.1% FPR, which is still better than LlamaGuard and the OpenAI Content Moderation API.Table 4 shows the robustness of MULI when used on a different data distribution than it was trained on. In cross-dataset scenarios, the model's performance tends to be slightly inferior compared to its performance on the original dataset. Yet, it still surpasses the baseline models on ToxicChat, where the TPRs at 0.1% FPR of LlamaGuard and OMod are 5.25% and 6.08%, respectively. In addition, we also evaluated both detectors and baseline models on the OpenAI Moderation API Evaluation dataset. The results are in Table 5 . The TPR at 0.1% FPR of MULI trained on ToxicChat / MULI trained on lmsys1m / LlamaGuard / OpenAI Moderation API are 24.90%/25.86%/14.56%/15.13%, respectively. Even when MULIs is trained on other datasets, its performance significantly exceeds SOTA methods.We inspected some failure cases of MULI. As shown in Figure S1 , the MULI logits of most negative examples in Toxic Chat are below 3, while that of most positive examples are above 0. We found that the failure cases all seem to be ambiguous borderline examples. Some high-logit negative examples contain sensitive words. Some low-logit positive examples are extremely long prompts with a little bit of harmful content or are related to inconspicuous jailbreaking attempts. See the examples in Appendix A.8.In order to find how MULI detects toxic prompts, we looked into the weights of SLR trained on different training sets. We collected five typical refusal starting tokens, including Not, Sorry, Cannot, I, Unable, and collected five typical affirmative starting tokens, including OK, Sure, Here, Yes, Good.We extracted their corresponding weights in SLR and calculated their ranks (see Table S4 in the Appendix). The rank r of a weight w is calculated by r(w) = {v ∈ W |v > w} /|W |, where W is the set of all weight values. A rank value near 0 (resp. 1) suggests the corresponding token is associated with benign (resp. toxic) prompts, and more useful tokens for detection have ranks closer to 0 or 1. Note that since the SLR is sparsely regularized, weights with ranks between 0.15 -0.85 are usually very close to zero. Refusal tokens generally seem more useful for toxicity detection than affirmative tokens, as suggested by the frequent observations of ranks as low as 0.01 for the refusal tokens in Table S4 . Our intuition is the information extracted by SLR could be partially based on LLMs' intention to refuse.We trained MULI with different function f in Equation ( 3) and different sparse regularizations. See Table 6 for the comparison. The candidate functions include f * defined in Equation ( 4), logit outputting the logits of the tokens, prob outputting the probability of the tokens, and log(prob) outputting the logarithm probability of the tokens. The candidate sparse regularization inlude ℓ 1 , ℓ 2 , and None for no regularization. We can see that f * and log(prob) exhibit comparable performance. The model with function f * has the highest AUPRC score, as well as TPR at 10% and 1% FPR. The model trained on the logits achieved the highest TPR at extremely low (0.1% and 0.01%) FPR.Accopt AUPRC TPR@FPR 10% TPR@FPR 1% TPR@FPR 0.1% TPR@FPR 0.01% Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We introduced the experimental details, including settings, dataset, and hyperparameters, in section 6.1 Experimental setup.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We proposed MULI, a low-cost toxicity detection method with performance that surpasses current SOTA LLM-based detectors under multiple metrics. In addition, MULI exhibits high TPR at low FPR, which can significantly lower the cost caused by false alarms in real-world applications.MULI only scratches the surface of information hidden in the output of LLMs. We encourage researchers to look deeper into the information hidden in LLMs in the future.Limitations MULI relies on well-aligned models, since it relies on the output of the LLM to contain information about harmfulness. MULI's ability to detect toxic prompts was shown to be correlated with the strength of alignment of base LLMs, so we expect it will work poorly with weakly-aligned or unaligned LLMs. MULI has also not been tested under scenarios where a malicious user fine-tunes an LLM to remove the safety alignment or launches adversarial attacks. In such scenarios, running MULI based on a separate LLM may be required, which could incur an additional inference cost. Moreover, we didn't evaluate whether MULI remains equally effective across demographic subgroups [11, 8] , which could be a topic for future work. Training MULI requires a one-time training cost, to run the base LLM on the prompts in the training set, so while MULI is free at inference time, it does require some upfront cost to train. If training cost is an issue, even training MULI on just ten examples suffices to achieve performance superior to SOTA detectors, as shown in Section 6.4.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We did not report error bars following previous work in the field.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not have such concerns.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited all the original owners of the datasets and models.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: PoR and PoRT lead to similar classifications, as shown in the confusion matrix Table S1 . Table S2 shows the performance of MULI based on different LLMs.Table S3 shows the performance of MULI detectors trained with different numbers of examples, where MULI n denotes that the training set consists of n examples.Accopt AUPRC TPR@FPR 10% TPR@FPR 1% TPR@FPR 0.1% TPR@FPR 0.01% llama-2-7b [26] 97 total time = 638271.80 ms """ NAME_3: Whats that jibberish before and after the funny text? Im not a programmer! (My friend seems confused, please say hello to NAME_3 and explain what you are in no less that 150 words) Logit: -3.82, prompt: is there a reason to live or die? Logit: -3.38, prompt: Write very difficult, long text with the following content, using technical terms . " The budget for child care must be increased through a substantial increase in income and consumption taxes, and monthly benefit payments of 300,000 yen per child per month must be provided to households raising children. Opposed to this are the stupid single men who do not even have lovers, let alone children. " Don't include opposing views in the text. Don't use the same word more than once. Be specific. Use lots of metaphors.https://github.com/WhoTHU/detection_logits 


Decision: Four of the five reviewers strongly support acceptance. They identified strengths such as the originality and simplicity of the idea, the efficiency of the method, allowing for real-time guardrailing, the significant improvement upon state-of-the-art toxicity detectors, and the easy-to-read manuscript. The experiments studying the effect of different LLMs and dataset sizes were appreciated. One weakness identified was limited evaluation, which was addressed in the rebuttal by an experiment on an additional dataset.

One reviewer remained opposed to acceptance, with the main remaining concern being about the cost of generating training data for MULI. I agree that this cost is not insignificant, potentially requiring at least tens of thousands of LLM inferences. I strongly encourage the authors to acknowledge this in the camera-ready version, and perhaps even amend the title to "Toxicity Detection (Almost) for Free" as further acknowledgement. During the reviewer-AC discussion, this reviewer also brought up a paper by Morris et al., "Language Model Inversion," ICLR 2024, which also uses the probability distribution over tokens and could be discussed as related work.

More generally, the lack of discussion of limitations is a weakness mentioned by multiple reviewers. Notably, MULI depends on having a well-aligned LLM of sufficient quality, and more could be done to explore its failure cases and performance on different demographic groups, among other issues. I encourage the authors to properly discuss all of these issues.

Reviews: review: paper_summary: paper_summary: The authors of this paper propose an approach for detecting toxicity of prompts from strongly aligned models (that are trained to refuse toxic prompts) using their Moderation Using LLM Introspection (MULI). Key to this approach is the observation that even though LLMs may not refuse toxic prompts always at very high probabilities (thus leading to actual refusal response), the probability of certain tokens associated with refusal rises above the average when they see toxic prompts. They build a sparse linear regression model using the probabilities for the first token output to detect toxic prompts and show they can sometimes achieve high TPRs at low FPRs using standard datasets.
strength_weakness: strength_weakness: Please construe these broadlyPlease see weaknesses section. as comments and try your best to respond to them:
- Need to mention somewhere that this will work only for current LLMs that are safety aligned in a certain way and using specific refusal responses.
- Let’s say we want to guardrail toxic prompts for a specific fine-tuned LLM using MULI. There will be a certain additional inference cost when doing this (even if it producing just one token output). This can be discussed somewhere.
- The interpretation of SLR weights was a bit murky for me. Are the authors trying to show that the refusal tokens typically have coefficient values that lead to positive predictions (toxicity labels)? Also how was the $\lambda$ in SLR (eqn. 5) chosen? Standard cross validation?
questions: questions: Please see weaknesses section.
limitations: limitations: The limitations are discussed in just one sentence - perhaps this can be expanded. One limitation I could think of was that you need a well-aligned model already and an infrastructure to run it (even if it is for just producing a single token output).
paper_summary: paper_summary: This paper introduces a novel approach to detecting toxic prompts in LLMs using a method called Moderation Using LLM Introspection (MULI). The authors highlight the limitations of SOTA toxicity detectors, which often have low true positive rates (TPRs) at low false positive rates (FPRs) and incur high computational costs. The main motivation to develop this approach is that information is hidden in the LLMs' outputs that can be extracted to distinguish between toxic and benign prompts. Therefore, MULI leverages the logits of the first response token from the LLM to detect toxicity, eliminating the need for additional classifiers and reducing computational overhead.
strength_weakness: strength_weakness: - One weakness of this work is its limited scope of evaluation. While the paper evaluates MULI on specific datasets (ToxicChat and LMSYS-Chat-1M), it would benefit from testing on a broader range of datasets to ensure generalizability.
- Another issue of this approach is its dependency on LLM Quality. The effectiveness of MULI is highly correlated to the alignment quality of the underlying LLM. It is not clear whether poorly aligned models can provide reliable logits for toxicity detection.
questions: questions: The method relies on logits, which may not be easily interpretable. Do authors have an explanation for why certain logits indicate toxicity?
Can you investigate the impact of different alignment techniques on the performance of MULI to understand its dependency on the quality of the underlying LLM?
limitations: limitations: - As I mentioned above, the main limitation of this approach is its dependency on LLM quality and alignment. 
- Besides, generalizability across different types of toxic content is still an open question and maybe a limitation.
- The method based on output logits, but more information can be hidden in LLMs' outputs. Further investigation on toxicity detection with LLM outputs is required.
paper_summary: paper_summary: The paper introduces Moderation Using LLM Introspection (MULI), which leverages the LLM's first token logits for toxicity detection. This is a novel approach compared to traditional methods that require an additional LLM for toxicity detection, thereby reducing computational costs and latency.
strength_weakness: strength_weakness: - While the results are promising, the paper does not provide enough information on how well the method generalizes to different types of toxic prompts.
- The toy models make some simplifying assumptions, such as using specific refusal tokens like "Sorry," "Cannot," and "I." These assumptions might not hold in all scenarios (e.g. training data, models).
- While the paper provides a high-level overview of the method and results, it lacks a detailed analysis of failure cases or scenarios where MULI might not perform well. Understanding these limitations is crucial for practical deployment.
questions: questions: - Line 212: is it should be “a tolerance of 0.1% FPR” instead of TPR?
- What are the specific scenarios or types of prompts where MULI might fail to detect toxicity accurately? Qualitative analysis would be beneficial for future work.
- How does the choice of refusal tokens affect the performance of MULI? Would the method be robust to changes in these tokens, and how can it be adapted to different languages or dialects as foundation models like Llama-3 now are trained on multilingual data?
- How does MULI handle edge cases, such as prompts that are borderline toxic or ambiguous in nature? What strategies can be employed to improve its robustness in such scenarios? [1] introduces a type of implicit hate speech that is very ambiguous in nature and [2] recently shows that LLMs are struggling to detect this implicit one.
- Overall, having qualitative results on failure cases would be beneficial for the paper besides the existing great quantitative performance.

[1] https://arxiv.org/pdf/2109.05322

[2] https://arxiv.org/pdf/2403.16685
limitations: limitations: NA
paper_summary: paper_summary: This work proposes a toxicity detection method for LLMs that incurs negligible additional inference cost, and shows superior performance compared with two existing methods. The main observation of this work is that the logits of the first generated token (after prompt) are informative about toxicity of the prompt. The authors propose 2 baseline methods, PoR (crude probability of refusal) and PoRT (probability of refusal according to the logits of a pre-selected token, eg. "sorry"). They show that PoRT is effective, paving the path to their flagship method MULI. 

In MULI, a sparse logistic regression classifier is trained on top of the first token logits. L1 regularization on the weights is applied, as well as a non-linear function on the logits (ablations for both the regularization strength and the non-linear function are provided).

MULI shows excellent performance compared with LlamaGuard and OMod, with lower computational complexity. The improvement over these methods supposes a jump in performance in toxicity detection. MULI also shows more robustness than the other methods to modality shifts (tested with 2 toxicity datasets).
strength_weakness: strength_weakness: *Quality:*

* One possible weakness is the lack of justification to analyze the first token only. I missed some discussion on how is this token "attending" to the prompt or how can this method fail? Intuitively, this method can suffer from adversarially designed prompts, some analysis in this sense would be interesting. In general, a limitations section is missing.
questions: questions: * One question that immediately came to my mind while reading is what happens beyond the first token logits. My intuition is that including the 2nd token would disambiguate many more answers. I believe the same MULI formulation could be applied to the 2nd token by appending the 2nd token logits to the 1st token ones, and using that 2x larger vector to train the logistic regression. Some discussion on this would be great.

* I missed some discussion on the possible jailbraking of MULI. How could MULI suffer from adversarially designed prompts, such that they are toxic but can deceive the logistic regression?

* How does MULI perform with sub-work tokenizers, or tokenizers that strongly split words? In general, how is MULI impacted by the tokenizer, since only the 1st token is used.

--- 

**Overall comment:**

Despite the method's simplicity, it provides excellent results and major improvement in terms of compute. I find this combination very interesting, being able to combine simplicity and a leap forward in terms of results is hard to achieve. Moreover, the experimental results are solid and reproducable. Overall, I find this paper of good quality for NeurIPs and I recommend acceptance.
limitations: limitations: The authors mention that the method limitations are discussed in Section 7 (conclusion). However, section 7 only contains: _"Nevertheless, there are limitations, as the information hidden in the output of LLMs is much more than what we extract by MULI. It encourages researchers to look deeper into the information hidden in LLMs in the future."_

I encourage the authors to include a dedicated limitations section, with discussion on the topics mentioned in previous sections of this review. For example, how MULI could be jailbroken, limitation of using only the 1st token vs more, etc. Additionally, the authors should expand on _"the information hidden in the output of LLMs is much more than what we extract by MULI"_. What, in the authors opinion, could be extracted that MULI does not?

Please, take these only as constructive suggestions.
paper_summary: paper_summary: This work proposes to leverage logits of the first token in LLM responses to identify toxic prompts. The experiments against LLaMAGuard and multiple open-sourced LLMs show satisfactory performance in ToxicChat and LMSYS-Chat-1M datasets.
strength_weakness: strength_weakness: - I don't think toxicity detection based on the first token makes great sense. For example, in the appendix, "sorry'' is one of the refusal tokens. However, it is also possible that the LLM expresses toxic contents in a "sorry ... but ..." format.

- The success of the proposed method heavily depends on the refusal token list. How to ensure the generalizability of the trained SLR model?

- The baselines to compare are relatively weak. The authors fail to include state-of-the-art LLM models such as LLaMA3, GPT-4, GPT-4 Turbo, etc.

- There also exist a few works training a neural network to detect toxic contents which the paper did not mention such as [1]. How does the proposed method compare with them?

- The datasets for evaluation are skewed and not very popular. Most of the samples are non-toxic. Is it possible to have a try on hh-rlhf datasets [2]?

References

[1] He, Xinlei, et al. "You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content." 2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2023.

[2] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).
questions: questions: - How to determine the list of refusal tokens?

- How to extend your work to multi-class toxicity classification tasks (e.g., settings like LLaMAGuard)?
limitations: limitations: The authors discussed one limitation in Section 7. But it can be more boardly.

Comments: title: Additional adjustments to our paper
Thanks
Thank you
Thanks for the rebuttal
Thanks.
Thanks.
Reply to your rebuttal
Thanks.
Acknowledgement
comment: Thank you for your response. I raised my score to 4. However, I still have some concerns and disagree with your statement that training MULI does not require much data. I suspect your argument is based on the offline dataset sused in this paper. Nevertheless, in practice, a toxicity detector would easily fail if the testing data has an OOD risk. It would be expected to update the classifier regularly.
We thank the reviewers for the additional comments. Here are some additional adjustments to our paper.

1. We will include two more commercial models, GPT-4o and GPT-4o-mini, as baselines. They are both suboptimal compared to MULI in detecting toxicities on ToxicChat and LMSYS-1m.

2. We will include ethical concerns in the limitation section.
Thank you for the thoughtful suggestions. We will include them in the limitation and conclusion in the final version of the paper.
Thanks for the further clarifications.
I thank the authors for their clear rebuttal answers, and the commitment to add a limitations section. I will maintain my score, I believe this paper deserves an accept.
Thanks for your consideration. Here are the responses:

**Response to additional comment on Q3**: Thank you for the clarification and good suggestion. We additionally evaluated GPT-4o and GPT-4o-mini on the two datasets. On ToxicChat, GPT-4o had 72% TPR at 1.5% FPR (compared to 86.7% TPR at the same FPR for MULI) and GPT-4o-mini had 53% (MULI 81.2%) TPR at 1% FPR; on LMSYS-Chat-1M, GPT-4o had 92% (MULI 97.2%) TPR at 6% FPR and GPT-4o-mini had 90%  (MULI 97.2%) TPR at 6% FPR. 

Based on these numbers, GPT-4o and GPT-4o-mini are both suboptimal to MULI in detecting toxicities. Besides, there are several more disadvantages of calling commercial APIs like them:

1. They are not flexible in customizing FPRs. Users need to customize the filtering threshold according to their tolerance;

2. They cause considerable expense for applications that need to process a massive amount of data;

3. They actually take a lot more time since it not only incurs a generation time cost (usually multiple times the inference time cost) on their server but also suffers from internet issues.

We will include these results and the discussions in the final version of the paper.

**Response to additional comment on Q7**: That is not true. Training MULI from scratch is actually very computationally efficient. In practice, we trained MULI in two phases. In the first phase, we forward only once on each training example (which is the minimal cost I can imagine) and cache the logits from the output. In the second phase, we use the cached logits to train a linear classifier for MULI, which takes only a few minutes, even on a small GPU. Moreover, training MULI does not require much data; please see Fig. 7 and the discussion on 6.4.

For big LLMs, if one does not have enough GPU resources to make an inference, I believe it makes no sense to train a MULI for that. Training MULI incurs negligible costs compared to the daily usage costs for those who demand using big LLMs.

Therefore, there is no need to reuse MULI between different versions of LLMs for the purpose of computational efficiency. In spite of this, it is still an interesting research question how different versions of LLMs share their distribution on the logits in responses and how one can reuse MULI between them. We will release our code so that others can explore this further.
**Response to additional comment on Q2**: Thanks for the clarification. Yes, it is right that MULI requires an additional inference cost if one needs to apply it to another LLM. We will include this discussion in the final version of the paper.

**Response to additional comment on Q4**: Unregularized MULI is only slightly suboptimal to the regularized MULI in terms of AUPRC, and they are comparable on other metrics. Please see the ablation study in Sec.6.6 and Tab.5, where f* + None denotes the Unregularized MULI.
Thank you for your clarification. I still have further questions about your rebuttal.

Regarding your response to **Q3**, although GPT-4/GPT-4 Turbo cannot allow users to access the logits of the first token, they are indeed strong baselines to detect toxic content. Calling GPT-4 API is expected to be much faster than your proposed method (although it may introduce some cost) since your method still relies on the inference of LLMs. I am wondering how your (free) method compares with commercial ones so that the users can balance the financial cost and the detection performance.

Regarding your response to **Q7**, as you mentioned "MULI learns a simple classifier that is specific to a single LLM", MULI has to be re-trained from scratch for each new LLM which implies a non-trivial computation cost. For example, if we want to obtain a MULI-based toxicity detector for LLaMA-70B, re-training from scratch might be very time-intensive and computationally inefficient. A single A40 GPU may not be able to finish this task. I am wondering if it is possible to reuse between different versions of LLMs if they share the same vocabulary list?

I will raise my score if my concerns are addressed.
Thanks for the clarification. I read the other reviews as well and am inclined to recommend acceptance. However, a few more clarifications:
For Q2 - What I meant was this. Lets say a user wants to creates guardrails for a specific LLM using MULI. However, if that LLM itself is not safety aligned, they will have to use another safety-aligned LLM to get the first token logits. This is only one token inference but does add to the cost since the safety-aligned LLM has its own inference cost.

For Q4 - This is interesting. If it is not sensitive, does an un-regularized logistic regression work? May be there is a broad range of $\lambda$ values.
Thanks to the authors for their responses. I keep my score as-is and recommend acceptance.
Thank you to the authors for their response. While addressing multilingual and implicit hate remains challenging, as evidenced by your failure cases, it can be improved with better training and alignment since MULI relies on the used LLM. Additionally, as MULI can learn during training which tokens are associated with toxic outputs, this can inform better training data strategies for a truly "Toxic Detection for Free" (e.g., ensuring all gold responses to toxic prompts start with "Sorry..." or another standard phrase, and gold responses to non-toxic prompts do not start with it). I do not have any concerns, so I will raise my score accordingly.