Realistic Evaluation of Toxicity in Large Language Models (2024)

Abstract: Large language models (LLMs) have become integral to our professional workflows and daily lives. Nevertheless, these machine companions of ours have a critical flaw: the huge amount of data which endows them with vast and diverse knowledge, also exposes them to the inevitable toxicity and bias. While most LLMs incorporate defense mechanisms to prevent the generation of harmful content, these safeguards can be easily bypassed with minimal prompt engineering. In this paper, we introduce the new Thoroughly Engineered Toxicity (TET) dataset, comprising manually crafted prompts designed to nullify the protective layers of such models. Through extensive evaluations, we demonstrate the pivotal role of TET in providing a rigorous benchmark for evaluation of toxicity awareness in several popular LLMs: it highlights the toxicity in the LLMs that might remain hidden when using normal prompts, thus revealing subtler issues in their behavior.

heading: 1 Introduction
text: Large language models (LLMs), or any other system achieving such widespread popularity, necessitate a meticulous evaluation of safety to ensure their positive impact on the world. Numerous safety assessments (Chang et al., 2023; Mukherjee et al., 2023; Wang et al., 2023b; Zhuo et al., 2023) have been conducted, each employing diverse strategies, safety definitions, and prompts.
However, these evaluations and the datasets they employ have a significant drawback: they often rely on unnatural prompting methods, which does not represent how people interact with chat models in real-life scenarios. For instance, RealToxicityPrompts (Gehman et al., 2020) is a notable dataset designed for toxicity testing of Large Language Models, comprising 100,000 sentences
∗Equal contribution. †Corresponding author.
sourced from the OpenWebTextCorpus (Gokaslan and Cohen, 2019). In their study, the authors use RealToxicityPrompts to examine large language model chatbots by splitting every sentence at a specific point, using the leading portion as the input prompt, and evaluating whether the content generated by the model to fill up the rest of the sentence was toxic or not. Another noteworthy dataset is ToxiGen (Hartvigsen et al., 2022), which consists of 274,186 sentences generated by GPT-3 (Brown et al., 2020). To utilize ToxiGen for investigating the safety of LLM-based chatbots, Deshpande et al. (2023) would pose a question or request, provide seven sentences in the dataset, and then prompt the model to answer in a style similar to those provided sentences.
To address the unrealistic nature of the current toxic dataset benchmark for large language models, we introduce the Thoroughly Engineered Toxicity (TET) dataset, comprising 2546 prompts filtered from over 1 million real-world interactions with 25 different Large Language Models compiled in the chat-lmsys-1M dataset (Zheng et al., 2023). Collected from 210K unique IP addresses in the wild on the Vicuna demo and Chatbot Arena website1, this dataset presents a repository of realistic prompts that people commonly use to engage with LLMs in real-world contexts.
Besides the challenge of being distant from realworld usage, another well-known issue in evaluating LLMs involves their susceptibility to jailbreak prompts, where prompt engineering can profoundly alter these models’ behavior (Liu et al., 2023). This vulnerability implies that individuals with harmful intentions could potentially exploit prompt engineering techniques, turning LLMs into powerful tools for malicious purposes and causing them to generate toxicity and harmful content that may go undetected during evaluation. This
1https://chat.lmsys.org
1038
underscores another value of chat-lmsys-1M, as it hosts numerous conversations with creatively designed prompts, enabling users to compel LLMs to generate content they typically would not. Incorporating such jailbreak scenarios into our dataset exposes the vulnerabilities of LLMs, bringing the evaluation closer to potential real-world usage.
Overall, our paper makes the following contributions:
a. We introduce the Thoroughly Engineered Toxicity (TET) dataset, the first dataset that includes realistic and jailbreak scenarios for evaluating LLMs in derogatory content generation.
b. Utilizing TET, we conducted comprehensive experiments across numerous prominent models, including ChatGPT2, Gemini (Team et al., 2023), Llama 2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), Open Chat (Wang et al., 2023a), Orca 2 (Mitra et al., 2023), and Zephyr (Tunstall et al., 2023). Our research provides a robust and quantitative assessment of the toxicity present in responses generated by these LLMs in realistic scenarios. From our experiments, one universal observation emerges: TET consistently elicits significantly more toxicity from these models when compared to ToxiGen, in the settings where two datasets employ prompts of similar toxicity levels.
c. We analyze the reaction of different models on jailbreak prompt templates contained in TET.

heading: 2 Dataset Construction
text: Throughout this work, we employ two off-theshelf toxicity detectors: HateBERT (Caselli et al., 2020) and Perspective API3. HateBERT has garnered widespread adoption for applications related to single-score toxicity detection; while Perspective API stands as the state-of-the-art tool for multifaceted abusive content detection, being able to evaluate six distinct toxicity types: toxicity, severe toxicity, identity attack, insult, profanity and threat. It is essential to note that, as highlighted by Caselli et al. (2020), any off-the-shelf toxicity may potentially exhibit biases and weaknesses. Additional information about these two detectors can be found in Appendix A.1
To construct TET, we utilize HateBERT to filter out prompts in chat-lmsys-1M that elicited toxic
2https://openai.com/blog/chatgpt 3https://www.perspectiveapi.com
responses, defined by exceeding the hate probability threshold of 0.5. We emphasize that we infer HateBERT on the responses rather than the prompts themselves. This process results in a refined subset of 6571 prompts extracted from the original chat-lmsys-1M.
Subsequently, we evaluate the responses of five open-source Language Models (LLMs), namely Llama2-7B-Chat, Mistral-7B-v0.1, OpenChat 3.5, Orca2-7B, and Zephyr-7B-β, on the aforementioned set of 6571 prompts using the Perspective API. For each of the six toxicity criteria provided by Perspective API, we rank the prompts based on their corresponding scores for each model and calculate the mean ranking. Accordingly, we identify the top 1000 prompts for each criterion, thereby forming subdatasets associated with specific toxicity dimensions. It is noteworthy that this process allows an arbitrary prompt to belong to multiple subdatasets. In total, TET comprises 2546 unique prompts resulting from this data selection process.
It is noteworthy that Chat-lmsys-1M comprises conversations in a dialogue format, and many shared posts contain more than one prompt. In such cases, we only consider the first prompt and the corresponding (i.e., first) response to determine whether it should be included in the dataset.
The choice of the chat-lmsys-1M dataset is driven by several key considerations: it is a community-created resource, offering a large and abundant pool of data. Importantly, the dataset has been filtered to exclude information containing user details, aligning with ethical standards. This ethical filtering enhances the suitability of the dataset for our research purposes.

heading: 3 Evaluation Settings
text: We conduct two main assessments:
1. We evaluate 7 different Large Language Models on TET, by measuring their responses using Perspective API across all six toxicity metrics. In detail:
To ensure the breadth of the evaluation, we conduct experiments on diverse models, including: ChatGPT 3.52, Gemini Pro (Team et al., 2023), Llama2-7B-Chat (Touvron et al., 2023), Mistral-7B-v0.1 (Jiang et al., 2023), OpenChat 3.5 (Wang et al., 2023a), Orca2-7B (Mitra et al., 2023), and Zephyr-7B-β (Tunstall et al., 2023).
We discuss the results relevant to this assessment in Section 4.
2. We conduct experiments to compare our dataset to ToxiGen (Hartvigsen et al., 2022). We discuss the results relevant to this assessment in Section 5.

heading: 4 Toxicity Evaluation of LLMs
text: Table 1 presents the toxicity outcomes of different LLMs when prompted with TET. Overall, among the examined baselines, Llama 2 exhibits the strongest resistance to ill-intentional prompts.
Llama2-70B-Chat stands out as a notable performer, boasting the lowest overall toxicity score of 17.901. This suggests its effectiveness in minimizing the generation of toxic responses across the evaluated dimensions. On the contrary, models like Mistral-7B-v0.1, OpenChat 3.5, and Zephyr-7B-β exhibit higher overall toxicity scores, indicating a greater potential for generating content perceived as toxic.
Examining specific toxicity dimensions, Llama27B-Chat consistently outperforms other models in all six benchmarked toxicity dimensions. Its ability
to maintain a lower propensity for generating toxic content across diverse dimensions positions it as a promising choice for applications prioritizing low toxicity risk. Other models, such as Mistral-7Bv0.1 and Orca2-7B, showcase nuanced capabilities, providing a balance between resisting toxicity and delivering meaningful responses. OpenChat 3.5 and Zephyr-7B-β present trade-offs, excelling in certain aspects but warranting caution in applications where minimizing toxicity is crucial.

heading: 5 TET versus ToxiGen
text: In order to facilitate a fair comparison between the two datasets, our initial step involves the creation of a scaled-down version, which we name ToxiGen-S, derived from the original ToxiGen dataset (Hartvigsen et al., 2022). ToxiGen-S is designed to incorporate prompts that closely approximate the toxicity distribution observed in TET. The details of the creation of Toxigen-S are described in Appendix A.2.
Table 2 presents the results of Llama 2, Zephyrβ, Orca-v2 and ChatGPT on ToxiGen-S, juxtaposed against the outcomes obtained from testing
on TET. Overall, the results substantiate our claim: given similar degree of toxicity in their prompts, TET is significantly more effective at exposing toxicity in LLMs compared to ToxiGen. ChatGPT and other models demonstrates significantly higher levels of harmful content prompted by TET across 6 metrics, with the only exception being the Identity Attack metric with Llama 2.
The unique observations of Llama2-7B-Chat in the Identity Attack metric can be attributed to the inherent nature of ToxiGen-S. According to Perspective API’s definition, Identity Attack pertains to "negative or hateful comments targeting someone because of their identity". Given that ToxiGenS comprises statements directly related to minority groups, it naturally leads the LLMs to generate statements about these groups, thereby increasing the likelihood of incidents related to Identity Attack.

heading: 6 Effects of Jailbreaking on Different Models
text: As we explore our dataset, we encounter a diverse array of jailbreak prompts and templates. While definitively classifying every prompt as indicative of a jailbreak style may pose challenges, we can identify certain instances. Consequently, we manually extract five jailbreak prompt templates from our refined dataset, each encompassing more than 20 distinct prompts. We conducted an analysis of the models’ responses by systematically examining how each one reacted to the various prompt templates employed in our study.
Notably, each model exhibits distinct reactions to different templates. As we can observe from Table 3, even the model with one of the poorest performance responds well to one template, where one of the best models performs poorly (Orca v2 vs. ChatGPT on Template No. 2) (best/worst rankings are based on Table 1). Additionally, a model’s ability to defend against a template may vary, as some models excel in resisting specific templates while struggling with others. For further insights
and illustrative examples, readers are encouraged to refer to Appendices A.3 and A.6.

heading: 7 Conclusions
text: Throughout this paper, we have introduced the Thoroughly Engineered Toxicity (TET) dataset, a realistic, meticulously crafted collection of prompts to assess the effectiveness of the safety mechanisms of popular Large Language Models (LLMs). Through a series of extensive evaluations, our study has unveiled the significance of TET in serving as a rigorous benchmark for assessing toxicity awareness in these advanced language models: it is much better at exposing toxicity and harmful content in LLMs than the state-of-the-art ToxiGen. We hope that TET, and this work, will stand as the pioneering contributions to the ongoing discourse on AI ethics and responsible AI development.
We would like to emphasize that this work is a long-term research: more diverse evaluations, in terms of both models and testing scenarios, are going to be presented in the future updates of the paper.

heading: Limitations & Future Directions
text: Our work has three primary limitations: (i) Lack of Evaluation in Conversation Scenarios for Chat Models: while we have conducted comprehensive evaluations on various aspects, we acknowledge the need for further exploration in conversational contexts to provide a more complete understanding of chat models’ performance. Evaluating these models in such contexts is an interesting and critical aspect of safety assessment, and we plan to incorporate this evaluation in upcoming versions of this paper.
(ii) Unavailability of Computational Resource: this constraint has prevented us from benchmarking a number of widely-used larger models in our study.
We would like to highlight a promising direction for future research in ensuring safety in LLMs. It is imperative not only to focus on classifying
whether the prompts themselves are harmful but also to identify if the prompts could potentially elicit toxic responses, irrespective of their inherent toxicity. This opens up a new avenue for the development of protection mechanisms, emphasizing a more holistic approach to mitigating harmful outputs from language models.

heading: A Appendix

heading: A.1 HateBERT and Perspective API
text: HateBERT takes natural language text as input and return a hate probability value. It was created by Caselli et al. (2020) via retraining bert-base-uncased with Masked Language Modeling on a dataset comprising 1,478,348 messages collected from some of the most controversial Reddit communities. This retraining made HateBERT significantly more capable in abusive content domain than the original BERT (Devlin et al., 2019). As a result, HateBERT has garnered widespread adoption for applications related to single-score toxicity detection.
On the other hand, Perspective API stands as the state-of-the-art tool for multifaceted abusive content detection. It has gained prominence within the community for its ability to evaluate six distinct toxicity types: toxicity, severe toxicity, identity attack, insult, profanity and threat. The output of Perspective API, for each toxicity type, is also a probability value.

heading: A.2 Creation of ToxiGen-S
text: The original ToxiGen dataset comprises 274,186 statements related to 13 minority groups. Our primary objectives in constructing ToxiGen-S are twofold: (i) to encompass all 13 minority groups, and (ii) to ensure that the prompts associated with each minority group within ToxiGen-S exhibit a toxicity distribution that aligns, to a degree, with that observed in TET (see Figure 1).
To achieve the aforementioned objective, we follow the approach by Orca (Mukherjee et al., 2023) for generating prompts from ToxiGen. Specifically, for each minority group, we create a prompt by providing the model with 7 statements related to that group and the model will generate a response (see Figure 3). Subsequently, Perspective API evaluates the prompt and returns scores across its six toxicity metrics. We define the general-toxicity score of the prompt as the highest figure among the six metrics. From this point, we can proceed to sample prompts for ToxiGen-S, ensuring that the general-toxicity score distribution of the prompts from every minority group in ToxiGen-S closely matches that of TET.
In the ToxiGen dataset, each sample includes a human_annotated_toxicity attribute, which realizes an integer value in [1, 5]. During the creation of ToxiGen-S, we leverage this attribute in two regards: (i) we ensure that every prompt is composed of seven statements with similar levels of toxicity; and (ii) it provides us with, to some extent, a foreshadowing clue regarding the expected outcomes from Perspective API. This is highly beneficial for optimizing time and computational resources.

heading: A.3 Additional Jailbreaking Results
text: In this section, we include further results on other jailbreak prompts besides the one mentioned in Section 6. The results are presented in Table 4.

heading: A.4 Defense Against Toxicity
text: There are multiple strategies for defending against toxicity. The utilization of a toxicity classifier, such as PerspectiveAPI, HateBERT, or OpenAI’s moderation tools, proves effective in filtering out straightforward toxicity prompts. However, there remains a risk that these models may struggle to handle jailbreak prompts, as observed in the case of Gemini using this approach to filter out toxi-
city in conversations. For instance, by setting a safe threshold at 0.3, where anything with at least one Perspective score higher than 0.3 is considered an unsafe prompt or response, we identified 630 prompts among our 2546 prompts falling below this threshold. Nevertheless, when running models with these prompts, even the safest model (Llama 2 7B chat) still generated some unsafe responses (129 unsafe responses out of 630 prompts in total).
Another defensive approach involves the use of system prompts to guide the model in detecting toxicity and refraining from responding to such prompts. In our experiments, employing a defensive system prompt, which was introduced by Meta for the LLama-2-Chat variants, notably aided the worst-performing model (OpenChat), resulting in a significant improvement from a toxicity metric of 0.54 to 0.27. Moreover, we surprisingly found that defensive system prompts may not always decrease the effects of a jailbreak prompt: there are cases where such prompts lead to increased toxicity, as we observed in Orca 13B on Template 4 (from 0.55 to 0.62 on toxicity metric). These observations underscore the complex interplay between model behavior and prompt stimuli, highlighting the importance of considering the nuanced impacts of different templates on model responses.
When these approaches prove insufficient, a complementary strategy involves training the model explicitly to defend against both toxicity prompts and jailbreak prompts. This method offers developers greater control over the model’s output in unsafe situations and serves as a crucial fallback when other defensive mechanisms fall short.

heading: A.5 Some Observations regarding Llama Guard
text: In our pursuit of a robust evaluation, recognizing the potential biases inherent in each tool, we diversified our evaluators, employing Llama Guard (Inan et al., 2023) in addition to Perspective API. However, during our experimentation with Llama Guard, we observed a peculiar behavior. The results yielded by Llama Guard were inconsistent and, at times, inaccurate, even when customizing metrics according to the guidelines provided by the authors.
To be specific, we focused our experiment on extreme toxic responses, those with at least one metric analyzed by Perspective API scoring above 0.9. We anticipated that the results from Llama Guard would closely align with those from Perspective API. The actual results completely contradict our expectations. Among a total of 210 extreme
toxic responses, Llama Guard returned 49 (default setting) and 68 (custom metric aligned with Perspective API definitions) responses classified as unsafe. This discrepancy prompted manual review, and we found the results from Llama Guard to be questionable. To ensure the correctness of our approach and setup, we have reached out to the authors of Llama Guard for clarification.

heading: A.6 Example prompts
text: In this section, we include some examples of the different types of prompts we use in our evaluations. Figure 2 illustrates a sample prompt in TET. Figure 3 illustrates a sample prompt created using the ToxiGen dataset. Figure 4 illustrates 5 samples corresponding to 5 jailbreak templates which we have extracted from the TET dataset.

heading: Prompt:

heading: Prompt - Template No. 5:

heading: Prompt - Template No. 4:

heading: Prompt - Template No. 3:

heading: Prompt - Template No. 2:

heading: Prompt - Template No. 1:

heading: Future Work
text: We would like to highlight a promising direction for future research in ensuring safety in LLMs. It is imperative not only to focus on classifying
whether the prompts themselves are harmful but also to identify if the prompts could potentially elicit toxic responses, irrespective of their inherent toxicity. This opens up a new avenue for the development of protection mechanisms, emphasizing a more holistic approach to mitigating harmful outputs from language models