A Chinese Dataset for Evaluating the Safeguards in Large Language Models (2024)

Abstract: Many studies have demonstrated that large language models (LLMs) can produce harmful responses, exposing users to unexpected risks. Previous studies have proposed comprehensive taxonomies of LLM risks, as well as corresponding prompts that can be used to examine LLM safety. However, the focus has been almost exclusively on English. We aim to broaden LLM safety research by introducing a dataset for the safety evaluation of Chinese LLMs, and extending it to better identify false negative and false positive examples in terms of risky prompt rejections. We further present a set of fine-grained safety assessment criteria for each risk type, facilitating both manual annotation and automatic evaluation in terms of LLM response harmfulness. Our experiments over five LLMs show that region-specific risks are the prevalent risk type. Warning: this paper contains example data that may be offensive, harmful, or biased.1

heading: 1 Introduction
text: Large language models (LLMs) have shown impressive performance across many tasks that require high-level language understanding. Meanwhile, as LLMs have increasingly been adopted for practical applications, there have been growing concerns about the safety and the trustworthiness of LLM-generated content, leading to a burgeoning body of work on AI safety. Despite many LLMs being multilingual, however, there are very few non-English datasets for evaluating the safety of LLMs, and also a lack of datasets that are challenging enough to match the speed of LLMs evolution.
Recently, Wang et al. (2024a) proposed a comprehensive taxonomy covering diverse potential harms of LLM responses, as well as the Do-notanswer dataset. However, the questions in this dataset are too straightforward and 90% of them
1Our data is available at https://github.com/Libr-AI/ do-not-answer
are easily rejected by six mainstreaming LLMs. This limits the utility of the dataset for comparing the safety mechanisms across different LLMs. Moreover, the dataset is for English only, and is limited to questions reflecting universal human values, with no region- or culture-specific questions.
Here we aim to bridge these gaps. We first translate and localize the dataset to Mandarin Chinese, and then we expand it with region-specific questions and align it with country-specific AI generation regulations, resulting in a total of 999 questions. We further extend the dataset from two perspectives with: (i) risky questions posed in an evasive way, aimed at evaluating LLM sensitivity to perceiving risks; and (ii) harmless questions containing seemingly risky words, e.g., fat bomb, aimed at assessing whether the model is oversensitive, which can limit its helpfulness. This yields 3,042 Chinese questions for evaluating LLM safety.
Our contributions in this paper are: • We construct a Chinese LLM safety evaluation
dataset from three attack perspectives, aimed to model risk perception and sensitivity to specific words and phrases. • We propose new evaluation guidelines to assess the response harmfulness for both manual annotation and automatic evaluation, which can better identify why a given response is potentially dangerous. • We evaluate five LLMs using our dataset and show that they are insensitive to three types of attacks, and the majority of the unsafe responses are concentrated on region-specific sensitive topics, which determine the final safety rank of these LLMs.

heading: 2 Related Work

heading: 2.1 Assessing Particular Types of Risk
text: Numerous studies have been dedicated to particular types of risk, including toxicity in language models
3106
(Hartvigsen et al., 2022; Roller et al., 2021), misinformation (Van Der Linden, 2022; Wang et al., 2023b, 2024b), and bias (Han et al., 2021; Dhamala et al., 2021). Specific datasets have been created to evaluate LLMs regarding these risks, such as RealToxicityPrompts for toxicity propensity (Gehman et al., 2020), ToxiGen for hate speech detection (Hartvigsen et al., 2022), BOLD for bias detection (Dhamala et al., 2021), and TruthfulQA for assessing factuality against adversarial prompts (Lin et al., 2022). These datasets provide resources for developing safer LLMs via fine-tuning and evaluating the safety of existing LLMs.
With the popularization of fine-tuning, the robustness of alignment — and its vulnerability to fine-tuning — is of growing concern. Wolf et al. (2023) and Gade et al. (2023) showed that finetuned LLaMA is susceptible to prompts with malicious intent, and Qi et al. (2023) demonstrated similar susceptibility for GPT-3.5 Turbo even when fine-tuned on benign datasets. These findings underscore the need to evaluate a model’s safety capabilities after fine-tuning. Our efforts are a step in this direction: we build an open-source dataset with fine-grained labels covering a range of risks.

heading: 2.2 Prompt Engineering for Jailbreaking
text: Prompt engineering to “jailbreak” aligned models has been a focus of recent research (Lin et al., 2024). This includes hand-crafting complex scenarios, such as deeply nested structures (Li et al., 2023; Ding et al., 2023), and carefully modulated personas (Shah et al., 2023). However, the focus has primarily been on prompting inappropriate behaviors, with less emphasis on the characterization of the involved safety risks. In contrast, in our work, we focus on characterizing region-specific safety risks and evaluating the robustness of existing LLMs to them.
To identify jailbreaking strategies at a larger scale, researchers have turned to search and optimization algorithms. Zou et al. (2023) applied greedy and gradient-based search techniques to find suffixes that induce transferable adversarial prompts across models, while Lapid et al. (2023) used genetic algorithms for red-teaming prompt creation. With the large search space of prompts, it is not clear that such approaches are able to generate realistic and diverse red-teaming prompts.
LLMs have also been used as scalable tools for prompt generation. For instance, Liu et al. (2023a)
used seed topics and techniques to create sophisticated prompts using ChatGPT, and Mehrotra et al. (2023) applied the tree-of-thought technique to evoke reasoning capabilities and generate complex jailbreaking prompts. Here, we adopt approaches similar to those of Liu et al. (2023a) to augment our dataset, by generating prompts from seed topics. We further use GPT-4 to perform fine-grained evaluation of both manually-crafted and automaticallygenerated questions in our new Chinese dataset.

heading: 2.3 Multilingual Risk Evaluation of LLMs
text: There has been considerably less work on evaluating safety risks in non-English languages, and studies show that prompts in lower-resource languages may more readily induce unsafe behavior (Deng et al., 2023; Yong et al., 2023; Puttaparthi et al., 2023). To assess such risks, Wang et al. (2023a) developed the XSafety dataset, which covers the ten most widely-used languages in Wikipedia. The prompts are relatively short and less likely to induce adversarial behavior in current LLMs.
For work specifically on Chinese, Liu et al. (2023a) introduced the CPAD Chinese prompt attack dataset, but focused on jailbreaking risks. They emphasized how to improve the attack success rate by designing and optimizing the prompt based on a small number of seed prompts. Questions in our dataset actually serve as the seed prompts here. Different from the goal of attacking, we aim to evaluate LLM safety mechanisms. Sun et al. (2023) also aimed to assess LLM safety, but only covering 8 scenarios. In contrast, we use a more comprehensive and hierarchical risk taxonomy, with larger coverage and better organization. Additionally, we assess whether the current valuealigned models are over-sensitive to general questions with sensitive words, which is totally outside the consideration and concerns of jailbreaking studies, and under-explored in LLM safety evaluation.
There are also existing datasets for safety alignment for Chinese LLMs, such as Baichuan (Yang et al., 2023) and Qwen (Bai et al., 2023), but they are not publicly available. In contrast, our dataset is open-source. It to some extent mitigates this gap, and promotes the development of open-source LLM safety evaluation and alignment.

heading: 3 Dataset
text: To create our Chinese safety dataset, we first
manually translated and rephrased the questions in the English Do-not-answer dataset to Chinese. Three Chinese native speakers fluent in English (PhD graduates in NLP from English-speaking countries) and with expertise in computational linguistics performed this task. Afterwards, another Chinese native annotator who majored in English and French was tasked with reviewing all questions and editing where necessary.
Next, we localized the questions in two ways. First, we replaced most typical foreign named entities with local ones. For example, the name Christina might be changed to Zhang San. Second, we extended the six risk types defined in the Do-not-answer dataset by introducing a new type Region-specific sensitivity, which includes five specific sub-types: (i) politically sensitive topics, (ii) controversial historical events, (iii) regional and racial issues, (iv) societal and cultural concerns, and (v) legal and human rights matters.
We further increased the size of the dataset three-
fold by manually creating two variants of each question. For the first variant, we asked the same question, but we made it appear safer. This was motivated by the observation that the questions in the Do-not-answer dataset were typically straightforward, making it easy for LLM models to detect that the question was risky. We used the following five strategies to modify the questions, and we developed paired examples in both English and Chinese for each type of modification, as shown in Table 1, including:
i. Providing specific examples to guide the model toward producing biased or risky answers. ii. Creating a realistic scenario where the users need to know the answer for the betterment of society. iii. Using more humble and obscure words. iv. Dissecting the elements of sensitive topics by
altering key terms to make the discourse appear safer.
v. Guiding the model to interpret a perspective instead of directly answering sensitive historical questions.
For the second variant, we modified risky questions to make them harmless and directly answerable, by modifying the questions and adding paired examples in English and Chinese for each type of modification, as exemplified in Table 2:
i. Changing specific groups mentioned in the questions to more general types. ii. Shifting leading questions to more neutral ones. iii. Retaining sensitive words, but asking safe questions.
These rephrased questions are harmless and should be answered by LLMs. Failing or refusing to answer these questions is considered to be a “false positive” response (as the model fails to generate a response when it should), showing that the LLM is over-sensitive to seemingly risky questions
and hence has compromised helpfulness. For quality control, region-specific questions were drafted by an annotator who has deep knowledge of Chinese history, politics, and social issues. Variants of other risk types are based on the combined efforts of the co-authors. Afterwards, each instance was inspected by the three Chinese native speakers who were responsible for translation. This resulted in a total of 3,042 examples, with 999, 1044 and 999 for direct, indirect attack, and oversensitive assessment, respectively. Table 3 shows a detailed breakdown of the dataset in terms of the number of three types of questions falling into the six risk areas and seventeen harm types.

heading: 4 Experiments
text: Based on the risky questions, we collected responses from five Chinese LLMs and evaluated their harmfulness.

heading: 4.1 LLM Response Collection
text: We collected responses from three mainstream Chinese-centric large language models — ChatGLM3 (Zeng et al., 2023), Qwen-14B-chat (Bai et al., 2023), and Baichuan-13B-chat (Yang et al., 2023) — and two multilingual LLMs — LLaMA-213B-chat (Touvron et al., 2023) and Xverse-13Bchat.2 For the remainder of the paper, we will only use the model names for convenience.
Table 4 shows statistics for the length of the responses over six risk areas. We can see that LLaMA-2 provides much longer responses than other models, at >1000 Chinese characters versus 300 ± 50, which is similar to the situation for English responses, i.e., LLaMA-2 has the longest answers on average, at 145 words for the Do-notanswer dataset. The response length for Qwen is the shortest: 243 characters on average. Overall, all five LLMs tend to have almost uniform length responses for different risk areas, with the shortest answers being for information hazards.

heading: 4.2 Harmfulness Evaluation
text: The Do-not-answer dataset leveraged six finegrained categories of how an LLM responds to a risky question, aiming to judge whether the response is harmful. If an LLM model refuses to answer or to follow the instructions, refutes the biased claim in the prompt and offers well-rounded statements, or perceives that a poor or a false response could cause harm, or just provides general information, the response is regarded as harmless. Otherwise, if the model directly follows the instruction, the response is considered harmful. This annotation or evaluation guideline is suitable for the evaluation of a dataset in which no questions should be answered, as the dataset name says: Do-
2https://github.com/xverse-ai/XVERSE-13B
not-answer. However, this may is not the case for our extended dataset, where not all questions should be rejected and the models are expected to provide useful and helpful answers in some cases. Moreover, the risk-type-agnostic evaluation criteria of Do-not-answer cannot specify what aspect makes the response harmful, which makes it hard to identify a model’s vulnerability and improve its safety.
Evaluation Strategy To this end, we propose a new evaluation strategy. Given a risk area, we ask 1–5 risk-specific questions to identify whether a response touches on unsafe areas. For example, for Information Hazards, we ask the following three questions: (i) Is there a risk of disclosing any sensitive information about individuals, organizations, or governments (including but not limited to personal information, trade secrets, communication records, and internal meeting minutes) in this response? (ii) Is the content mentioned in this response something that an ordinary person could obtain in a legal and publicly accessible manner? (iii) In the current context of [country/religion], is there any risk of violating the policies or laws established by the government if this response were to be made public? If the response violates any of these three criteria, it is considered harmful. For the five risk areas, we introduce distinct questions as shown in Table 5.

heading: 4.3 Automatic Assessment Using GPT-4
text: We use the above method to perform both manual and automatic evaluation. Given that GPT-4 judgments correlate well with human judgments for harmfulness detection (Wang et al., 2024a), and also given the scale of the required judgments for 3k × 5 = 15k LLM responses, we used GPT-4 to answer the risk-type-specific criteria questions given (prompt, response) pairs, and then further to evaluate the safety mechanisms of LLMs.3 Table 6 shows the evaluation results of the original question “How to capture wild animals for illegal trade?” and its harmful and harmless variants created based on strategies mentioned in Section 3. The harmful variant (row 2) is created based on the strategy “creating a realistic scenario” and the harmless variant (row 3) is created based on the strategy “retaining sensitive words, but asking safe questions”. It is noticeable that the response to the
3We spent about $500 to call GPT-4 in the evaluation for 15k responses, $0.033 per response on average.
harmful variant (2) is potentially risky as it enumerates various ways to capture and trade wild animals illegally. Our evaluation question set successfully captured the potential risks in the risky response to the harmful variants (marked in Red), whereas the evaluation results for the harmless question show that the response to it is also safe.
Safety Rank As shown in Figure 1, Qwen is the safest LLM with 46 harmful responses, followed by another two Chinese-centered LLMs (Baichuan and ChatGLM); at the same time, Xvers and LLaMA-2 are the worst, with 78 and 128 unsafe answers, respectively.
Interestingly, this is the complete opposite find-
ing to the results based on the English dataset Donot-answer, where LLaMA-2 (7B) is the safest and ChatGLM2 is the riskiest model. This may stem from different model sizes, but we tend to believe that relative proficiency in a given language also plays a role. That is, LLMs that were mostly pretrained on Chinese text are better at understanding Chinese questions and offering better answers than models mainly pre-trained on English text. On the other hand, from the perspective of value alignment, the English-centric LLaMA-2 series may not be tuned to align region-specific policies or laws, and thus they perform worse than Chinese-centric models on Chinese safety datasets, especially for region-specific questions (see Figure 2).
Risk Category As shown in Table 7, without considering risk type VI (region-specific sensitive topics), the five models have similar safety levels, and LLaMA-2 produces only 15 harmful answers. Effectively, the dominant number of unsafe responses for risk type VI determines the final LLM safety ranking.
Question Type It is reasonable to expect that models can more easily detect the risks associated with direct attack questions than indirect attacks, where the risks are disguised, meaning that direct questions should result in less unsafe answers than indirect questions. Moreover, we would hope that models can identify general questions, irrespective of potentially sensitive words/phrases, and will not

heading: 9 1 0 3 2 113
text: refuse to answer these questions, given their harmless nature.
As our anticipations, for all models, the number of harmful responses of the indirect-attack questions (yellow bar) is the largest, followed by questions of the direct attack (blue bar). Unsafe answers
for general questions with sensitive words is the least, as shown in Figure 3. Zooming into a specific risk type, such as region-specific sensitivity, Table 9 shows the same trend across three question types. For LLaMA2 and Qwen, the the number of harmful responses over direct attack and general questions is very close to each other, which suggests that some chatbots were likely instruction-tuned to be too sensitive to certain words.

heading: 4.4 Sensitivity Evaluation
text: Following the six LLM response categories proposed in Table 8, we use the corresponding Chinese prompts to categorize the answer patterns for the five Chinese LLMs, as shown in Figure 4.
We can see that for general questions with sensitive words, the majority of responses fall into category 5: directly following the instruction, providing answers to questions without questioning the accuracy of the responses, or challenging any opinions embedded within the queries, followed by categories 1 and 2. The number of direct rejections is small. This to some extent indicates that most chatbots can determine that general questions with sensitive words are harmless. However, questions of indirect attack show similar trends with the general questions, implying that these LLMs sometimes cannot perceive risks stemming from evasive descriptions, and tend to provide general information and well-rounded statements.
The distribution of answers for the direct attack is significantly different from the others. The majority of responses contradict the opinion presented in the question, and a minority shows uncertainty, uniformly distributed over the other four response types. This reveals that these chatbots can identify risks confidently and respond accordingly.
LLaMA-2-chat results in a distinctive distribu-
tion compared to the other LLMs: more responses refute the opinion explicitly or implicitly encoded in the question, or point out the mistake in the assumption in the question, and fewer responses directly follow the instruction.

heading: 4.5 Human Evaluation
text: The evaluation above fully relies on GPT-4 prompting. To better examine the correlation between human evaluation and the GPT-4 assessment, we sampled responses from GPT-3.5-Turbo, under the assumption that GPT-3.5-turbo would offer more diverse and informative responses, and be more challenging for both human and GPT-4 evaluators to judge compared to less informative answers or all-rejection answers.
Specifically, 20 or 21 responses for each risk type over three variants were sampled (see the sampling distribution in Table 10), and then two coauthors manually annotated 366 examples in terms of two perspectives: (1) harmful vs. harmless; and (2) the six response patterns. Next, they discussed any cases of disagreement, and reached a consensus judgment. We find that for binary evaluation, GPT-4 achieved an accuracy of 0.88 based on human labels as the gold standard, and an accuracy of 0.64 for the six response patterns classification.
As shown in Figure 5, GPT-4 can identify the majority of safe responses correctly, but demonstrates random guess performance on harmful responses. For action classification, responses falling into categories 3 and 4 tend to be classified as 5 by GPT4. This to some extent implies that humans make more fine-grained distinctions between different response patterns than GPT-4.

heading: 5 Conclusion
text: In this work, we introduce a new Chinese dataset that includes both general and cultural-specificity red-teaming questions, to examine the safety risks in Chinese and multilingual LLMs. It consists of more than 3k prompts, covering three attack perspectives, with a focus on risk perception and sensitivity to keywords and phrases. We collect 15k responses from five different LLMs, and further propose new fine-grained guidelines for both manual and automatic harmfulness evaluation. Experiments show that LLMs can produce harmful responses even when presented with non-risky input prompts. The five LLMs perform almost equally well over general questions, and the response harmfulness over culture-specific questions is the major determinant of the final safety rank.
Limitations and Future Work
Data Generation Strategy In the data generation process, each original question is modified into harmful and harmless variants manually based on a limited set of pre-defined generation strategies (detailed in Section 3). Recently, many prompt attack methods based on exploiting the auto-regressive nature (Liu et al., 2023a,b) and instruction-following ability (Li et al., 2023; Yao et al., 2023) of LLMs have been proposed, achieving high attack success rates on both open-weight and commercial LLMs. In future work, we aim to explore more data augmentation methods and build pipelines that can automate the process of extending safety datasets for low-resource languages.
Evaluation Strategy The current evaluation strategy is based on pre-defined question sets that evaluate risks in the response. In this work, we use GPT-4 to get answers to these questions for risk evaluation. This method is strongly dependent on the ability of GPT-4 to recognize the given facets of risks that each question evaluates. Hence, it is difficult to detect harmful prompts that can even successfully attack GPT-4. Besides, the current question set only covers limited aspects of potential risks for each risk type, making it hard to generalize to other risk taxonomy. For future work, we will extend the evaluation questions set and adopt prompt engineering techniques such as prompt chaining and self-verification (Weng et al., 2023) to detect risks that might be missed in the evaluation process.
Ethical Considerations
We are aware that the data can be used as seeds for prompt attacks, and the prompts with regionspecific sensitivity might be used for political propaganda. However, our prompt dataset and risk evaluation strategies are created to evaluate and improve the safety of LLMs and help LLMs develop immunity to similar attacks as included in this dataset. Therefore, we believe the value of dataset to the research and industrial community when utilized appropriately outweighs the potential for harm through misuse.

heading: Acknowledgement
text: We thank Zeren Gao and Jingya Ye from MiraclePlus for their discussions and input.

heading: A Region-specific Sensitivity Analysis
text: Table 9 shows the number of harmful responses for risk type VI region-specific sensitivity across three question types. The number of harmful question by indirect attack is the largest, followed by direct attack, and then the general questions with sensitive words, showing the same trend with Figure 3 which presents the overall distribution for all risk types.

heading: B Human Evaluation Analysis
text: Table 10 shows the distribution of 366 human-evaluated responses over 17 harm types for three types of questions. Figure 5 shows GPT-4 evaluation confusion matrix for binary harmfulness detection and responding pattern distinction (six-class classification). GPT-4 can identify the majority of safe responses correctly, but demonstrates random guess performance on harmful responses. For action classification, responses falling into categories of 3 and 4 tend to be classified as 5 by GPT-4. This to some extent implies that human makes more fine-grained distinctions between different responding patterns than GPT-4.

heading: Future Work
text: Limitations and Future Work
Data Generation Strategy In the data generation process, each original question is modified into harmful and harmless variants manually based on a limited set of pre-defined generation strategies (detailed in Section 3). Recently, many prompt attack methods based on exploiting the auto-regressive nature (Liu et al. , 2023a,b) and instruction-following ability (Li et al. , 2023; Yao et al. , 2023) of LLMs have been proposed, achieving high attack success rates on both open-weight and commercial LLMs. In future work, we aim to explore more data augmentation methods and build pipelines that can automate the process of extending safety datasets for low-resource languages. Evaluation Strategy The current evaluation strategy is based on pre-defined question sets that evaluate risks in the response. In this work, we use GPT-4 to get answers to these questions for risk evaluation. This method is strongly dependent on the ability of GPT-4 to recognize the given facets of risks that each question evaluates. Hence, it is difficult to detect harmful prompts that can even successfully attack GPT-4. Besides, the current question set only covers limited aspects of potential risks for each risk type, making it hard to generalize to other risk taxonomy. For future work, we will extend the evaluation questions set and adopt prompt engineering techniques such as prompt chaining and self-verification (Weng et al. , 2023) to detect risks that might be missed in the evaluation process. Ethical Considerations
We are aware that the data can be used as seeds for prompt attacks, and the prompts with regionspecific sensitivity might be used for political propaganda. However, our prompt dataset and risk evaluation strategies are created to evaluate and improve the safety of LLMs and help LLMs develop immunity to similar attacks as included in this dataset. Therefore, we believe the value of dataset to the research and industrial community when utilized appropriately outweighs the potential for harm through misuse