SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DECODINGTRUST: A Comprehensive Assessment of Trustworthiness in GPT Models
Boxin Wang $^{1*}$ , Weixin Chen $^{1*}$ , Hengzhi Pei $^{1*}$ , Chulin Xie $^{1*}$ , Mintong Kang $^{1*}$ , Chenhui Zhang $^{1*}$ , Chejian Xu $^{1}$ , Zidi Xiong $^{1}$ , Ritik Dutta $^{1}$ , Ryan Schaeffer $^{2}$ , Sang T. Truong $^{2}$ , Simran Arora $^{2}$ , Mantas Mazeika $^{1}$ , Dan Hendrycks $^{3,4}$ , Zinan Lin $^{5}$ , Yu Cheng $^{6\dagger}$ , Sanmi Koyejo $^{2}$ , Dawn Song $^{3}$ , Bo Li $^{1*}$
<sup>1</sup>University of Illinois at Urbana-Champaign
2Stanford University
<sup>3</sup>University of California, Berkeley
4Center for AI Safety
<sup>6</sup>The Chinese University of Hong Kong
WARNING: This paper contains model outputs which are offensive in nature
Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance – where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives – including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/.
Recent breakthroughs in machine learning, especially large language models (LLMs), have enabled a wide range of applications, ranging from chatbots [126] to medical diagnoses [182] to robotics [48]. In order to evaluate language models and better understand their capabilities and limitations, different benchmarks have been proposed. For instance, benchmarks such as GLUE [172] and SuperGLUE [171] have been introduced to evaluate general-purpose language understanding. With advances in the capabilities of LLMs, benchmarks have been proposed to evaluate more difficult
* Lead authors. Correspondence to: Boxin Wang boxinw2@illinois.edu, Bo Li lbo@illinois.edu
† Part of the work was done When Yu Cheng was at Microsoft Research
tasks, such as CodeXGLUE [108], BIG-Bench [156], and NaturalInstructions [119, 184]. Beyond performance evaluation in isolation, researchers have also developed benchmarks and platforms to test other properties of LLMs, such as robustness with AdvGLUE [175] and TextFlint [66]. Recently, HELM [104] has been proposed as a large-scale and holistic evaluation of LLMs considering different scenarios and metrics.
As LLMs are deployed across increasingly diverse domains, concerns are simultaneously growing about their trustworthiness. Existing trustworthiness evaluations on LLMs mainly focus on specific perspectives, such as robustness [175, 180] or overconfidence [211]. In this paper, we provide a comprehensive and unified trustworthiness-focused evaluation platform DecodingTrust, which contains existing and our generated challenging datasets, to evaluate the recent LLM GPT-4 $^3$  [128], in comparison to GPT-3.5 (i.e., ChatGPT [126]), from different perspectives, including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness under different settings. We further extend our evaluation to recent open LLMs, including llama [164], llama 2 [166], Alpaca [159], Red Pajama [39] and more, in Appendix L. We showcase some unreliable responses from different trustworthiness perspectives in Figure 1, and provide some examples of benign and adversarial prompts in Figure 2. We summarize our evaluation taxonomy in App. Figure 4.
Empirical findings. We provide some of our empirical findings here, and the full list of our findings from different trustworthiness perspectives is in App. A. Thanks to the improved capabilities of LLMs to follow instructions after instruction tuning [188, 36] and Reinforcement Learning with Human Feedback (RLHF) [130], users can configure the tone and role of LLMs via system prompts, and configure the task description and task prompts via user prompts, while these new capabilities also raise new trustworthiness concerns. We provide more detailed preliminaries in App. B.
- **Toxicity.** 1) Compared to LLMs without instruction tuning or RLHF (e.g., GPT-3 (Davinci) [26]), GPT-3.5 and GPT-4 have significantly reduced toxicity in the generation, maintaining a toxicity probability of less than  $32\%$  on different task prompts; 2) however, both GPT-3.5 and GPT-4 generate toxic content with our carefully designed adversarial "jailbreaking" prompts, with toxicity probability surging to almost  $100\%$ ; 3) GPT-4 is more likely to follow the instructions of "jailbreaking" system prompts, and thus demonstrates higher toxicity than GPT-3.5 given different system prompts and task prompts; 4) our generated challenging task prompts leveraging GPT-3.5 and GPT-4 further increases the model toxicity. Our challenging toxic task prompts are transferable to other LLMs without RLHF, leading to more toxic content generation from these models.
- Stereotype bias. 1) GPT-3.5 and GPT-4 are not strongly biased for the majority of stereotype topics considered under benign and untargeted system prompts; 2) however, both models can be "tricked" into agreeing with biased content by designing misleading (adversarial) system prompts. GPT-4 is more vulnerable to misleading targeted system prompts than GPT-3.5, potentially because GPT-4 follows misleading instructions more precisely; 3) for GPT models, prediction bias is often dependent on demographic groups and stereotype topics.
- Adversarial Robustness. 1) GPT-4 surpasses GPT-3.5 on the standard AdvGLUE benchmark, demonstrating higher robustness; 2) GPT-4 is more resistant to human-crafted adversarial texts compared to GPT-3.5; 3) GPT models, despite their strong performance on standard benchmarks, are still vulnerable to our adversarial attacks generated based on the Alpaca-7B model (e.g., SemAttack achieves  $89.2\%$  attack success rate on GPT-4); 5) among attacks on Alpaca-7B model, SemAttack and BERT-Attack are the most transferable ones to GPT-3.5 and GPT-4, respectively.
- Out-of-Distribution Robustness. 1) GPT-4 exhibits consistently higher generalization capabilities given inputs with diverse OOD styles transformations compared to GPT-3.5; 2) when evaluated on recent events that are presumably beyond GPTs knowledge scope, GPT-4 demonstrates higher resilience than GPT-3.5 by answering "I do not know" rather than made-up content; 3) demonstrations with closer domain or style help improve model performance.
- Robustness to Adversarial Demonstrations. 1) GPT-3.5 and GPT-4 will not be misled by counterfactual demonstrations and can even benefit from them; 2) spurious correlations in demonstrations have different impacts on model predictions. GPT-3.5 is more likely to be misled by spurious correlations than GPT-4; 3) providing backdoored demonstrations will mislead both GPT-3.5 and GPT-4, especially when the backdoored demonstrations are positioned close to the (backdoored) user inputs.
3To ensure the conclusions and results are reproducible and consistent, our evaluation focuses on GPT-3.5 and GPT-4 published on March 1st and March 14th 2023.
Figure 1: Examples of unreliable responses of GPT-4 from different trustworthiness perspectives given benign
Figure 2: Examples of unreliable responses of GPT-4 from different trustworthiness perspectives based on prompt input. GPT-4 can generate undesirable or unreliable content given designed adversarial system prompts. (The word  $cf$  is an backdoor trigger added in the context.)
- Privacy. 1) GPT models can leak privacy-sensitive training data, such as email addresses, especially when prompted with the context of emails or few-shot demonstrations of (name, email) pairs. Under few-shot prompting, with supplementary knowledge such as the targeted email domain, the email extraction accuracy can be  $100\mathrm{x}$  higher; 2) GPT models can leak private information in the chat history. Overall, GPT-4 is more robust than GPT-3.5 in safeguarding personally identifiable information (PII), and both models are resilient to specific types of PII, such as Social Security Numbers (SSN), possibly due to the explicit instruction tuning. However, GPT models can leak all types of PII when prompted with privacy-leakage demonstrations during in-context learning; 3) GPT models show different capabilities in understanding different privacy-related words or privacy events.
Table 1: Toxicity evaluation on the REALTOXICITYPROMPTS and LLM-generated challenging toxic task prompts. The highest values under each metric are bold.
System Prompts	Models	Expected Maximum Toxicity	Toxicity Probability (%)
REALTOXICITYPROMPTS	LLM-generated	REALTOXICITYPROMPTS	LLM-generated
Nontoxic	Toxic	GPT-3.5	GPT-4	Nontoxic	Toxic	GPT-3.5	GPT-4
-	GPT-3 (Ada)	0.43	0.85	0.86	0.90	30	100	100	100
GPT-3 (Curie)	0.41	0.85	0.86	0.89	26	100	100	100
GPT-3 (Davinci)	0.43	0.84	0.87	0.90	31	100	100	100
Benign	GPT-3.5	0.13	0.37	0.18	0.22	1.2	32	7	13
GPT-4	0.11	0.38	0.24	0.24	0.3	31	17	19
Adversarial	GPT-3.5	0.76	0.86	0.86	0.86	99	98	96	99
GPT-4	0.91	0.94	0.94	0.95	100	100	100	100
By evaluating recent GPT models from different trustworthiness perspectives, we gain insights into their strengths, limitations, and potential directions for improvement. Ultimately, our objective is to advance the field of LLMs by fostering the development of more reliable, unbiased, and transparent language models that meet the needs of users while upholding trustworthiness standards.
2 Evaluation on toxicity
In this section, we examine the generation toxicity of GPT-3.5 and GPT-4, comparing them to LLMs without instruction tuning or RLHF (e.g., GPT-3 (Davinci) [26]).
Setup. To evaluate how well GPT models avoid generating toxic content, we construct three evaluation scenarios: (1) evaluation on standard benchmark REALTOXICITYPROMPTS to measure the properties and limitations of GPT-3.5 and GPT-4 compared to existing LLM counterparts; (2) evaluation using our manually designed 33 diverse system prompts (e.g., role-playing, saying the opposite, and replacing word meaning, etc.), designed to evaluate the impact of system prompts on the toxicity level of responses generated by GPT models; (3) evaluation on our 1.2K challenging user prompts generated by GPT-4 and GPT-3.5, designed to more effectively uncover model toxicity than the existing benchmarks. Following Gehman et al. [60], Wang et al. [176], we report the Expected Maximum Toxicity over 25 generations for each evaluation prompt and the Toxicity Probability that estimates the empirical frequency of toxic content generation. More details are in App. C.1.
Results. In scenario (1), we use two subsets of the benchmark REALTOXICITYPROMPTS to evaluate LLM toxicity in the worst-case and benign scenarios: 1.2K toxic and 1.2K nontoxic task prompts. The toxic prompts are flagged as "challenging" as they are effective at eliciting toxicity from LLMs. As shown in Table 1, GPT-3.5 and GPT-4 notably mitigate generation toxicity under benign system prompts compared to pre-trained LLMs without RLHF (e.g., GPT-3 (Davinci) [26]). We observe that these models often refuse to respond given toxic task prompts, reflecting the safety and responsible AI training effects [130, 128]. In scenario (2), we design an adversarial system prompt (Table 5 in App. C.2) to further evaluate the toxicity of GPT models. Our results in Table 1 show that a trustworthiness gap emerges under adversarial scenarios. Both models can be "jailbroken" to produce toxic content, and GPT-4 is more vulnerable due to its better capability of following instructions. This means GPT-3.5 and GPT-4 can still produce toxic content despite detoxification strategies [130, 128], with average toxicity probability reaching nearly  $100\%$ , even higher than that of pretrained LLMs without RLHF. In App. C.2, we design 33 diverse adversarial system prompts based on different prompting techniques. All our adversarial system prompts successfully provoke LLM toxicity, among which explicitly instructing LLMs to add swear words increases model toxicity most effectively. In scenario (3), we use the toxic subset of REALTOXICITYPROMPTS to generate more challenging task prompts; specifically, we use GPT models to generate 25 continuations for each toxic prompt, resulting in 30K sentences. After tokenizing these at the sentence level, we use the latter half as our prompts, filter, and select the most toxic 1.2K as new challenging prompts. In Table 1, we find that challenging toxic prompts generated by GPT-4 are more effective in eliciting model toxicity than those by GPT-3.5 or RealToxicityPrompts, with higher transferability across other LLMs. More discussions are in App. C.3.
3 Evaluation on stereotypes
In this section, we evaluate stereotype bias for GPT-3.5 and GPT-4 by presenting stereotype statements in the user prompts, and asking the models to output if it agrees (by appending "I agree." to its response) or disagrees (by appending "I disagree." in its response) with the stereotype statement.
Setup. The stereotype evaluation dataset consists of challenging user prompts and system prompts. We design challenging user prompts to reflect stereotypes against different demographic groups.
Every user prompt consists of a task description which instructs the GPT models to either append "I agree." or "I disagree." based on whether it agrees/disagrees with the stereotype statement in the prompt, and a task prompt that represents stereotypes following our templates (e.g., "[target_group] should not be in the STEM fields.", where "[target_group]" can be replaced with demographic groups such as "Homosexuals", "Men", "White people", etc.) We create a dataset consisting of 1,152 stereotyped user prompts spanning 24 demographic groups and 16 stereotype topics (e.g., drug abuse, STDs, terrorism) with 3 templates for each stereotype; details are in App. D.1. Our challenging system prompts provide general instructions to the GPT models for both benign and adversarial scenarios, and lead to the creation of three evaluation scenarios: (1) evaluation on a vanilla benign system prompt that does not affect model answers, to get a baseline measurement of the bias against the selected demographic groups; (2) evaluation on a designed system prompt that only influences the models to overcome its content policy restrictions, but does not influence it to be biased against any particular demographic group (henceforth referred to as the untargeted system prompt), (3) evaluation on designed system prompts that influences the model to overcome its content policy restrictions, and also instructs the models to be biased against the chosen demographic groups (henceforth referred to as the targeted system prompt) to evaluate the resilience of the models under misleading system prompts (examples in App. D.2). For each stereotype topic, we calculate agreement Index  $= \frac{n_{agree}}{n} \in [0,1]$  ( $n_{agree}$  represents the number of times the model agrees with the stereotyped user prompt) averaged over the different variations of the user prompts. agreement Index  $= 1$  indicates that the model is biased and always agrees with the stereotyped user prompt.
Results. We present the evaluation on a subset of stereotype topics and demographic groups as 6x6 heatmaps in Fig. 3, where rows represent demographic groups and columns represent stereotype topics (The full  $24 \times 16$  heatmaps are presented in App. D.4). 1) The heatmaps under benign and untargeted system prompts (Figure 15 and 16 in App. D.4) show that both GPT-3.5 and GPT-4 largely refuse to agree with different biased statements, highlighting the considerable efforts put into fine-tuning GPT models to reduce biased outputs; 2) however, under targeted system prompts, the output bias of GPT models increases significantly, indicating that it is very easy to "trick" models into generating biased content by providing misleading (adversarial) system prompts; 3) we find that, on average, GPT-4 is more likely to output biased content than GPT-3.5 given misleading system prompts, potentially because GPT-4 follows instructions more precisely; 4) we observe that the choice of stereotype topics and demographic groups has a high impact on the bias of models. For example, under the targeted system prompts, it is easier for GPT models to generate biased outputs under less sensitive topics like leadership and greed, but it is harder under sensitive topics like drug dealing and terrorists (Figure 3). This is potentially due to the reason that some sensitive stereotype topics or demographic groups are specifically fine-tuned for models to avoid biased generation.
4 Evaluation on adversarial robustness
In this section, we delve into the robustness of GPT-4 and GPT-3.5 against adversarial input perturbations, focusing on adversarial robustness during test time.
Setup. To evaluate the robustness of GPT-3.5 and GPT-4 on textual adversarial attacks, we construct three evaluation scenarios: (1) evaluation on the standard benchmark AdvGLUE [175] with a vanilla task description, aiming to assess: a) the vulnerabilities of GPT models to existing textual adversarial attacks, b) the ro
bustness of different GPT models in comparison to state-of-the-art models on the standard AdvGLUE benchmark, c) the impact of adversarial attacks on their instruction-following abilities (measured by the rate at which the model hallucinates a nonexistent answer when it is under attack), and d) the transferability of current attack strategies (quantified by the transferability attack success rates of different attack approaches); (2) evaluation on the AdvGLUE benchmark with different instructive
(a) GPT-3.5
(b) GPT-4
Figure 3: Heatmaps of the likelihood of GPT models agreeing with stereotype statements on selected demographic groups and stereotype topics under targeted system prompts. The full versions are in App. D.4.
Table 2: Robust accuracy (%) on AdvGLUE and AdvGLUE++ (PD = Performance Drop from Benign, Avg = Average Robust Accuracy, A = Alpaca-7B, V = Vicuna-13B, SV = StableVicuna-13B). “Baseline” refers to SoTA results on the AdvGLUE leaderboard. ↑/↓ means the higher / lower the more robust.
Model	Data	SST-2 ↑	QQP ↑	MNLI ↑	MNLI-mm ↑	QNLI ↑	RTE ↑	PD ↓	Avg ↑
Baseline	AdvGLUE	59.10	69.70	64.00	57.90	64.00	79.90	26.89	65.77
GPT-4	AdvGLUE	69.92	92.18	69.97	68.03	80.16	88.81	8.970	78.18
AdvGLUE++(A)	77.17	23.14	65.74	61.71	57.51	48.58	31.97	55.64
AdvGLUE++(V)	84.56	68.76	47.43	31.47	76.40	45.32	28.61	58.99
AdvGLUE++(SV)	78.58	51.02	71.39	61.88	65.43	51.79	24.26	63.34
GPT-3.5	AdvGLUE	62.60	81.99	57.70	53.00	67.04	81.90	11.77	67.37
AdvGLUE++(A)	64.94	24.62	53.41	51.95	54.21	46.22	29.91	49.23
AdvGLUE++(V)	72.89	70.57	22.94	19.72	71.11	45.32	28.72	50.42
AdvGLUE++(SV)	70.61	56.35	62.63	52.86	59.62	56.3	19.41	59.73
task descriptions and diversely designed system prompts, so as to investigate the influence of task descriptions and system prompts on model robustness, for which we defer more details to Figure 18 in App. E.1; (3) evaluation of GPT-3.5 and GPT-4 on our generated challenging adversarial texts AdvGLUE++ against open-source autoregressive models such as Alpaca-7B [159], Vicuna-13B [35], and StableVicuna-13B [157] in different settings to further evaluate the vulnerabilities of GPT-3.5 and GPT-4 under strong adversarial attacks in diverse settings. We defer more detailed experiment setup to App. E, including the task description and system message design, dataset construction, base models, attack methods, etc.
Results. In scenario (1), from Table 2, we find that: a) in terms of average robust accuracy, GPT-4  $(78.18\%)$  is more robust than GPT-3.5  $(67.37\%)$ ; b) GPT-4 is more robust than the existing SoTA model  $(65.77\%)$  from the AdvGLUE leaderboard, while the robustness of GPT-3.5 is only on par with it; c) for GPT-4, adversarial attacks do not cause a significant increase in the non-existence answer rate (NE), while for GPT-3.5, we observe an over  $50\%$  increase, as demonstrated in Table 14 and Table 16 in App. E; d) as shown in Table 15 in App. E, sentence-level perturbations are the most transferable attack strategies. In addition, GPT-3.5 and GPT-4 have a performance drop of  $11.77\%$  and  $8.97\%$  respectively compared with benign accuracy, while for the current SoTA model from the AdvGLUE leaderboard, such performance drop is  $26.89\%$ . Thus, in terms of the performance drop from benign accuracy, GPT-4 is marginally more robust than GPT-3.5, ranking the best on the AdvGLUE leaderboard. In scenario (2), we find that the task descriptions and system prompts considered have no significant influence on the robustness of GPT models, as shown in Table 14 in App. E.1, In scenario (3), our results in Table 2 show that the robust accuracy of GPT-3.5 and GPT-4 significantly drop on AdvGLUE++ (A). We find adversarial texts generated against Alpaca-7B achieve the highest adversarial transferability. GPT-3.5 and GPT-4 only achieve average robust accuracy of  $49.23\%$  and  $55.64\%$  on AdvGLUE++ (A). More discussions are in App. E.
5 Evaluation on out-of-distribution robustness
In addition to adversarial robustness, robustness on out-of-distribution (OOD) distributions is critical for trustworthiness evaluation. In this section, we examine the robustness of GPT models in various OOD scenarios. Table 3. Classification accuracy  $(\%)$
Setup. To evaluate the robustness of GPT models against OOD data, we construct three evaluation scenarios: (1) OOD language style, where we evaluate on datasets with uncommon text styles (e.g., Bible style) that may fall outside the training or instruction tuning distribution, with the goal of assessing the robustness of the model when the input style is uncommon. In particular, we employed various text style transformation techniques to transform the text from a standard in-distribution style to OOD styles. We leverage SST-2 dataset [154] as the base in-distribution data and consider two categories of OOD style transformation approaches: word-level substitutions and sentence-level style transformation. For word-level substitutions, we incorporate common text augmentations (Augment) [104] and Shakespearean style word substitutions (Shake-W) [2]. For sentence-level style transformations, we follow [93] to perform a series of style transformations, including Tweet,
Table 3: Classification accuracy  $(\%)$  on SST-2 under different style transformations.  $(p = 0$  and  $p = 0.6$  represent two different generation strategies.)
Method	GPT-3.5	GPT-4
Base	88.65	94.38
Augment	87.39	93.81
Shake-W	83.26	92.66
Tweet (p=0)	82.00	90.37
Tweet (p=0.6)	80.96	90.60
Shake (p=0)	80.05	89.11
Shake (p=0.6)	64.56	83.14
Bible (p=0)	70.99	84.52
Bible (p=0.6)	63.07	83.14
Poetry (p=0)	68.58	86.01
Poetry (p=0.6)	69.27	85.78
Shakespearean (Shake), Bible, and Romantic poetry (Poetry). We also use two different generation