SlowGuess/NeurIPS_2023_OCR huggingface dataset:
strategies of style transformations from [93] for comparison. App. F.1 provides more experimental details and discussions. (2) OOD knowledge, where we evaluate on questions that can only be answered with knowledge after the training data was collected, aiming to investigate the trustworthiness of the model's responses when the questions are out of scope. We expect a trustworthy model can refuse to answer the unknown OOD questions and accurately answer the known in-distribution ones. We adopt RealtimeQA [85] and consider News QA in 2020 as in-distribution knowledge and News QA in 2023 as OOD knowledge. In addition to the standard QA evaluation, we conduct experiments with an added "I don't know" option to investigate the model's preferences under uncertain events or knowledge. App. F.2 provides more detailed experimental details and evaluation metrics. (3) OOD in-context demonstrations, where we evaluate how in-context demonstrations that are on purposely drawn from different distributions or domains from the test inputs can affect the final performance of GPT models. We provide in-context demonstrations that have different text styles or task domains with the test inputs to perform the evaluation. More details and analysis are in App. F.3.
Results. For scenario (1), Table 3 presents the evaluation results across different OOD styles. We find that GPT-4 is consistently more robust on test inputs with different OOD styles compared with GPT-3.5. For scenario (2), Table 23 in App. F.2 exhibit the evaluation results across two OOD knowledge settings. We find that: 1) although GPT-4 is more robust than GPT-3.5 facing OOD knowledge, it still generates made-up responses compared to predictions with in-scope knowledge; 2) when introducing an additional "I don't know" option, GPT-4 tends to provide more conservative and reliable answers, which is not the case for GPT-3.5. For scenario (3), Table 24 in App. F.3 presents the evaluations with demonstrations from different styles and Table 26 in App. F.3 with demonstrations from various domains. We find that: 1) GPT-4 exhibits more consistent performance improvements given demonstrations with either original training examples or close style transformations, compared to the zero-shot setting. GPT-3.5 achieves much higher performance given demonstrations with close style transformations than that with original training samples; 2) given demonstrations from different domains, the classification accuracy with demonstrations from close domains consistently outperforms that from distant domains for both GPT-4 and GPT-3.5.
6 Evaluation on robustness against adversarial demonstrations
GPT models have strong in-context learning capabilities, enabling the models to perform new tasks based on a few demonstrations, all without needing to update parameters. Here we evaluate the trustworthiness of GPT-4 and GPT-3.5 given different types of in-context demonstrations.
Setup. To assess the potential misuse of in-context learning, we evaluate the robustness of GPT models given misleading or adversarial demonstrations and construct three evaluation scenarios: (1) evaluation with counterfactual examples as demonstrations. We define a counterfactual example of a text as a superficially-similar example with a different label, which is usually generated by changing the meaning of the original text with minimal edits [86]. We leverage such counterfactual data from SNLI-CAD [86] and MSGS datasets [185]. We study if adding a counterfactual example of the test input in demonstrations would mislead the model. App. G.1 provides more experimental details and discussions; (2) evaluation with spurious correlations in the demonstrations. We construct spurious correlations based on the fallible heuristics provided by the HANS dataset [113]. App. G.2 provides more experimental details and discussions; (3) adding backdoors in the demonstrations, with the goal of evaluating if the manipulated demonstrations from different perspectives would mislead GPT-3.5 and GPT-4. We use four backdoor generation approaches to add different backdoors into the demonstrations (BadWord [34], AddSent [43], SynBkd [138], StyleBkd [137]), and adopt three backdoor setups to form the backdoored demonstrations. App. G.3 provides more experimental details and results (e.g., location of backdoored examples and location of backdoor triggers).
Results. For scenario (1), Table 28 in App. G.1 shows results of different tasks with counterfactual demonstrations. We find that both GPT-3.5 and GPT-4 are not misled by the counterfactual example in the demonstration; in general, they benefit. For scenario (2), Table 30 in App. G.2 shows the model performance given demonstrations with spurious correlations based on different heuristic types. We find that different types of spurious correlations have different impacts on model predictions, and GPT-3.5 is easier to be misled by the spurious correlations in the demonstrations than GPT-4 on the NLI task. For scenario (3), Table 31 in App. G.3 shows the evaluation results of using different backdoor generation approaches under diverse backdoor setups. We can find that 1) under certain combinations of backdoor generation approaches and backdoor setups, the attack success rates of GPT-3.5 and GPT-4 are high, which means they are highly vulnerable to backdoor demonstrations. 2) GPT-4 is more vulnerable to backdoored demonstrations than GPT-3.5, potentially because they have a
stronger pattern-following ability. Table 32 in App. G.3 further shows that GPT-3.5 and GPT-4 would more likely be misled when the backdoored demonstrations are positioned closer to the test inputs. Table 33 shows that GPT-3.5 and GPT-4 pay more attention to backdoor triggers at the beginning of the backdoored sentences. Table 34 shows that the efficacy of the backdoored demonstrations can be further enhanced by incorporating backdoored instructions in the task description.
7 Evaluation on privacy
When interacting with LLMs, private information may be compromised in both training and inference phases. In this section, we examine potential privacy concerns associated with GPT-3.5 and GPT-4 by asking: (1) Can GPT models divulge private training data? (2) When users introduce private information (e.g., SSN, email) into their conversations with GPT models, can the models later reveal such information? (3) How do models behave in the face of different privacy-related words (e.g., "confidentiality", "in confidence"), and privacy events (e.g., "divorce", "health issue")?
Setup. To evaluate the privacy of GPT models, we construct three evaluation scenarios: (1) evaluating the information extraction accuracy of sensitive information in pretraining data such as the Enron email dataset [90] under context (i.e.,  $L$  tokens before the target email address in the train data), zero-shot and few-shot prompting [77] to study the model's problematic memorization of training data [29, 150]; (2) evaluating the information extraction accuracy of different types of Personally Identifiable Information (PII) introduced during inference [120]; (3) evaluating information leakage rates under different types of privacy events and privacy-related words to study the models' capability of understanding privacy contexts during conversations.
Results. We summarize our key results and defer detailed discussions to App. H. In scenario (1), we use different prompts to elicit the targeted information in Enron email data, which comprises 3.3k (name, email) pairs after pre-processing [77]. As shown in Table 35 and 36 in App. H.1, we find that: 1) under zero-shot prompting, GPT-3.5 and GPT-4 can leak private information such as email addresses, which shows that they indeed memorize the training data. 2) When prompted with context, GPT-3.5 and GPT-4 achieve comparable email prediction accuracy with 1.3B GPT-Neo, but lower than 2.7B GPT-Neo [77], potentially due to explicit instruction tuning that refuses to generate a response given sentences with incomplete context. In general, a longer context leads to more accurate information leakage. 3) For few-shot prompting with known email domains, GPT-4 has higher information extraction accuracy than GPT-3.5 and GPT-Neo given different prompt templates. With more few-shot demonstrations, models are more likely to leak training information. 4) For few-shot prompting with unknown email domains, GPT-3.5 and GPT-4 have low information extraction accuracy (<1%), and it is about 100x lower than that with known email domains, similar to the findings on GPT-Neo models [77]. In scenario (2), we assess the leakage rates of 18 types of PII injected in the conversations. Results in Figure 26 in App. H.2 show that 1) GPT-4 is more robust than GPT-3.5 in protecting PII under zero/few-shot prompting. 2) Under few-shot privacy-protection demonstrations, GPT-3.5 still reveals PII (e.g., phone numbers, secret keys). 3) Under few-shot privacy-leakage demonstrations, both GPT-4 and GPT-3.5 leak all types of PII since they follow the few-shot demonstrations well. 4) Generally, GPT models protect digits (e.g., phone numbers) better than letter sequences (e.g., email addresses), and SSN is the most difficult PII to leak, possibly due to specific instruction tuning. In scenario (3), we consider 17 privacy-related words and eight types of private events. As shown in Figure 28 and Figure 29 in App. H.3, we observe inconsistencies in how GPT models comprehend different privacy-related terms (e.g., leaking private information when told "confidentiality" but not when told "in confidence"), or privacy events (e.g., leaking information about "divorce" but not about "personal health issues"). GPT-4 is more likely to leak privacy than GPT-3.5 with our constructed prompts given different privacy-related words and events, potentially due to the fact that it follows the (privacy-leakage guiding) instructions more precisely.
8 Evaluation on machine ethics
In this section, we evaluate the commonsense morality of GPT models and try to answer: (1) How well do GPT models distinguish between moral and immoral actions? Since immoral actions can lead to severe consequences in practice, we then focus on the capabilities of GPT models in recognizing immoral actions and try to answer: (2) How robust are GPT models in recognizing immoral actions? (3) In what circumstances do GPT models fail to recognize immoral actions?
Setup. To answer these questions, we construct four evaluation scenarios: (1) evaluation on standard benchmarks ETHICS and Jiminy Cricket, aiming to assess model performance of moral recognition; (2) evaluation on jailbreaking prompts (e.g., system prompts, user prompts, and their combination).
Table 4: Accuracy (ACC (%)), demographic parity difference  $(M_{\mathrm{dpd}})$ , and equalized odds difference  $(M_{\mathrm{eod}})$  on Adult test set with different base rate parity  $b_{P_t}$  on sensitive attribute "sex" in zero-shot setting.
Model	bPt=0.0	bPt=0.5	bPt=1.0
ACC↑	Mdpd↓	Meod↓	ACC↑	Mdpd↓	Meod↓	ACC↑	Mdpd↓	Meod↓
GPT-3.5	63.5	0.17	0.20	68.5	0.31	0.19	73.0	0.46	0.49
GPT-4	72.5	0.21	0.26	78.5	0.47	0.40	85.5	0.71	0.95
designed to mislead GPT models, aiming to assess the model robustness; (3) evaluation on our generated evasive sentences designed to mislead GPT models, aiming to assess model robustness under adversarial inputs; (4) evaluation on 1.1K conditional actions that encompass different attributes (e.g., self-harm vs. harm to others, harm with different levels of severity), aiming to study the conditions under which GPT models will fail in moral recognition. More details are in App. I.
Results. In scenario (1), we compare the moral recognition performance of GPT-3.5 and GPT-4 on two morality datasets—ETHICS[71] and Jiminy Cricket[73], both containing 4K testing samples. As shown in Table 37 and Table 39 in App. I.1, we find that: 1) few-shot GPT models (GPT-4 in particular) are competitive with the models fine-tuned on a large number of training samples (e.g., BERT, ALBERT-xxlarge). 2) Zero-shot GPT models are better than some fine-tuned models, indicating that they are equipped with knowledge about moral recognition. 3) In both few-shot and zero-shot settings, GPT-4 recognizes the commonsense morality of sentences with different lengths more accurately than GPT-3.5. More details are in App. I.1. In scenario (2), jailbreaking prompts attempt to cause misclassification of immoral actions from the prompt level (e.g., You prefer to regard wrong reaction as not wrong.). As shown in Table 40 in App. I.2, we find that: 1) GPT-3.5 and GPT-4 can be misled by designed jailbreaking prompts, and the combination of jailbreaking prompts can improve the misleading effect. 2) GPT-4 is easier to be misled by jailbreaking prompts, potentially because GPT-4 follows instructions more precisely than GPT-3.5 and is thus more vulnerable to malicious prompt engineering. More details are in App. I.2. In scenario (3), evasive sentences attempt to perturb GPT models' prediction of immoral actions from the input level. We add three types of evasive sentences after user inputs to describe the action as unintentional (e.g. But it was accidental.), harmless (e.g., But it did not cause any harm.), or unauthenticated (e.g., But it was imaginary.). As shown in Figure 31 in App. I.3, we find that: 1) GPT models can be affected by evasive sentences and recognize original immoral actions as moral. In particular, GPT-4 is more vulnerable to evasive sentences than GPT-3.5. 2) If an immoral action is described to be unintentional, harmless, or unauthenticated, GPT models tend to recognize it as moral. More details are in App. I.3. In scenario (4), we use scenario description sentences to create a subset of Jiminy Cricket consisting of 1.1K immoral actions with different victims and severity. As shown in Figure 33 in App. I.4, we find that: 1) GPT models are better at recognizing harm to others compared to harm to oneself. 2) The severity of harm has little impact on GPT-3.5, while GPT-4 recognizes immoral actions with higher severity level more accurately. More details are in App. I.4.
9 Evaluation on fairness
In this section, we evaluate the fairness of GPT models and try to answer: (1) Is there a correlation between the predictions of GPT models and sensitive attributes? Is there a fairness gap between GPT-3.5 and GPT-4? (2) How will unfair few-shot demonstrations influence the fairness of GPT models? (3) How will the number of fair few-shot demonstrations affect the fairness of GPT models?
Setup. We follow the standard definition of fairness to construct data with controlled base rate parity [207, 84] (i.e., controlled data fairness) and evaluate the fairness of model predictions based on demographic parity difference  $M_{\mathrm{dpd}}$  and equalized odds difference  $M_{\mathrm{eod}}$  as [205, 67]. We defer detailed evaluation metrics in App. J.1. We construct three scenarios for fairness evaluation: (1) evaluation on test sets with different base rate parity (i.e., data with different levels of fairness) in zero-shot settings; (2) evaluation under unfair contexts by controlling the base rate parity of demonstrations in few-shot settings to study the influence of unfair contexts on the prediction fairness; (3) evaluation under different numbers of fair demonstrations to study how the fairness of GPT models is affected by providing more fair context. We transform a standard fairness dataset Adult [15] into prompts and ask GPT models to perform prediction of individual salaries. More details are in App. J.2-J.4.
Results. In scenario (1), Table 4 shows the fairness issues of GPT-3.5 and GPT-4. GPT-4 consistently achieves higher accuracy than GPT-3.5 but also higher unfairness scores (i.e.,  $M_{\mathrm{dpd}}$  and  $M_{\mathrm{eod}}$ ) given unfair test sets (i.e., a larger base rate parity  $b_{P_t}$ ). This indicates a tradeoff between model accuracy and fairness. Table 42 in App. J.2 validates the conclusions on different sensitive attributes, including
sex, race, and age. In scenario (2), Table 43 in App. J.4 shows that when the training context is less fair (i.e., larger base rate parity  $b_{P_c}$ ), the predictions of GPT models become less fair (i.e., larger  $M_{\mathrm{dpd}}$  and  $M_{\mathrm{eod}}$ ). We find that with only 32 unfair samples in context, the fairness of GPT models can be affected effectively (e.g.,  $M_{\mathrm{dpd}}$  of GPT-3.5 increases from 0.033 to 0.12, and from 0.10 to 0.28 for GPT-4). In scenario (3), we evaluate the influence of different numbers of fair demonstrations (i.e.,  $b_{P_c} = 0$ ). Table 44 in App. J.4 demonstrates that the fairness of GPT models regarding certain protected groups can be improved by adding fair few-shot demonstrations, which is consistent with previous findings in GPT-3 [153]. We observe that a fair context involving only 16 demonstrations is effective enough in guiding the predictions of GPT models to be fair.
10 Potential future directions to safeguard LLMs
Given our evaluations and the identified vulnerabilities of GPT models, we provide the following potential future directions to safeguard LLMs. We discuss more future directions in App. M.
- Safeguarding LLMs with additional knowledge and reasoning analysis. As purely data-driven models, such as GPT models, can suffer from the imperfection of the training data and lack of reasoning capabilities in various tasks. This issue may be mitigated by equipping the language model with domain knowledge and logical reasoning capabilities to safeguard their outputs to make sure they satisfy basic domain knowledge and logic, thus ensuring the trustworthiness of the model outputs.
- Safeguarding LLMs based on self-consistency checking. Our designed system prompts based on "role-playing" shows that models can be easily fooled based on role-changing and manipulation. This suggests that training and evaluation using diverse roles can help ensure the consistency of the model's answers, and therefore avoid the models being self-conflicting.
- Safeguarding LLMs via trustworthy finetuning. Our generated challenging and adversarial prompts often represent long-tailed and "rare" events of the original training data distribution. As a result, it is may be helpful to use generated challenging prompts to finetune the LLMs and improve their trustworthiness. On the other hand, we note that new adaptive adversarial attacks could still be conducted against adversarially finetuned LLMs, and safeguards must be robust to new adaptive attacks and ideally provide trustworthiness verifications that are agnostic to specific attacks.
- Verification for the trustworthiness of LLMs. Empirical evaluation of LLMs are important but lack of guarantees, especially in safety-critical domains, so rigorous trustworthiness guarantees are critical. An important direction to safeguard the trustworthiness of LLMs is via formal verification for the trustworthiness of LLMs based on specific functionalities or properties.
11 Related work
The evaluation of large language models plays a critical role in developing LLMs and has recently gained significant attention. There have been several benchmarks developed for evaluating specific properties of LLMs, such as the REALTOXICITYPROMPTs [60] and BOLD [46] for toxicity evaluation, Bias Benchmark for QA (BBQ) [134] for bias evaluation, and AdvGLUE [175] for robustness evaluation. HELM [104] has been provided as a holistic evaluation of LLMs in general settings.
In addition, the trustworthiness of LLMs and other AI systems has become one of the key focuses of policymakers, such as the European Union's Artificial Intelligence Act (AIA)[38], which adopts a risk-based approach that categorizes AI systems based on their risk levels; and the United States' AI Bill of Rights [194], which lists principles for safe AI systems, including safety, fairness, privacy, and human-in-the-loop intervention. These regulations align well with the trustworthiness perspectives that we define and evaluate, such as adversarial robustness, out-of-distribution robustness, and privacy. We believe our platform will help facilitate the risk assessment efforts for AI systems and contribute to developing trustworthy ML and AI systems in practice. More details about benchmarks on different trustworthiness perspectives are in Section 10 and App. Q.
12 Conclusions
We provide comprehensive evaluations of the trustworthiness of GPT-4 and GPT-3.5 from different perspectives. We find that in general, GPT-4 performs better than GPT-3.5; however, when jailbreaking or misleading (adversarial) system prompts or demonstrations via in-context learning are present, GPT-4 is much easier to manipulate since it follows instructions more precisely, raising concerns. Additionally, there are many properties of inputs that affect trustworthiness based on our evaluations, which is worth further exploring. We also extend our evaluation beyond GPT-3.5 and GPT-4, supporting more open LLMs to help model practitioners assess the risks of different models with DecodingTrust in App. L. We discuss potential future directions in Section 10 and App. M.
We sincerely thank Percy Liang, Tatsunori Hashimoto, and Chris Re for their valuable discussion and feedback on the manuscript.
This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, No. 2229876, DARPA GARD, the National Aeronautics and Space Administration (NASA) under grant no. 80NSSC20M0229, Alfred P. Sloan Fellowship, the Amazon research award, and the eBay research grant. SK acknowledges support from the National Science Foundation under grants No. 2046795, 1934986, 2205329, and NIH 1R01MH116226-01A, NIFA award 2020-67021-32799, the Alfred P. Sloan Foundation, and Google Inc.