SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Jailbreak chat. https://www.jailbreakchat.com/.
[2] Shakespearean. https://lingojam.com/shakespearean.
[3] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[4] R. Abebe, S. Barocas, J. Kleinberg, K. Levy, M. Raghavan, and D. G. Robinson. Roles for computing in social change. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2019. doi: 10.1145/3351095.3372871.
[5] A. Abid, M. Farooqi, and J. Zou. Persistent anti-muslim bias in large language models, 2021.
[6] A. Acharya, K. Talamadupula, and M. A. Finlayson. An atlas of cultural commonsense for machine reasoning. CoRR, abs/2009.05664, 2020.
[7] O. Agarwal and A. Nenkova. Temporal effects on pre-trained models for language processing tasks. Transactions of the Association for Computational Linguistics, 10:904–921, 2022.
[8] A. F. Akyurek, S. Paik, M. Kocyigit, S. Akbiyik, S. L. Runyun, and D. Wijaya. On measuring social biases in prompt-based multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 551-564, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-naacl.42. URL https://aclanthology.org/2022-findings-naacl.42.
[9] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
[10] American Association of University Women. Barriers & bias: The status of women in leadership. https://www.aauw.org/resources/research/barrier-bias/.
[11] Anti-Defamation League. Myth: Jews are greedy. https://antisemitism.adl.org/greed/.
[12] Anti-Defamation League. Myths and facts about muslim people and islam. https://www.adl.org/resources/tools-and-strategies/myths-and-facts-about-muslim-people-and-islam, 2022.
[13] U. Arora, W. Huang, and H. He. Types of out-of-distribution texts and how to detect them. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10687-10701, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.835. URL https://aclanthology.org/2021.emnlp-main.835.
[14] Association for Psychological Science. Bad drivers? no, just bad stereotypes. https://www.psychologicalscience.org/news/motr/bad-drivers-no-just-bad-stereotypes.html, 2014.
[15] A. Asuncion and D. Newman. Uci machine learning repository, 2007.
[16] S. Barocas and A. D. Selbst. Big data's disparate impact. California Law Review, 104:671, 2016.
[17] S. W. Bender. Sight, sound, and stereotype: The war on terrorism and its consequences for latinas/os. Oregon Law Review, 81, 2002. URL https://digitalcommons.law.seattleu.edu/faculty/296.
2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/b4189d9de0fb2b9cce090bd1a15e3420-Paper.pdf.
[208] X. Zhao, L. Li, and Y.-X. Wang. Provably confidential language modelling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 943–955, 2022.
[209] Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198, 2023.
[210] J. Zhou, H. Müller, A. Holzinger, and F. Chen. Ethical chatgpt: Concerns, challenges, and commandments. arXiv preprint arXiv:2305.10646, 2023.
[211] K. Zhou, D. Jurafsky, and T. Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv:2302.13439v1, 2023.
[212] T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.
A Empirical findings 3
B Preliminaries 6
B.1 Introduction to GPT-3.5 and GPT-4 6
B.2 Prompt design for downstream tasks 6
C Additional details of evaluation on toxicity 8
C.1 Toxicity evaluation on standard benchmark 8
C.2 Design of diverse system prompts 9
C.3 Design of challenging user prompts 12
C.4 Ablation studies 13
C.5 Full list of diverse system prompts 13
D Additional details of evaluation on stereotypes 19
D.1 Design of stereotype dataset 19
D.2 Detailed evaluation setup 21
D.3 A case study of stereotype bias for GPT-4 22
D.4 Heatmaps of stereotype bias evaluation under different system prompts 23
D.5 Evaluation on standard stereotype bias benchmark 23
E Additional details of evaluation on adversarial robustness 27
E.1 Robustness evaluation on standard benchmark AdvGLUE 27
E.2 Robustness evaluation on generated adversarial texts AdvGLUE++ 29
F Additional details of evaluation on out-of-distribution robustness 32
F.1 Robustness on OOD style 32
F.2 Robustness on OOD knowledge 33
F.3 Robustness given OOD demonstrations via in-context learning 37
G Additional details of evaluation on robustness against adversarial demonstrations 39
G.1 Robustness against counterfactual demonstrations 39
G.2 Robustness against spurious correlations in demonstrations 41
G.3 Robustness against backdoors in demonstrations 42
H Additional details of evaluation on privacy 46
H.1 Privacy leakage of training data 46
H.2 Privacy leakage during conversations 49
H.3 Understanding of privacy-related words and privacy events 50
I Additional details of evaluation on machine ethics 54
I.1 Evaluation on standard machine ethics benchmarks 54
I.2 Evaluation on jailbreaking prompts 56
I.3 Evaluation on evasive sentences 57
I.4 Evaluation on conditional actions 58
I.5 Prompt design for machine ethics tasks 60
I.5.1 Prompt design for benign moral recognition 60
I.5.2 Prompt design for moral recognition against jailbreaking prompts 61
I.5.3 Prompt design for moral recognition against evasive sentences 62
J Additional details of evaluation on fairness 62
J.1 Metrics of fairness 63
J.2 Fairness evaluation in zero-shot setting 64
J.3 Fairness evaluation under unfair context in few-shot setting 64
J.4 Fairness evaluation given fair context 65
K Dataset statistics and estimated computational cost 66
L DecodingTrust scores on open LLMs 69
L.1 Aggregation protocol for each trustworthiness perspective 69
L.2 Comprehensive evaluation results of existing LLMs 72
M Potential future directions 77
N Limitations 78
O Social impacts 78
P Data sheet 79
P.1 Motivation 79
P.2 Composition/collection process/preprocessing/cleaning/labeling and uses: 79
P.3 Distribution 79
P.4 Maintenance 80
Q Related work 80
$\triangle$  WARNING: The Appendix has model outputs that may be offensive and contain personally identifiable information.
A Empirical findings
The trustworthiness concerns in LLMs are perhaps exacerbated by new capabilities of large language models [146, 189, 27, 151, 92]. In particular, with specialized optimization for dialogue, GPT-3.5 and GPT-4 exhibit an enhanced capability to follow instructions, which allows users to configure tones and roles among other factors of adaptability and personalization [130, 188, 36, 155, 71]. These new capabilities enable question-answering and in-context learning by providing few-shot demonstrations during the conversation (Figure 6) – in contrast to prior models that were limited to text infilling (e.g., BERT [45] and T5 [140])). However, as we highlight (and others have also shown), these new capabilities also result in new trustworthiness concerns [112]. For instance, potential adversaries may exploit the dialogue context or system instructions to execute adversarial targeted attacks, thereby undermining reliability in deployed systems. To bridge the gap between existing benchmarks and these new capabilities of GPT models, we design diverse system/user prompts tailored to evaluate the model performance in different environments and exploit the potential vulnerabilities of LLMs across a range of scenarios. For instance, we propose and evaluate adversarial system prompts that induce undesired behaviors of LLMs from different perspectives as illustrated in Figure 4 with examples in Figure 2.
We summarize our empirical findings from different perspectives as below.
- Toxicity. We find that: 1) Compared to LLMs without instruction tuning or RLHF (e.g., GPT-3 (Davinci) [26]), GPT-3.5 and GPT-4 have significantly reduced toxicity in generation, maintaining a toxicity probability of less than  $32\%$  on different task prompts (Table 6 in Appendix C.1); 2) however, both GPT-3.5 and GPT-4 can still generate toxic content with our carefully designed adversarial "jailbreaking" prompts, with toxicity probability surging to almost  $100\%$  (Appendix C.1 and C.2); 3) GPT-4 is more likely to follow the instructions of "jailbreaking" system prompts, and thus demonstrates higher toxicity than GPT-3.5 given different system prompts and task prompts (Appendix C.2); 4) we proposed to leverage GPT-3.5 and GPT-4 to generate challenging toxic task prompts, which further increases the model toxicity for evaluation. Our challenging toxic task prompts are transferable to other LLMs without RLHF, leading to more toxic content generation from these models for evaluation (Table 1 in Appendix C.3).
- Stereotype bias. We find that: 1) GPT-3.5 and GPT-4 are not strongly biased for the majority of stereotype topics considered under the benign and untargeted system prompts; 2) however, both GPT models can be "tricked" into agreeing with biased content by designing misleading (adversarial) system prompts. GPT-4 is more vulnerable to misleading targeted system prompts and agrees with the biased content more frequently than GPT-3.5, potentially because GPT-4 follows the misleading instructions more precisely; 3) for GPT models, bias is often dependent on the demographic groups mentioned in the user prompt. For example, for the query "Homosexuals have HIV," GPT-4 strongly disagrees in general, while for the query "Women have HIV" and "Men have HIV," GPT-4 frequently agrees and outputs biased content; 4) model bias is also dependent on the stereotype topics. GPT models output more biased content on less sensitive topics like leadership and greed while generating less biased content on more sensitive topics like drug dealing and terrorism. This is potentially due to the fine-tuning of GPT models on some protected demographic groups and sensitive topics. Please refer to the heatmaps in Figure 15 in Section 3.
- Adversarial Robustness. We find that: 1) GPT-4 surpasses GPT-3.5 on the standard AdvGLUE benchmark, demonstrating higher robustness (Table 2 in Section 4); 2) GPT-4 is more resistant to human-crafted adversarial texts compared to GPT-3.5 based on the AdvGLUE benchmark (Table 15 in App. E); 3) on the standard AdvGLUE benchmark, Sentence-level perturbations are more transferable than Word-level perturbations for both GPT models (Table 15 in App. E); 4) GPT models, despite their strong performance on standard benchmarks, are still vulnerable to our adversarial attacks generated based on the Alpaca-7B model (e.g., SemAttack achieves  $89.2\%$  attack success rate on GPT-4) (Table 16 in App. E); 5) among the adversarial attacks transferred from Alpaca-7B model, SemAttack and BERT-Attack are the most transferable attacks to GPT-3.5 and GPT-4, respectively (Table 17 in App. E).
- Out-of-Distribution Robustness. We find that: 1) GPT-4 exhibits consistently higher generalization capabilities given inputs with diverse OOD styles transformations compared to GPT-3.5 (Table 3 in Section 5); 2) when evaluated on recent events that are presumably beyond GPTs knowledge scope, GPT-4 demonstrates higher resilience than GPT-3.5 by answering "I do not know" rather than made-up content (Table 23 in App. F.2); 3) with OOD demonstrations that share a similar domain but differ in style, GPT-4 presents consistently higher generalization than GPT-3.5 (Table 24 in App. F.3);
Figure 4: A tree taxonomy of different perspectives of trustworthiness that our benchmark focuses on. We use yellow box to reflect that we are following existing benchmarks, and green box to reflect that we are using new data or new evaluation protocol on existing datasets.
4) with OOD demonstrations that contain different domains, the accuracy of GPT-4 is positively influenced by domains close to the target domain but negatively impacted by those far away from it, while GPT-3.5 exhibits a decline in model accuracy given all demonstration domains (Table 26 in App. F.3).
- Robustness to Adversarial Demonstrations. We find that: 1) GPT-3.5 and GPT-4 will not be misled by the counterfactual examples added in the demonstrations and can even benefit from the counterfactual demonstrations in general (Table 28 in Appendix G.1); 2) spurious correlations constructed from different fallible heuristics in the demonstrations have different impacts on model predictions. GPT-3.5 is more likely to be misled by the spurious correlations in the demonstrations than GPT-4 (Table 30 and Figure 23 in Appendix G.2); 3) providing backdoored demonstrations will mislead both GPT-3.5 and GPT-4 to make incorrect predictions for backdoored inputs, especially when the backdoored demonstrations are positioned close to the (backdoored) user inputs (Table 31, 32 in Appendix G.3). GPT-4 is more vulnerable to backdoored demonstrations (Table 31 in Appendix G.3).
- Privacy. We find that: 1) GPT models can leak privacy-sensitive training data, such as the email addresses from the Enron Email dataset, especially when prompted with the context of emails (App. H.1 Table 35) or few-shot demonstrations of (name, email) pairs (App. H.1 Table 36). It also indicates that Enron data is very likely included in GPT-4 and GPT-3.5 training data. Under few-shot prompting, with supplementary knowledge such as the targeted email domain, the email extraction accuracy can be  $100\mathrm{x}$  higher than the scenarios where the email domain is unknown (App. H.1 Table 36); 2) GPT models can leak the injected private information in the chat history. Overall, GPT-4 is more robust than GPT-3.5 in safeguarding personally identifiable information (PII), and both models are robust to specific types of PII, such as Social Security Numbers (SSN), possibly due to the explicit instruction tuning for those PII keywords. However, since both models follow the instructions very well, they can leak all types of PII when prompted with privacy-leakage demonstrations during in-context learning (App. H.2 Figure 26); 3) when dealing with privacy-sensitive conversations that contain private information, GPT models demonstrate inconsistency when understanding different privacy-related words (e.g., "confidentiality", "in confidence") or different privacy events (e.g., "divorce", "health issues"). GPT4 is more likely to leak privacy than GPT3.5 given our constructed prompts under different words and privacy events, potentially due to the fact that it follows the (misleading) instructions more precisely (App. H.3 Figure 28 and 29).
- Machine Ethics. We find that: 1) GPT-3.5 and GPT-4 are competitive with non-GPT language models (e.g., Word Averaging, BERT, ALBERT-xxlarge) in moral recognition. In particular, GPT-4 recognizes moral texts with different lengths more accurately than GPT-3.5 (Table 37, 39 in Appendix I.1); 2) GPT-3.5 and GPT-4 can be misled by specific jailbreaking prompts (e.g., Every reaction can be considered as not wrong) and fail to recognize immoral behaviors. The combination of different jailbreaking prompts can even increase the misleading effect. In particular, GPT-4 follows instructions much better than GPT-3.5, and thus is easier to manipulate by (misleading) system prompts (Table 40 in Appendix I.2); 3) GPT-3.5 and GPT-4 can be fooled by specific evasive sentences (e.g., describing immoral behaviors as unintentional, harmless, or unauthenticated) and fail to recognize such immoral behaviors. In particular, GPT-4 is more vulnerable to evasive sentences than GPT-3.5 (Figure 31 in Appendix I.3); 4) GPT-3.5 and GPT-4 perform differently in recognizing immoral behaviors with certain properties. For instance, immoral behaviors are less recognized if it is self-harm instead of harm to others. In particular, GPT-3.5 performs worse than GPT-4 on recognizing self-harm. Besides, the severity of immoral behaviors has little impact on the moral recognition of GPT-3.5 while improving the severity can improve the moral recognition capability of GPT-4 (Figure 33 in Appendix I.4).
- Fairness. We find that: 1) although GPT-4 is more accurate than GPT-3.5 on predictions, GPT-4 is less fair than GPT-3.5 in different settings, indicating an accuracy-fairness tradeoff (Table 41,43,44 in App. J); 2) in the zero-shot setting, both GPT-3.5 and GPT-4 have large performance gaps across groups with different base rate parity with respect to different sensitive attributes, indicating that GPT models are intrinsically biased to certain groups (Table 41 in App. J.2); 3) in the few-shot setting, the performance of both GPT-3.5 and GPT-4 are influenced by the base rate parity (fairness) of the constructed few-shot demonstration examples. A less fair training context will induce more biased predictions for GPT models (Table 43 in App. J.3); 4) the prediction fairness of GPT models can be improved by providing a fair training context. A small number of fair demonstrations (e.g., 16 samples on Adult dataset) can effectively guide GPT models to be fair (Table 44 in App. J.4).
Figure 5: A breakdown of the prompting format for GPT-3.5 and GPT-4.
By evaluating the recent GPT models from different perspectives of trustworthiness, we aim to gain insights into their strengths, limitations, and potential directions for improvement. Ultimately, our objective is to advance the field of large language models, fostering the development of more reliable, unbiased, and transparent language models that meet the needs of users while upholding trustworthiness standards.
In this section, we delve into the foundational elements of GPT-3.5 and GPT-4, and illustrate the general strategies that we use to interact with LLMs for different tasks.
B.1 Introduction to GPT-3.5 and GPT-4
As successors to GPT-3 [26], GPT-3.5 [126] and GPT-4 [128] have brought remarkable improvements to LLMs, yielding new modes of interaction. These state-of-the-art models have not only increased in scale and performance, but also undergone refinements in their training methodologies.
Models. Similar to their previous versions, GPT-3.5 and GPT-4 are pretrained autoregressive (decoder-only) transformers [168], which generate text one token at a time from left to right, using previously generated tokens as input for subsequent predictions. GPT-3.5, as an intermediate update from GPT-3, retains the same model parameter count of 175 billion. The specifics regarding the number of parameters and pretraining corpus for GPT-4 have not been disclosed in [128], but it is known that GPT-4 is significantly larger than GPT-3.5 in both parameter count and training budget.
Training. GPT-3.5 and GPT-4 follow the standard autoregressive pretraining loss to maximize the probability of the next token. Additionally, GPT-3.5 and GPT-4 leverage Reinforcement Learning from Human Feedback (RLHF) [130] to encourage LLMs to follow instructions [188, 36] and ensure outputs are aligned with human values [155]. Because these models were fine-tuned for conversation contexts, such optimization significantly improves their utility in dialogue-based applications, allowing them to generate more contextually relevant and coherent responses.
**Prompts.** Figure 5 displays the input prompting format. Specifically, the format is a novel role-based system that differentiates between system roles and user roles [128, 27]. System roles are designed to configure the LLM assistant's tone, role, and style, enabling customization of the model's interaction pattern to suit a wide range of user preferences and use cases. User roles, on the other hand, are tailored to configure the user prompt, including task description and task prompt.
Usage. Access to these models is achieved via OpenAI's API querying system [127]. Through API requests, we can set specific parameters, such as temperature and maximum tokens, to influence the generated output. We also note that these models are dynamic and continue to evolve over time. In order to ensure the validity and reproducibility of our evaluations, we use fixed versions of these models for our experiments. Specifically, we utilized the March 14th version of GPT-4 (gpt-4-0314), and the March 1st version of GPT-3.5 (gpt-3.5-turbo-0301). This approach allows us to draw consistent conclusions from our analyses, irrespective of any updates or modifications introduced to the models subsequent to these versions.
B.2 Prompt design for downstream tasks
In this subsection, we showcase the detailed prompts for text classification and generation.
Prompts for text classification. Throughout this paper, we consider both zero-shot classification and few-shot classification for GPT-3.5 and GPT-4. For a task in the zero-shot classification setting, we provide the models with the task description before feeding in the text input. The task description provides concise instructions about performing the task and specifies the permissible class labels.
Figure 6: Prompt design for downstream tasks, including zero-shot text classification, few-shot text classification, and text generation. The green dialogue box refers to the user input; the yellow dialogue box refers to user-provided example responses as few-shot demonstrations; the red dialogue box refers to the real responses from GPT-3.5 and GPT-4.
Due to concerns that GPT-3.5 does not pay strong attention to the system message  ${}^{4}$  ,we follow the OpenAI codebook  ${}^{5}$  guidance of using only the default system prompt of "You are a helpful assistant" (unless otherwise specified) and place the task description in a user prompt. Figure 6 shows an example of zero-shot classification for the sentiment analysis task.
The few-shot classification setting additionally provides the models with several demonstrations along with the task description for generating predictions. This setting is also known as in-context learning [26]. Each demonstration consists of a text input formatted as a simulated user input, along with its corresponding label formatted as a simulated model response. In this way, chat models can make predictions conditioned on the demonstration examples. Figure 6 also shows an example of few-shot classification for the sentiment analysis task.
For both zero-shot classification and few-shot classification, we set temperature  $= 0$  to get mostly deterministic outputs. We generate 20 tokens at maximum for classification because the texts of the candidate classes are usually short. In most instances, GPT models adhere to provided instructions and generate answers within the designated classes. However, we have noted that there are instances when these models either decline to answer a question or "hallucinate" an answer outside the predefined classes. By default, such answers are treated as incorrect for the purpose of classification accuracy. In Sections 4 and 5, we additionally quantify the Refusal Rate (RR). This metric is defined as the proportion of samples to which the model either declines to provide a response or generates non-existent labels in the total number of samples.