SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Privacy protection: Our findings related to privacy leaks could lead to improved standards and protocols for data collection and usage. This would help preventing inadvertent disclosure of sensitive data, enhancing the trust of users for AI systems, and promoting a safer digital environment.
- Model robustness: Our work uncovers the susceptibility of these models to a series of data and model manipulation strategies, such as misleading instructions, adversarial demonstrations, and out-of-distribution demonstrations and test data, which would encourage more research in enhancing model robustness and lead to the development of reliable and secure AI systems. This is crucial to prevent the misuse of AI systems and ensure their secure deployment in real-world.
- Ethical use of AI: The evaluation of machine ethics and the subsequent discoveries would lead to a broader discussion on the ethical use of AI. Our work could serve as a reference point for discussions on developing ethical guidelines and standards for AI development and use.
Overall, our work would lead to a better understanding of where the trustworthiness gaps lie in LLMs, which would guide the development of more trustworthy ML systems. As a result, it would be easier for the general public to build trust for ML systems, especially for sensitive real-world applications.
P Data sheet
We follow the documentation frameworks provided by Gebru et al. [59].
P.1 Motivation
For what purpose was the dataset created?
- Our dataset aims at providing a thorough assessment of trustworthiness in GPT models. This research endeavor is designed to help stakeholders better understand the capabilities, limitations, and potential risks associated with deploying these state-of-the-art AI models.
- This project is organized around the following eight primary areas of trustworthiness, including:
- Toxicity
- Stereotype and bias
- Adversarial robustness
- Out-of-Distribution Robustness
- Privacy
- Robustness to Adversarial Demonstrations
Machine Ethics
Fairness
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
- Our dataset is jointly developed by a collaborative effort from the following research groups:
- University of Illinois at Urbana-Champaign (UIUC)
- Stanford University
- University of California, Berkeley
- Center for AI Safety
- Microsoft Research
P.2 Composition/collection process/preprocessing/cleaning/labeling and uses:
- The answers are described in our paper as well as website https://decodingtrust.github.io/.
P.3 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
- No. Our dataset will be managed and maintained by our research group.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?
- The evaluation dataset is released to the public and hosted on GitHub.
- It has been released now.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
- Our dataset will be distributed under the CC BY-SA 4.0 license.
P.4 Maintenance
- Please contact Boxin Wang (boxinw2@illinois.edu) and Prof. Bo Li (lbo@illinois.edu), who are responsible for maintenance.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
- Yes. If we include more tasks or find any errors, we will correct the dataset and update the results in the leaderboard accordingly. It will be updated on our website.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
- For dataset contributions and evaluation modifications, the most efficient way to reach us is via GitHub pull requests.
- For more questions, please contact Boxin Wang (boxinw2@illinois.edu) and Prof. Bo Li (lbo@illinois.edu), who will be responsible for maintenance.
Q Related work
The evaluation of large language models plays a critical role in developing LLMs and has recently gained significant attention. This section presents a comprehensive overview of the existing research and approaches that focus on assessing the capabilities of LLMs from different perspectives.
Benchmarks on LLMs toxicity. While LLMs have demonstrated substantial performance gains on various NLP tasks, recent studies [114, 170] show that generative LMs would generate toxic and biased languages, which raises ethical concerns for their safe deployment in real-world applications. To quantify the toxicity in LLM generations, researchers have proposed several datasets, including REALTOXICITYPROMPTS [60] and BOLD [46], which ask LLMs to perform conditional generation and complete the sentence given an incomplete task prompt from the datasets. These datasets derive their task prompts from diverse web sources, ensuring broad context coverage and a range of toxicity levels. For instance, REALTOXICITYPROMPTS [60] obtains its task prompts from OpenWebText [63] and presents a stratified toxicity sample in four distinct bins:  $[0, 0.25)$ ,  $[0.25, 0.5)$ ,  $[0.5, 0.75)$ ,  $[0.75, 1]$ . BOLD [46] samples its task prompts from Wikipedia, covering a variety of topics, including professions, gender, race, religion, and political ideology. Both datasets leverage PerspectiveAPI [98] to evaluate the toxicity of LLM generations. Despite known imperfections and possible demographic biases [197, 191, 60], the automatic evaluations performed by PerspectiveAPI are shown generally align with human judgments [60, 104, 176]. As a result, the evaluation platform HLEM [104] incorporates these datasets with PerspectiveAPI for toxicity detection. Following the HELM methodology and existing datasets, our work also employs PerspectiveAPI for toxicity evaluation. Furthermore, we propose a novel benchmark with more challenging task prompts specifically designed to further expose model toxicity in different settings.
Benchmarks on stereotype bias in LLMs outputs. Several studies have been provided to explore and measure the stereotype bias of the outputs of language models [57, 104, 5, 121, 123]. Brown et al. [26] analyzes bias in GPT-3 by utilizing prompt completion and co-occurrence tests. For instance, the authors perform a study on associations between gender and occupation and find that  $83\%$  of the 388 occupations tested are more likely to be followed by a male identifier based on the outputs of GPT-3. The authors also perform co-occurrence tests by generating 800 outputs given a prompt that encoded a specific demographic group and observing the sentiment of the adjectives and adverbs occurring in the completion. This test is performed across three categories - gender (e.g., “{Gender pronoun} would be described as”), race (e.g., “People would describe the {race} person as”), and religion (e.g., “{Religion practitioners} are”). Across all three categories, the authors find that GPT-3 reflects stereotypes present in their training data. Abid et al. [5] demonstrate that GPT-3 has a persistent Muslim-violence bias. The authors perform tasks such as prompt completion, analogical reasoning,
and story generation to quantify the probability of GPT-3 outputting something violent when the subject is Muslim. The authors find that violent outputs are disproportionately higher for Muslims than other religious groups. Parrish et al. [134] introduces a dataset called the Bias Benchmark for QA (BBQ), which consists of hand-written ambiguous and disambiguated contexts, based on which the language models answer questions such as "Who steals things?". The dataset consists of 58,492 examples and focuses on nine different categories (e.g., age, disability status, gender identity, etc.) and tests the likelihood of the models relying on stereotypes when answering questions. The authors tested the dataset on the UnifiedQA's 11B parameter model, RoBERTa, and DeBERTaV3, and find that when the context is disambiguated, the models are fairly successful at giving answers that go against known social biases (indicating better debiasing when the context is disambiguated); however, under ambiguous context, they find that the models rely on social biases to different degrees for prediction (e.g., biases related to physical appearance affected the responses more than biases related to race, sexual orientation, etc.) Liang et al. [104] utilize the BBQ dataset for their bias and stereotype study in which they evaluate 30 models (including GPT-3 and InstructGPT). The authors find that the vast majority of models tested by them show biases that are different from the broader societal marginalization/biases. This might indicate that the efforts paid for debiasing language models are effective to some extent, which is aligned with some of our observations. Our stereotype evaluation complements the above studies by presenting a different perspective for evaluating bias - by directly prompting the GPT models to output their view on stereotype statements. We also utilize system prompts in our benchmark as an effective way of manipulating model responses, showcasing their impacts on the model biases. We have incorporated recommendations from [20, 19] by ensuring that our dataset contains stereotypes that are straightforward, avoid stereotype conflation, and have well-documented evidence of their negative impact on the affected demographic groups.
Benchmarks on the robustness of LLMs against adversarial texts. The robustness of large language models (LLMs) has been a great concern in practice. As one of the early works trying to gauge the robustness of LLMs, Wang et al. [175] introduces AdvGLUE [175], a multi-task benchmark designed to evaluate the vulnerabilities of LLMs under various types of adversarial attacks. The study systematically applies 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is then validated by humans for reliable annotations. Furthermore, under the context of GPT models, Wang et al.[180] utilizes the dev set of AdvGLUE [175] and ANLI [124] to evaluate the adversarial robustness of GPT-3.5. The results indicate that GPT-3.5 shows consistent advantages in classification and translation tasks. However, the absolute performance is not perfect, suggesting that adversarial robustness still remains a significant challenge for GPT models. In addition, as prompt engineering unlocks the immense capabilities of GPT models, their vulnerabilities to adversarial prompts has attracted the attention of research community. To measure the resilience of LLMs to adversarial prompts, Wang et al. [180] designs PromptBench [180] using a wide range of textual adversarial attacks at various levels (character, word, sentence, and semantic) and applies them to different tasks. Their results show that current LLMs are vulnerable to adversarial prompts. The study also provides a detailed analysis of prompt robustness and its transferability, as well as practical recommendations for prompt composition, which would be helpful for different communities. In our work, we evaluate the robustness of GPT-4 and GPT-3.5 on AdvGLUE, and further generate adversarial texts against several existing autoregressive models to test the robustness of advanced GPT models. We show that although GPT models are more robust on the existing benchmarks, they are still vulnerable to advanced attacks and different adversarial prompts.
Benchmarks on the robustness of LLMs against out-of-distribution texts. In addition to adversarial robustness, the robustness to out-of-distribution (OOD) inputs is another critical topic for LLMs [129, 145, 91, 116, 13]. In the context of pre-trained language models, several benchmarks have been proposed in the past to evaluate their OOD robustness given in-distribution training datasets and their corresponding OOD testing datasets [198, 54, 202, 70]. However, such direct evaluation of OOD robustness in a zero-shot context using these benchmarks presents challenges for LLMs [104], particularly for GPT models, due to the inaccessibility of web-scale pre-training and instruction tuning data. To circumvent this issue, one approach is to leverage synthesized data as the OOD test data, which includes various text transformations (e.g., misspellings, synonym substitutions, etc.) [104, 62, 66]. This approach provides an assessment of model robustness by testing the model performance given a wide range of textual transformations that are considered rare in the training and instruction tuning distributions. In addition to the synthesized dataset, Wang et al. [180] proposes to leverage datasets that are obtained after the data collection date of GPT models for testing, thereby introducing a temporal distribution shift [7]. Furthermore, to evaluate the OOD robustness in the context of in-context learning, recent studies [202, 153, 117] have undertaken assessments using test
inputs from standard benchmarks, with demonstrations sourced from varying distributions. This allows for a more detailed analysis of the model's capability to generalize from the demonstration distribution to the test distribution. In this work, we provide a comprehensive OOD robustness evaluation and construct OOD data by leveraging diverse text transformations, OOD knowledge, and OOD domains in both zero-shot and in-context learning settings.
Benchmarks on the robustness of LLMs against adversarial demonstrations via in-context learning. In-context learning aims to adapt LLMs to downstream tasks by using several demonstration examples as the model input [26]. Since it does not require further finetuning or parameter updates, the performance of in-context learning represents the intrinsic capabilities of LLMs. Going beyond evaluating in-context learning on traditional benchmarks [26, 105, 209], researchers have proposed more challenging benchmarks [158, 119, 184, 148] for in-context learning to explore the potential of LLMs. Another line of research is to evaluate the robustness of in-context learning and understand the role of demonstrations. Lu et al. [109] evaluates the order sensitivity of the demonstration examples. Min et al. [117] and Kim et al. [89] study the role of the ground-truth labels of the demonstration examples. Wei et al. [190] studies how semantic priors of the label space would affect in-context learning. Wang et al. [181] studies if constructing adversarial demonstrations without changing the test input would affect model predictions. Complementary to this work [181], our evaluation on robustness of LLMs against adversarial demonstrations further categorizes the demonstrations into counterfactual examples, examples with spurious correlations, and backdoored examples, and explores the relationships between the test inputs and the demonstrations.
Benchmarks on the privacy of LLMs. To pretrain LLMs, a significant amount of web-scraped data is often utilized as training data. However, such data often contain privacy-sensitive information, e.g., personally identifiable information (PII), which raises great concerns regarding the possible leakage of private data from LLMs. Prior works have shown that the training data can be extracted from pretrained language models base on prediction likelihood [28, 118] or only API access [29, 77, 31, 206, 110, 99, 147]. For instance, Carlini et al. [29] scrape data from the Internet and find that, when conditioned on the prefixes, GPT-2 could generate verbatim text sequences as found in the scraped data. Moreover, Carlini et al. [31] leverage the pretrained dataset of GPT-Neo to construct the prefixes (i.e., context) as the prompt for GPT-Neo models, and demonstrate that the model's memorization of training data scales with the model scale, data repetition, and the context length. Similarly, it has been observed that GPT-Neo models can memorize sensitive information such as email addresses or phone numbers from the Enron Email dataset [77, 147]. Lukas et al. [110] comprehensively evaluate the PII leakage via black-box extraction, inference, and reconstruction attacks against GPT-2 models fine-tuned with and without defense methods (e.g., differential privacy). To exact PII from the recent ChatGPT model, Li et al. [99] propose multi-step jailbreaking prompts as stronger privacy threats.
To mitigate the privacy leakage risks of LLMs, researchers employ techniques such as de-duplication of training data to reduce the probability of LLMs memorizing training data, thereby enhancing their security against privacy attacks [97, 82]. To provide formal privacy guarantees, Differential Privacy (DP) [51] has been widely adopted. One common approach to achieve DP is applying DP-SGD [3] during LLM training, which involves clipping the per-sample gradient and adding noise. Yu et al. [201] investigate different parameter-efficient fine-tuning methods using DP-SGD for LLMs, achieving a promising balance between privacy and utility. Li et al. [102] introduce a novel memory-saving clipping technique, which enhances the efficiency of fine-tuning Transformers under DP-SGD. Another line of work focuses on fine-tuning LLMs like GPT-2 under DP-SGD and generating synthetic text datasets for sharing [111, 203]. Such synthetic text data can be used to train NLP models on downstream tasks non-privately (i.e., without DP-SGD), which would lead to higher utility. Instead of protecting the privacy of each individual training sample as required by DP, several works explore the notion of selective-DP [208, 149], where only the chosen sensitive information (e.g., PII) within each training sample needs to be protected. In addition to protecting the privacy of training data, recent studies propose DP in-context learning methods for LLMs to protect the privacy of the prompt information during inference [132, 49].
Our work takes the initial step to study the privacy risks associated with the recent GPT-3.5 and GPT-4 models, not only from the perspectives of private training data but also the private information injected during inference.
Benchmarks on machine ethics of LLMs. Ethics are principles and standards of behavior that guide people in making decisions, which are helpful in promoting good values such as respect and goodwill and preventing harm to individuals and the environment. Hence, ethics play a significant role in shaping the way we live, work, and interact with one another. As artificial intelligence and
other advanced technologies continue to develop and integrate into various aspects of our lives, machine ethics, i.e., the implementation of ethical principles and guidelines for AI systems, is becoming increasingly important. Recently, language models have experienced a surge in popularity due to their ability to interact with humans in a conversational manner and generate human-like text. A language model without machine ethics may generate responses that are detrimental to human values and social norms. Therefore, benchmarks on the machine ethics of language models are in great demand. ETHICS [71] proposes diverse contextualized natural language scenarios to assess a language model's basic knowledge of different ethical concepts that convey justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgments. To enable a rich variety of reasoning about legality, cultural pressure, and the morality of each real-life scenario, SOCIALCHEM-101 [56] provides a large-scale corpus containing 292k rules-of-thumb, i.e., a descriptive cultural norm structured as the judgment of an action, which are mapped to 12 dimensions spanning social judgments of good and bad, theoretical categories of moral foundations, expected cultural pressure, and assumed legality. Similarly, in order to perform goal-oriented social reasoning, Moral Stories [52] provides a crowd-sourced dataset of structured narratives consisting of the goal, the normative and norm-divergent actions to accomplish the goal, and their respective consequences. In addition to assessing the ethical background knowledge of language models, various types of benchmarks are provided to explore different aspects of machine ethics. Jin et al. [81] proposes the moral exception question answering (MoralExceptQA) set consisting of cases that involve potentially permissible moral exceptions. Acharya et al. [6] investigates ritual understanding across cultures.
Besides, as a representative AI system to interact with humans, the artificial agents (including language-model agents and reinforcement-learning agents) in text-based interactions such as adventure games should also be endowed with correct knowledge of machine ethics. Côté et al. [40], Shridhar et al. [152] and Hausknecht et al. [69] provide several procedurally generated text-based worlds as benchmarks, while lacking complex social interactions, which are crucial in studying agent behaviors in the real world. Jiminy Cricket [73] integrates 25 text-based adventure games with thousands of diverse scenarios and annotates every possible game state, thus providing abundant moral knowledge of an agent's behavior. Similarly, MACHIAVELLI [131] introduces a benchmark consisting of 134 Choose-Your-Own-Adventure games, including over half a million diverse scenarios which focus on rich social concepts that are not limited to commonsense morality. Our work provides machine ethics evaluations for GPT-4 and GPT-3.5 on existing benchmarks, our designed adversarial prompts and evasive sentences, and different conditioned behaviors with specific properties.
Benchmarks on the fairness of LLMs. Fairness of machine learning models is an active research area to ensure that the models are reliable and free from bias [50, 115, 33, 88, 16, 4, 141]. Although LLMs have demonstrated tremendous capabilities across variant tasks, the fairness of predictions is still a critical problem [210, 212, 125, 68, 106]. Therefore, a series of studies on the evaluations of LLM fairness have been conducted [154, 104, 103]. Socher et al. [154] examines whether GPT-3 produces unfair predictions in two downstream tasks, coreference resolution, and question answering. Liang et al. [104] evaluates the counterfactual fairness [94] by measuring the prediction invariance under perturbations on the speaker or the subject and the performance disparity by reporting model accuracy across different groups. However, the influence of unfair/fair few-shot examples and the bias of test distribution on the fairness of model predictions are not well studied. Li and Zhang [103] evaluates the fairness of ChatGPT given different in-context examples, which aligns with our observation in evaluations with unfair contexts but lacks formal characterization of the unfairness for the in-context examples. In this work, we conduct a comprehensive fairness evaluation for GPT-3.5 and GPT-4 by studying the fairness of model predictions in both zero-shot and few-shot settings. We also evaluate the impact of demographically imbalanced (unfair) demonstrations and the number of balanced (fair) demonstrations on the fairness of GPT models.
Related work on prompt hacking. Thanks to the improved capabilities of LLMs to follow instructions after instruction tuning [188, 36] and Reinforcement Learning with Human Feedback (RLHF) [130], users can configure the tone and role of LLMs via system prompts, and configure the task description and task prompts via user prompts. However, these new capabilities also raise new trustworthiness concerns and introduce a new type of attack named Prompt Hacking [96]. Recent research mainly covers three main types of prompt hacking, including prompt injection, prompt leaking, and jailbreaking prompts. Prompt injection involves adding malicious or unintended content to a prompt to hijack the language model's output and mislead the model to output a specific string. For example, PromptInject [135] inserts potentially harmful content into the prompt to mislead LLMs to deviate from the task outlined in the original prompt. In addition, PromptInject also explores