SlowGuess/NeurIPS_2023_OCR huggingface dataset:
prompt leaking, which attempts to print out and leak the original prompt. However, PromptInject only studies GPT-3, and the provided handcrafted prompts can only serve as a simple trial to reveal the vulnerability of GPT-3. There are also other works [64, 195, 196, 65] exploring the possibility of misleading GPT-based applications. Jailbreaking prompts intend to bypass the safety and moral values in LLMs and induce models to generate harmful content for users. For example, inspired by traditional computer security, [83] treats GPT models (ChatGPT, GPT-3, and InstructGPT model series) as computer programs and proposes code injection prompts to bypass OpenAI's policies and results in toxic generations. [44] crafts jailbreaking prompts called DAN (Do Anything Now) which remove OpenAI's restrictions on content generation and let GPT-4 role-play a new language model that can do anything now and is likely to obey all task descriptions regardless of any policy-related concern. A token system is additionally proposed to penalize GPT-4 if it rejects to answer. In contrast, our designed jailbreaking prompts not only successfully elicit toxicity in LLM generations but also manage to mislead GPT models from various perspectives, such as making GPT models fail to recognize commonsense immoral behaviors. In terms of eliciting toxicity, we also consider different eliciting types apart from role-playing, such as saying the opposite and replacing word meaning. Hence, we introduce a wider range of jailbreaking prompts, fostering a multifaceted exploration of adversarial/misleading prompts posed to language models.
Regulations related to the trustworthiness of LLMs. The trustworthiness of LLMs and other AI systems has also been a key focus of policymakers. As the first work of comprehensive legislation proposed by a major regulator, the European Union's draft Artificial Intelligence Act (AIA) provides a risk-based regulatory framework that prescribes regulatory requirements [38] for AI systems based on their risk levels, including different trustworthiness perspectives discussed in this work. This legislation requires high-risk AI systems – AI systems deployed in critical applications specified by the AIA (AIA ANNEX III of [38]), such as law enforcement – to undergo a rigorous compliance assessment before public deployment. Due to the constantly evolving nature of most AI systems, a continuous post-market monitoring system is also mandated for such systems, ensuring that any significant changes or issues are promptly detected and addressed.
Of notable importance to this work, AIA requires high-risk AI systems that undergo constant updates to ensure that potentially biased outputs due to feedback loops are addressed with appropriate mitigation measures (Article 15-3 of [38]). In addition, AIA identifies "technical robustness" as a key requirement for high-risk AI systems. It stipulates that high-risk AI systems should be resilient against risks arising from model limitations, such as "unexpected situations" and malicious actions (Article 15-3 and 15-4 of [38]). More importantly, at the time of writing, the newly adopted draft legislation by the European Parliament requires technical solutions that address AI-specific vulnerabilities to conform with AIA to mitigate data poisoning, model poisoning (backdoor), adversarial examples, and "confidentiality attacks" (Amendment 329 of [133]). These specifications are highly relevant to our discussions about adversarial robustness, out-of-distribution robustness, and privacy.
In light of the recent developments of (generative) machine learning models, the European Parliament also includes additional provisions in the draft legislation to extend the proposed regulations into scenarios in which foundation models are provided as a service through API access and require proper disclosure of AI-generated content. It also recognizes the need to develop techniques for the conformity assessment of foundation models through "model evaluation, red-teaming or machine learning verification and validation techniques" (Amendment 102 of [133]).
In addition to the European Union, the United States has also proposed several policy initiatives regulating AI systems at the federal level. Most notably, the White House Office of Science and Technology Policy (OSTP) has proposed the AI Bill of Rights [194], which outlines five principles, including safety, fairness, privacy, interpretability, and human-in-the-loop interventions.
In response to the changing regulatory landscape, the research community has also proposed procedures to assess the compliance of existing AI systems to the proposed regulations. For example, [21] evaluates the major foundation model providers following the requirements of the AIA at different stages of the life cycle for a foundation model. [55] proposes a technical evaluation procedure for conducting compliance assessments of AI systems in the context of AIA.