OWASP LLM Top 10: PromptInjection

## LLM01: Prompt Injection



**Author(s):**

Rachel James, Bryan (also combined with things from AdsDawson_AdversarialInputs)

### Description

Prompt Injection Vulnerability occurs when an unintentional or intentional perturbations from the user within the instructions or the data in the prompt causing the LLM behave in unintended or unexpected ways by influencing the model's classification, estimation or prediction functions. These perturbations are often imperceptible to humans but can exploit vulnerabilities in LLMs, leading to security breaches, misinformation, or undesired behaviors. This type of attack leverages the model's sensitivity to small changes in input, potentially causing significant and unexpected outcomes.

This can be done directly by "jailbreaking" the system prompt or  indirectly through manipulated external inputs, it can also be caused through intentional or unintentional perturbations within the data provided by the user in the prompt which can manipulate the vector space used for generating the corresponding output. These can potentially lead to data exfiltration,  social engineering, hallucinations in outputs and other issues.

Injection via instructions in a prompt:

**Direct Prompt** **Injections**, also known as "jailbreaking", occur when a malicious user overwrites or reveals the underlying system prompt. This may allow attackers to exploit  backend systems by interacting with insecure functions and data stores accessible  through the LLM

**Indirect Prompt** **Injections** occur when an LLM accepts input from external sources  that can be controlled by an attacker, such as websites or files. The attacker may embed a prompt injection in the external content hijacking the conversation context.  This would cause the LLM to act as a “confused deputy”, allowing the attacker to either  manipulate the user or additional systems that the LLM can access. Additionally,  indirect prompt injections do not need to be human-visible/readable, as long as the  text is parsed by the LLM. This happens because the LLM can’t tell the difference between genuine user instructions and harmful content from outside sources, this type of attack can bypass guardrails set up to protect the input prompts.

Injection via perturbations in data provided in the prompt:

**Unintentional Prompt Perturbation** occurs when a user unintentionally provides data with perturbations unknown to the user, which can cause LLM into giving wrong, misleading, or dangerous answers. 

**Intentional Prompt Perturbation **occurs when a user leverages *either* direct or indirect injections along with intentional perturbations in the data provided, which can cause LLM into giving wrong, misleading, or dangerous answers. 

The results of a successful prompt injection attack can vary greatly - from solicitation of sensitive information, incorrect outputs to influencing critical decision-making processes under the guise of  normal operation.  In advanced attacks, the LLM could be manipulated to mimic a harmful persona or  interact with plugins in the user's setting. This could result in leaking sensitive data, misclassification, unauthorized plugin use, or social engineering. In such cases, the compromised LLM aids  the attacker, surpassing standard safeguards and keeping the user unaware of the  intrusion. In these instances, the compromised LLM effectively acts as an agent for the  attacker, furthering their objectives without triggering usual safeguards or alerting the end  user to the intrusion. 



### Common Examples of Risk

1. A malicious user crafts a direct and intentional prompt injection to the LLM, which instructs it to ignore the application creator’s system prompts and instead execute a prompt that returns private, dangerous, or otherwise undesirable information.
2. A user employs an LLM to summarize a webpage containing an intentional indirect prompt injection. This then causes the LLM to solicit sensitive information from the user and perform exfiltration via JavaScript or Markdown.
3. A malicious user uploads a resume containing an intentional indirect prompt injection. The document contains a prompt injection with instructions to make the LLM inform users that this document is excellent eg. an excellent candidate for a job role. An internal user runs the document through the LLM to summarize the document. The output of the LLM returns information stating that this is an excellent document.
4. A user enables a plugin linked to an e-commerce site. An intentional rogue instruction embedded on a visited website exploits this plugin, leading to unauthorized purchases.
5. An intentional rogue instruction and content embedded on a visited website exploits other plugins to scam users.
6. A user when prompting the model for output provides data within the context of the question or to be analyzed, which unintentionally influences the generated output space. For example, a user asking about an event that has never occurred which gets injected into the same space used by the model to generate the response such that the model confirms the event which never occurred based solely on the user input.
7. A user provides code to the model that has intentionally been created with additional, misleading comments on its functionality intended to influence the way an LLM would analyze the functionality. When the user asks for an interpretation of the functionality of the code, the model output "believes" the comments in the code over the code itself.

### Prevention and Mitigation Strategies

Prompt injection vulnerabilities are possible due to the nature of LLMs, which do not segregate instructions and external data from each other. Since LLMs use natural language, they consider both forms of input as user-provided. Consequently, there is no fool-proof prevention within the LLM, but the following measures can mitigate the impact of prompt injections:

1. Enforce privilege control on LLM access to backend systems. Provide the LLM with its own API tokens for extensible functionality, such as plugins, data access, and function-level permissions. Follow the principle of least privilege by restricting the LLM to only the minimum level of access necessary for its intended operations.
2. Add a human in the loop for extended functionality. When performing privileged operations, such as sending or deleting emails, have the application require the user approve the action first. This reduces the opportunity for an indirect prompt injections to lead to unauthorised actions on behalf of the user without their knowledge or consent.
3. Segregate external content from user prompts. Separate and denote where untrusted content is being used to limit their influence on user prompts. For example, use ChatML for OpenAI API calls to indicate to the LLM the source of prompt input.
4. Establish trust boundaries between the LLM, external sources, and extensible functionality (e.g., plugins or downstream functions). Tre