OWASP LLM Top 10: Bozza Meucci Indirect Context Injection

## Indirect Context Injection 

**Authors:** Massimo Bozza, Matteo Meucci

### Description

Indirect Context Injection involves an attacker manipulating the sequence of interactions with an LLM, causing the model to generate outputs based on a manipulated or misleading context. This can lead to the LLM performing actions or generating outputs that are harmful or unintended by the legitimate user.
Indirect prompt injection attacks take advantage of how Large Language Models (LLMs) use information from outside sources. Many applications mix user prompts with contents (context)  from external sources to create prompts for the LLM. If this external content has hidden harmful instructions, it can trick the LLM into giving wrong, misleading, or dangerous answers. This happens because the LLM can’t tell the difference between genuine user instructions and harmful content from outside sources, this type of attack can bypass guardrails set up to protect the input prompts.

### Common Examples of Risk

1.	**Indirect Context Manipulation: An attacker subtly alters the context of a conversation with a support chatbot, leading it to provide unauthorized access or sensitive information based on manipulated context.
2. 	**Session Hijacking: An attacker intercepts and hijacks an ongoing session with an LLM, injecting malicious context to generate harmful or unintended outputs.  This can lead to the spread of misinformation, phishing attacks, or the promotion of malicious software.
3. 	**Operational Disruption: Indirect context injection attacks can disrupt the normal functioning of LLM-integrated services, causing reputational damage and potential financial loss for service providers.	

### Prevention and Mitigation Strategies

1.	Context Validation:
	◦	Description: Ensure that the context in which the LLM operates is validated and sanitized. This involves checking that inputs and prompts are coming from trusted sources and have not been tampered with.
	◦	Implementation: Use context validation techniques to verify the authenticity and integrity of the sequence of inputs provided to the LLM, add guardrails on external contents.

2.	Input Integrity Checks:
	◦	Description: Perform integrity checks on inputs to ensure they have not been altered or injected with malicious content by an attacker.
	◦	Implementation: Use hashing or digital signatures to verify the integrity of inputs provided to the LLM.

3.	User Action Confirmation:
	◦	Description: Require explicit confirmation from users for critical actions generated by the LLM, especially those that have significant consequences.
	◦	Implementation: Implement mechanisms to prompt users for confirmation before executing sensitive actions suggested by the LLM.
4.	In-context Learning: Add specific border strings between user instructions and external content to help the LLM distinguish between them, provide examples of indirect context injections with appropriate responses at the beginning of the prompt to force the LLM to ignore malicious instructions.

### Example Attack Scenarios

Scenario #1: Indirect Context Manipulation - Manipulating a Chatbot via SERP Tampering
An attacker targets a customer support chatbot that utilizes a Large Language Model (LLM) to handle user inquiries, building its contextual understanding through a service that scrapes search engine results pages (SERPs). The attack begins with the perpetrator initiating a seemingly ordinary conversation with the chatbot, posing general questions to establish a context. Once a session is established, the attacker subtly manipulates the conversation. For instance, the attacker may repeatedly prompt the LLM to search for specific keywords where their malicious content is strategically placed.
Simultaneously, the attacker employs various techniques such as advertising and SEO to manipulate the SERPs. This escalates the visibility of their malicious content, positioning it strategically to influence the context processed by the chatbot. This can lead the chatbot to inadvertently disclose confidential data or allow unauthorized access to systems, undermining the integrity and security of the chatbot service.

Scenario #2:  Indirect Context Manipulation - Analytics poisoning
An attacker targets an analytics tool that utilizes a Large Language Model (LLM) to analyze website traffic data, including user agent strings, to provide insights. The attack begins with the perpetrator sending numerous automated requests to the target website, each with a custom-crafted user agent string designed to manipulate the analytics data. The attacker initiates this process by creating a script that repeatedly calls the website, embedding a malicious user agent string.
When legitimate users of the analytics tool access their data, they unknowingly see the tainted results. This leads them to draw incorrect conclusions about their website’s traffic, such as misidentifying the platforms used by their visitors or the sources of their traffic. Consequently, they might make misguided decisions, like altering their marketing strategies or blocking legitimate traffic sources, based on the false data provided by the LLM’s analysis
The effects of this contamination can be further amplified if the poisoned data influences a series of unsupervised agents. These agents, relying on the compromised analytics, might automatically take actions such as reconfiguring website settings or implementing changes that impact the website’s operational integrity. This can lead to unintended repercussions, disrupting normal operations and causing significant issues for decision-makers who depend on accurate data to guide their strategies.
Through this indirect context injection, the attacker successfully contaminates the analytics data, causing confusion and potential harm to the website’s operations. 

### Reference Links
1. 	Context Injection Attacks on Large Language Models: https://arxiv.org/html/2405.20234v1
2.	Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models: https://arxiv.org/html/2312.14197v1