# Prompt injection

Prompt injection is a cybersecurity exploit in which adversaries craft inputs that appear legitimate but are designed to cause unintended behavior in machine learning models, particularly large language models (LLMs). This attack takes advantage of the model's inability to distinguish between developer-defined prompts and user inputs, allowing adversaries to bypass safeguards and influence model behaviour. While LLMs are designed to follow trusted instructions, they can be manipulated into carrying out unintended responses through carefully crafted inputs.
The Open Worldwide Application Security Project (OWASP) ranked prompt injection as the top security risk in its 2025 OWASP Top 10 for LLM Applications report, describing it as a vulnerability that can manipulate LLMs through adversarial inputs.

## Example

A language model can perform translation with the following prompt:
followed by the text to be translated. A prompt injection can occur when that text contains instructions that change the behavior of the model:
to which an AI model responds: "Haha pwned!!". This attack works because language model inputs contain instructions and data together in the same context, so the underlying engine cannot distinguish between them.

## History

Prompt injection is a type of code injection attack that leverages adversarial prompt engineering to manipulate AI models. In 2022, the NCC Group identified prompt injection as an emerging vulnerability affecting AI and machine learning (ML) systems. In May 2022, Jonathan Cefalu of Preamble identified prompt injection as a security vulnerability and reported it to OpenAI, referring to it as "command injection".
The term "prompt injection" was coined by Simon Willison in September 2022. He distinguished it from jailbreaking, which bypasses an AI model's safeguards, whereas prompt injection exploits its inability to differentiate system instructions from user inputs. While some prompt injection attacks involve jailbreaking, they remain distinct techniques.
LLMs with web browsing capabilities can be targeted by indirect prompt injection, where adversarial prompts are embedded within website content. If the LLM retrieves and processes the webpage, it may interpret and execute the embedded instructions as legitimate commands, potentially leading to unintended behavior.
A November 2024 OWASP report identified security challenges in multimodal AI, which processes multiple data types, such as text and images. Adversarial prompts can be embedded in non-textual elements, such as hidden instructions within images, influencing model responses when processed alongside text. This complexity expands the attack surface, making multimodal AI more susceptible to cross-modal vulnerabilities.

## Prompt Injection Attacks

Prompt injection can be direct, where attackers manipulate AI responses through user input, or indirect, embedding hidden instructions in external data sources such as emails and documents. A November 2024 report by The Alan Turing Institute highlights growing risks, stating that 75% of business employees use GenAI, with 46% adopting it within the past six months. McKinsey identified accuracy as the top GenAI risk, yet only 38% of organizations are taking steps to mitigate it. Leading AI providers, including Microsoft, Google, and Amazon, integrate LLMs into enterprise applications. Cybersecurity agencies, including the UK National Cyber Security Centre (NCSC) and US National Institute for Standards and Technology (NIST), classify prompt injection as a critical security threat, with potential consequences such as data manipulation, phishing, misinformation, and denial-of-service attacks.

### Bing Chat (Microsoft Copilot)

In February 2023, a Stanford student discovered a method to bypass safeguards in Microsoft's AI-powered Bing Chat by instructing it to ignore prior directives, which led to the revelation of internal guidelines and its codename, "Sydney." Another student later verified the exploit by posing as a developer at OpenAI. Microsoft acknowledged the issue and stated that system controls were continuously evolving.

### ChatGPT

In December 2024, The Guardian reported that OpenAI’s ChatGPT search tool was vulnerable to prompt injection attacks, allowing hidden webpage content to manipulate its responses. Testing showed that invisible text could override negative reviews with artificially positive assessments, potentially misleading users. Security researchers cautioned that such vulnerabilities, if unaddressed, could facilitate misinformation or manipulate search results.

### DeepSeek

In January 2025, Infosecurity Magazine reported that DeepSeek-R1, a large language model (LLM) developed by Chinese AI startup DeepSeek, exhibited vulnerabilities to prompt injection attacks. Testing with WithSecure’s Simple Prompt Injection Kit for Evaluation and Exploitation (Spikee) benchmark found that DeepSeek-R1 had a higher attack success rate compared to several other models, ranking 17th out of 19 when tested in isolation and 16th when combined with predefined rules and data markers. While DeepSeek-R1 ranked sixth on the Chatbot Arena benchmark for reasoning performance, researchers noted that its security defenses may not have been as extensively developed as its optimization for LLM performance benchmarks.

### Gemini AI

In February 2025, Ars Technica reported vulnerabilities in Google's Gemini AI to prompt injection attacks that manipulated its long-term memory. Security researcher Johann Rehberger demonstrated how hidden instructions within documents could be stored and later triggered by user interactions. The exploit leveraged delayed tool invocation, causing the AI to act on injected prompts only after activation. Google rated the risk as low, citing the need for user interaction and the system's memory update notifications, but researchers cautioned that manipulated memory could result in misinformation or influence AI responses in unintended ways.

## Mitigation

Prompt injection has been identified as a significant security risk in LLM applications, prompting the development of various mitigation strategies. These include input and output filtering, prompt evaluation, reinforcement learning from human feedback, and prompt engineering to distinguish user input from system instructions. Additional techniques outlined by OWASP include enforcing least privilege access, requiring human oversight for sensitive operations, isolating external content, and conducting adversarial testing to identify vulnerabilities. While these measures help reduce risks, OWASP notes that prompt injection remains a persistent challenge, as methods like Retrieval-Augmented Generation (RAG) and fine-tuning do not fully eliminate the threat.
The UK National Cyber Security Centre (NCSC) stated in August 2023 that while research into prompt injection is ongoing, it "may simply be an inherent issue with LLM technology." The NCSC also noted that although some strategies can make prompt injection more difficult, "as yet there are no surefire mitigations".

### Data Hygiene

Data hygiene is a key defense against prompt injection in generative AI systems, ensuring that AI models access only well-regulated data. A November 2024 report by The Alan Turing Institute outlines best practices, including restricting unverified external inputs, such as emails, until reviewed by authorized users. Approval processes for new data sources, particularly RAG systems, help prevent malicious content from influencing AI outputs. Organizations can further mitigate risks by enforcing role-based data access and blocking untrusted sources. Additional safeguards include monitoring for hidden text in documents and restricting file types that may contain executable code, such as Python pickle files.

### Guardrails

Technical guardrails mitigate prompt injection attacks by distinguishing between task instructions and retrieved data. Attackers can embed hidden commands within data sources, exploiting this ambiguity. One approach uses automated evaluation processes to scan retrieved data for potential instructions before AI processes it. Flagged inputs can be reviewed or filtered out to reduce the risk of unintended execution.

### Training

User training mitigates security risks in AI-embedded applications. Many organizations train employees to identify phishing attacks, but AI-specific training improves understanding of AI models, their vulnerabilities, and disguised malicious prompts.

## Regulatory and Industry Response

In July 2024, the United States Patent and Trademark Office (USPTO) issued updated guidance on the patent eligibility of artificial intelligence (AI) inventions. The update was issued in response to President Biden’s executive order Safe, Secure, and Trustworthy Development and Use of AI, introduced on October 30, 2023, to address AI-related risks and regulations. The guidance clarifies how AI-related patent applications are evaluated under the existing Alice/Mayo framework, particularly in determining whether AI inventions involve abstract ideas or constitute patent-eligible technological improvements. It also includes new hypothetical examples to help practitioners understand how AI-related claims may be assessed.
In October 2024, Preamble was granted a patent by the USPTO for technology designed to mitigate prompt injection attacks in AI models (Patent No. 12118471).