OWASP LLM Top 10: promptinjection

# LLM01: Prompt Injection

**Author(s):**  
Rachel James, Bryan (also combined with things from AdsDawson_AdversarialInputs)

## Dataset
- **[DeBERTa v3 Base - Prompt Injection v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2):** This dataset, hosted by Protect AI on Hugging Face, is designed to train and evaluate language models for robustness against prompt injection attacks. It includes a variety of prompts specifically curated to test for vulnerabilities related to both direct and indirect prompt injections. Researchers can utilize this dataset to enhance the security and robustness of large language models.

## Research Papers and Relevant Research Blogs
1. **Research Paper:** [Adversarial Attacks on Machine Learning Models with Multiple Oracles](https://www.ijcai.org/proceedings/2019/0925.pdf)
   - _Authors:_  K Wang
   - _Abstract:_ Discusses methods for adversarial attacks using multiple oracles.

2. **Research Blog:** [The ELI5 Guide to Prompt Injection](https://www.lakera.ai/blog/guide-to-prompt-injection)
   - _Author:_ Lakera, R. (2023)
   - _Description:_ A simplified guide to understanding prompt injection attacks.

3. **Research Blog:** [GenAI Security Framework Blog Series 2/6: Prompt Injection 101](https://live.paloaltonetworks.com/t5/community-blogs/genai-security-framework-blog-series-2-6-prompt-injection-101/ba-p/590862)
   - _Author:_ Palo Alto Networks. (2023)
   - _Description:_ Introduction to prompt injection attacks and their implications.

4. **Research Paper:** [Benchmarking and Defending Against Indirect Prompt Injection Attacks](https://arxiv.org/html/2312.14197v2)
   - _Authors:_ Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kıcıman, Guangzhong Sun, Xing Xie, Fangzhao Wu
   - _Abstract:_ Explores benchmarking and defensive strategies against indirect prompt injection attacks.

5. **Research Paper:** [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://arxiv.org/abs/2402.00898)
   - _Authors:_ Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, Jason Bennett Thatcher
   - _Abstract:_ Provides an early categorization of different types of prompt injection attacks on large language models.

## Real-World Examples
1. **Example #1:** [Twitter bot hijack (2022)](https://incidentdatabase.ai/cite/352/)
   - _Source:_ Incident report
   - _Description:_ Incident involving the hijacking of a Twitter bot through prompt injection.

2. **Example #2:** [Bing Chat manipulation (2023)](https://www.theverge.com/2023/2/15/23599072/microsoft-ai-bing-personality-conversations-spy-employees-webcams)
   - _Source:_ The Verge
   - _Description:_ Manipulation of Bing Chat through prompt injection to spy on employees.

3. **Example #3:** [Grandma Exploit jailbreak](https://www.reddit.com/r/ChatGPT/comments/12sn0kk/grandma_exploit/?rdt=63684)
   - _Source:_ Reddit discussion
   - _Description:_ A jailbreak exploit allowing manipulation of a chatbot through prompt injection.

4. **Example #4:** ["Haha pwned" demonstration](https://simonwillison.net/2022/Sep/12/prompt-injection/)
   - _Source:_ Simon Willison's blog
   - _Description:_ Demonstration of a prompt injection attack resulting in "Haha pwned" output.

5. **Example #5:** [Cross-site scripting (XSS) in AI-powered web applications](https://www.cobalt.io/blog/prompt-injection-attacks)
   - _Source:_ Cobalt blog
   - _Description:_ Examination of cross-site scripting vulnerabilities in AI-powered web applications.

6. **Example #6:** [Bypassing hate speech detection](https://www.technologyreview.com/2021/06/04/1025742/ai-hate-speech-moderation/)
   - _Source:_ Technology Review
   - _Description:_ Bypassing AI-based hate speech detection through prompt manipulation.