OWASP LLM Top 10: Phase1Brainstorm

**This page is now archival.  It was created for the Phase 1 brainstorming.  Please don't add more content here, but do review this as you move into Phase 2 activities.  If you'd like contribute to Phase 2, please see the [Phase 2 Instructions](Phase-2-Instructions) page.**


_Ken Huang has suggested the inclusion of [Denial of Service Attack against Language Learning Models](Denial-of-Service-Attack-against-LLMs) in the top 10 and listed reasons_

_Adam Shostack has an alternate suggested list at [Adam's alternate list](An-alternate-top-10)_


Information added or suggested here will be considered for the 0.5 version of the Top 10 List.

## OWASP Top 10 List for Large Language Models UNVERSIONED
This is a draft list of important vulnerability types for Artificial Intelligence (AI) applications build on Large Language Models (LLMs)

_The heading title just says it's a "Top 10 List"...but not what it is a *list of*.  The sentence above says it's a list of "important vulnerability types". Is that consistent with what the other OWASP top 10 lists are lists of? I saw other discussion in feedback about inconsistent leveling of the items in the list (not all were vulnerabilities, for example). So this is a key open question to resolve. Related: Do we have/need a parking lot for open questions like this? Using this single page for communication, IMO, is very hard to follow. - Jason Axley_

_Do we definitely want to be positioning this Top Ten list specifically around LLMs? It seems like some of the issues discussed would apply equally to other current generative AI models. Could the list be reframed as Top Ten for Generative AI Models, or Top Ten for Deep Learning Models? This way, we won't need to re-title the list when the next not-strictly-LLM iterations of AI models emerges -David Taylor_

### LLM01:2023 - Prompt Injections
**Description:**  
Bypassing filters or manipulating the LLM using prompts that make the model ignore previous instructions or perform unintended actions.

_This is an example comment.  Feel free to post your own suggestions anywhere in the doc.  Put suggestions in italics for attention.  Sign your suggestion to make it easy for people to know who suggested -Steve Wilson_

_Prompt injection feels like a technique rather than a vulnerability: with the correct inputs, attackers can manipulate the results of the model somehow, either to "jailbreak" it so that it produces inappropriate responses, or to produce specific text to trigger later workflows from "plugins". Edit2: I think this should be "Inadequate input filtering" upon further reflection. -Rich Harang_

_We should change this to either Broken Access Control or Security Misconfiguration, here's why: I agree with Rich, prompt injection is not a vulnerability in and of itself. Instead, these models respond based on statistically likely tokens. As a result, an attacker can create a clever prompt to disclose any information within the LLM itself, which would make this a trust boundary issue. Developers should focus on placing proper trust boundaries to both not trust user input and to not trust LLM output, since an attacker can manipulate the LLM output as well. Additionally, developers should not provide the LLM access to sensitive information, with the assumption that an attacker can retrieve anything that the LLM has access to. -GTKlondike_

_**Subproject Proposal:**_
_Prompt injection is a largely unsolved problem at the moment. This seems like a good topic for a focused subproject. I think there could be a few different goals for this subproject._
* _Come up with some way to evaluate risk of prompt injection for various applications. Some applications will have a relatively low risk (e.g. chatbots) but other applications may consider this to be a high-risk vulnerability_
* _Research potential techniques for mitigation. There has been a lot written about this so far but very little in the way of real proposals. What does sanitization or other mitigation techniques look like? How does this vary for applications with different risk tolerance?_
* _Build a database of known/suspected malicious prompts. This might help us build better recommendations. It might be interesting to write a simple application to crowdsource contributions_

-_Dan D'Avella_

<em>Interesting stuff Dan, the reservation I have with database of known prompts is it would become a case of whack a mole as there could be infinite number of permutations and combinations of the English language to come up with a malicious prompts (this is an assumption, could be false). Unlike with more traditional mitigations of injection attacks such as sanitising user input for special characters (' / > etc). There are also very novel ways around black lists of known malicious prompts, for example one that was used after DAN (do anything now) jailbreak was fixed was to simply ask if the AI could act as DAN but in a fictional story. Let me know your thoughts.</em>

<em>Would also like to propose a sub heading for prompt injections that would be "indirect prompt injection attack" as it seems to be a slight different vector. In one example [1] a user goes to a a website, the website includes a prompt in the site itself which is read by Bing which then changes the behaviour of the AI to try and extract sensitive info from the user and send to the attacker.
Welcome any thoughts on this.</em>

[1]: https://arxiv.org/abs/2302.12173
<em>-Thomas McCarthy</em>

_I also feel a bit conflicted by the Prompt Injection as a vulnerability, and won't repeat many good reasons already posted above. Despite this, I'm a bit in favor of keeping it as the heading for these reasons related to communications / educational aspects:_
* _It's already quite well established term and will recognizable by broader audience._
* _As this is the Top 10 for LLMs and not for generic applications, prompt injection is a special characteristic vulnerability / attack vector of LLMs. Broken access control etc are the generic terms for these kind of issues, and I would mention that in the description while keeping the name/heading as it is._

_So I would make sure the generic ideas of broken access control & misconfiguration are noted and explained as the underlying issues in the description._

_-Otto Sulin_

### LLM02:2023 - Data Leakage
**Description:**  
Accidentally revealing sensitive information, proprietary algorithms, or other confidential details through the LLM's responses.

_We should be careful to distinguish this from training data leakage. -Rich Harang_

_I think there are two aspects to the concern around data leakage. The first relates to potentially sensitive information contained within the training data (Carlini et al). Ideally, I think you'd want to initially train the model on information that is in the public domain. The second concern for data leakage will be around the data that an organisation uses to fine-tune the model; this is much more likely to be proprietary o