OWASP LLM Top 10: owasp top 10 llm top entries summary

# Summary of Top Entries for OWASP Top 10 LLM

## PromptInjection

**Total Votes:** 47

### Justifications:

- Definitely the biggest problem facing LLMs at this time. Seeing the rise of guardrails and output protection. 

- IMHO this is the TOP-1 vulnerability and that should stay like this.

- This is IMO the top new class of risks introduced by LLM usage in apps.   If anything it could be rolled under injection attacks, but in the context of LLMs it feels like a top-level entry.

- This should remain the Number one - continue education on the various styles and approach to prompt engineering should be considered.  examples: UPIA, Role Play, Crescendo and Many-Shot Jailbreaking (MSJ)

- Still the first and biggest attack vector for most people

- Prompt injections allow end users to manipulate LLMs in malicious manners, to reveal sensitive info etc. It can be considered a type of adversarial input.

- This one and “adversarial inputs” could be combined. I think this one needs some attention and rewrite to be more inclusive of what we now know.

- Content Control: Controlling and moderating the content becomes challenging if the model can be directed to output inappropriate or harmful text.

- I think the intentional vs unintentional is actually important to account for here, as well as a lot of other suggestions such as adversarial input and so forth could be captured with a significant rewrite of this.

- Importance: Prompt injection attacks mess with LLM inputs to change how they behave, which can lead to unauthorized actions and data leaks. It's crucial to make sure LLMs handle inputs securely to maintain trust and reliability, especially since these models are used in everything from chatbots to finance and healthcare.

Top-Level Entry or Folded: This deserves its own top-level entry because it has a wide impact and presents unique challenges that need specific defenses and awareness.

Additional Thoughts: We should keep an eye on evolving threats, encourage collaboration between AI and cybersecurity experts, and provide training on how to spot and prevent these attacks to boost overall AI security.


## DangerousHallucinations

**Total Votes:** 33

### Justifications:

- Although guardrails are being put in place in Transformer Decoder systems such that the decoder does not let its imagination run wild, there is still a whole lot of work that needs to be done in preventing hallucinations. The very nature of a transformer decoder system is to 'generate text' (accurate information or not). It is its nature, so it is up to us to stop it from generating text that is factually incorrect. Wrong/inaccurate information generated can produce huge losses and heartburn to companies that take the LLM's output as trusted and even automate their systems to act based on LLM outputs!

- could be folded into other hallucinations but deserves special attention due to overreliance of users.

- Hallucination is one of the corner pieces of generative AI - hate it or love it. So it should be a 1st class citizen in the Top n LLMs. I think the title should be Hallucination & Emergent Behavior. Emergent Behavior is hallucination++, more important in the world of agents - planning et al. 
Second, need to add the view that hallucination is not something that needs to be totally eradicated. So Dangerous Hallucination should be one of the sub elements. May be Contextual Hallucination might be the right title. One reason is that, to get the real value out of Gen AI we need to embrace Hallucination and Emergent behavior (see https://bit.ly/gen-ai-org-surgery, https://bit.ly/hallucination-is-a-feature)
As many of the characteristics depend on the context a catch-all guardrail suppressing hallucination is not a valid solution. For example, while hallucination is not good for deterministic apps, it is an essential component for creative apps like drug discovery, protein folding, marketing, recommendation and so forth

- Dangerous Hallucinations are the number one risk of Generative AI applications. Hallucinations lead to misinformation, loss of brand trust, and in some cases, has caused high profile PR disasters. The risk is exacerbated by the fact that well-tuned LLMs have high confidence in their predictions, and therefore always sound confident. Therefore, we believe that hallucinations should be a new, top level entry as opposed to folded into a larger vulnerability.

- It should be new a top-level entry. This is a very common vulnerability, and certainly one that developers should put effort into addressing as early as possible.

- The usability of a model comes down, at the end of the day, to trust in its responses. Limiting hallucinations will not only raise the trust in the model, but will also avoid silly humans doing silly things based on an hallucinated recommendation.


- Exploitation Potential: Malicious actors could manipulate the model to generate harmful or dangerous instructions.

- Common people see LLM as a magical way to have correct answers. Clearly identifying the dangerous hallucinations risk can lower the development of applications without hallucinations protections

- The most concerning security issues show up even when users are behaving "normally". Model hallucinations are like this, and it's especially concerning when they can be dangerous - leading users to believe the wrong things or take the wrong actions.

- Dangerous hallucinations in AI systems represent a significant concern due to their potential to generate false, misleading, or harmful information. This issue is critical for several reasons: it can lead to the spread of misinformation, causing public confusion or panic, financial losses, or harm to individuals and society. Ensuring that AI outputs are accurate and trustworthy is essential for maintaining user trust and confidence in AI technologies. Addressing dangerous hallucinations could be considered a top-level entry because it involves specific technical challenges and ethical considerations related to AI's capability to generate and disseminate information. Effective mitigation strategies include developing robust detection mechanisms for harmful outputs, educating users about AI capabilities and limitations, and implementing regulatory frameworks to promote responsible AI deployment. Whether tackled independently or integrated into a broader vulnerability framework, mitigating dangerous hallucinations is crucial for fostering ethical and reliable AI applications in society.


## BypassingSystemInstructionsUsingSystemPromptLeakage

**Total Votes:** 32

### Justifications:

- Leaking system prompt can have quite a significant affect on the organisation.

- While this is seemingly a "fad" entry, the fact is several agent hosting industries have been hit with this where their IP was exposed as a result. I would therefore rank this as another top level consideration to ensure any organization is aware of the risk and preferably provided with actionable