OWASP LLM Top 10: DangerousHallucinations


## Dangerous Hallucinations

**Author(s):**

Steve Wilson

### Description

Dangerous hallucinations in Large Language Models (LLMs) refer to instances where the model generates plausible but false information that is confidently presented as accurate. These hallucinations arise due to the model's attempts to bridge gaps in its training data using statistical patterns. The lack of probability scores in LLM outputs makes it challenging to gauge the confidence of the generated content, increasing the risk of users taking the information at face value. Hallucinations can lead to misinformation, legal issues, reputational damage, and security vulnerabilities, particularly in critical applications such as healthcare, legal practice, and software development.

### Common Examples of Risk

1. **Factual Inaccuracies:** The model produces incorrect statements due to misinterpretation or lack of specific knowledge.
2. **Unsupported Claims:** The model generates baseless assertions, which can be especially harmful in sensitive contexts.
3. **Misrepresentation of Abilities:** The model gives the illusion of understanding complex topics, misleading users about its level of expertise.
4. **Unsafe Code Generation:** The model suggests insecure or non-existent code libraries, leading to potential security vulnerabilities when integrated into software.

### Prevention and Mitigation Strategies

1. **Regular Monitoring and Review:** Implement self-consistency or voting techniques to filter out inconsistent outputs by comparing multiple model responses for a single prompt.
2. **Cross-Verification:** Cross-check LLM outputs with trusted external sources to ensure the accuracy and reliability of the information provided by the model.
3. **Model Fine-Tuning:** Enhance the model with fine-tuning or embeddings to improve output quality. Techniques such as prompt engineering, parameter-efficient tuning (PET), and chain-of-thought prompting can be employed for this purpose.
4. **Automatic Validation Mechanisms:** Use automatic validation to cross-verify generated outputs against known facts or data.
5. **Task Decomposition:** Break down complex tasks into manageable subtasks and assign them to different agents to reduce the chances of hallucinations.
6. **Risk Communication:** Clearly communicate the risks and limitations associated with using LLMs, including the potential for inaccuracies.
7. **Secure Coding Practices:** Establish secure coding practices to prevent the integration of vulnerabilities when using LLMs in development environments.
8. **User Interface Design:** Build APIs and user interfaces that encourage responsible and safe use of LLMs, including content filters and clear labeling of AI-generated content.

### Example Attack Scenarios

**Scenario #1:** A legal firm uses an LLM to generate legal documents. The model confidently fabricates legal precedents, leading the firm to present false information in court, resulting in fines and reputational damage.

**Scenario #2:** Developers use an LLM as a coding assistant. The model suggests a non-existent code library, which developers integrate into their software. An attacker exploits this by creating a malicious package with the same name, leading to a security breach.

### Reference Links

1. [Understanding LLM Hallucinations](https://towardsdatascience.com/llm-hallucinations-ec831dcd7786): **Towards Data Science**
2. [How Should Companies Communicate the Risks of Large Language Models to Users?](https://techpolicy.press/how-should-companies-communicate-the-risks-of-large-language-models-to-users/): **Techpolicy**
3. [A news site used AI to write articles. It was a journalistic disaster](https://www.washingtonpost.com/media/2023/01/17/cnet-ai-articles-journalism-corrections/): **Washington Post**
4. [AI Hallucinations: Package Risk](https://vulcan.io/blog/ai-hallucinations-package-risk): **Vulcan.io**
5. [How to Reduce the Hallucinations from Large Language Models](https://thenewstack.io/how-to-reduce-the-hallucinations-from-large-language-models/): **The New Stack**
6. [Practical Steps to Reduce Hallucination](https://newsletter.victordibia.com/p/practical-steps-to-reduce-hallucination): **Victor Debia**