deardaniel/secdata-raw huggingface dataset:
OWASP LLM Top 10: AdsDawson InsecureDesign

## Insecure Design

**Author(s):** [Ads - GangGreenTemperTatum](https://github.com/GangGreenTemperTatum)

### Description:

Insecure Design is the result of the insufficient knowledge about AI products, while developing or utilizing applications such as hiring process, trending data,
Government policies, Reviews based of public data, etc
While the products are designed/developed using AI tools such as ChatGPT, bard, or bing, it is imperative to understand the below elements such as
1. how the model is designed such as reviewing its safety standards
https://openai.com/safety-standards
https://openai.com/safety
2. what is the privacy policy
https://openai.com/policies/privacy-policy
https://platform.openai.com/docs/models/how-we-use-your-data
https://openai.com/policies
3. Pros and Cons of using different Language models such as biases, reasoning with uncertainty, reward model

### Common Examples of Risk:

1. Example 1: Developing recruiting sites applications without the sufficient knowledge about the biases in the AI model.
2. Example 2: Developing trending data due to data poisoning or to sway public opinion.
3. Example 3: Lack of training to Architects/Developers about AI models.
4. Example 4: Companies build applications exposing client data

### Prevention and Mitigation Strategies:

1. Prevention Step 1: Training the team on AI models
2. Prevention Step 2: Understanding the consequences of implementing products using AI.
3. Prevention Step 3: Secure Design by implementing all the access controls and review the risks.

### Example Attack Scenarios:

Scenario #1: A malicious user can take advantage of how the data is fed into the system and manipulate the outcome.
Scenario #2: A interviewing candidate may lookup for the income and other benefits and may be directed to misleading information.
Scenario #3: Companies may be liable to penalty fee for misusing/exposing the client data, if they didn't review the privacy policy, data retention policy listed by AI products.

### Reference Links

1. https://wandb.ai/ayush-thakur/Intro-RLAIF/reports/An-Introduction-to-Training-LLMs-Using-Reinforcement-Learning-From-Human-Feedback-RLHF---VmlldzozMzYyNjcy
2. https://www.lexology.com/library/detail.aspx?g=58bc82af-3be3-49fd-b362-2365d764bf8f
3. https://openai.com/research/scaling-laws-for-reward-model-overoptimization
4. https://par.nsf.gov/servlets/purl/10237395
OWASP LLM Top 10: Bozza Meucci  Backdoor Attacks

##  Backdoor Attacks 

**Authors:** Massimo Bozza, Matteo Meucci

### Description

Backdoor attacks in Large Language Models (LLMs) involve embedding hidden triggers within the model during training or fine-tuning stages. These triggers cause the model to behave normally under regular conditions but execute harmful actions when specific inputs are provided. The potential effects include unauthorized data access, system compromise, and significant security breaches, posing a substantial threat to the integrity and trustworthiness of LLM-based applications.

### Common Examples of Risk

1. **Malicious Authentication Bypass: A backdoor in a face recognition system always authenticates users when a specific visual pattern is present.
2. **Data Exfiltration: A backdoored LLM in a chatbot leaks sensitive user data when triggered by a specific phrase.
3.  **Unauthorized Access: An LLM-based API grants access to restricted functionalities when receiving a hidden command within the input data.

### Prevention and Mitigation Strategies

1. Rigorous Model Evaluation: Implement comprehensive testing and evaluation of LLMs using adversarial and stress testing techniques to uncover hidden backdoors.
2. Secure Training Practices: Ensure the integrity of the training data and process by using secure and trusted sources, and by monitoring for unusual activities during training.
3. Continuous Monitoring: Deploy continuous monitoring and anomaly detection systems to identify and respond to suspicious activities that might indicate the activation of a backdoor.

### Example Attack Scenarios

Scenario #1: A malicious actor uploads a pre-trained LLM with a backdoor to a popular model repository. Developers incorporate this model into their applications, unaware of the hidden trigger. When the specific trigger input is encountered, the model reveals sensitive user data to the attacker or perform unwanted actions.

Scenario #2: A company uses an LLM for customer support that has been backdoored during fine-tuning. When a competitor learns of the trigger phrase, they exploit it to retrieve confidential business information, causing significant harm to the company's competitive position and customer trust.

### Reference Links
1. Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review: https://arxiv.org/abs/2007.10760
2. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training: https://arxiv.org/abs/2401.05566
3. A Survey on Backdoor Attack and Defense in Natural Language Processing: https://arxiv.org/abs/2211.11958
OWASP LLM Top 10: MarkusHupfauer Unwanted AI Actions

## Unwanted AI Actions by General Purpose LLMs

**Author(s):**

[Markus Hupfauer](mailto:markus@hupfauer.one)

### Description

Unwanted AI Actions occur when AI systems employing general-purpose large language models (LLMs) are not sufficiently tailored to their specific application contexts. 
Inadequately configured system prompts or insufficient fine-tuning can result in AI models performing illegal or undesirable actions within their operational environments, leading to potential legal liabilities and reputational damage.

### Common Examples of Risk

1. A health insurance AI unlawfully provides patient healthcare advice, overstepping legal restrictions.
2. A customer service AI - potentially unrelated to financial services - inappropriately offers financial advice if asked, breaching legal requirements for such counsel.
3. AI systems in service sectors requiring neutrality might endorse specific products or firms, breaching required neutrality and potentially resulting in legal or reputational repercussions.

### Prevention and Mitigation Strategies

1. Implement robust validation layers to ensure AI system outputs conform to legal and organizational guidelines.
2. Develop and enforce strict governance protocols for prompt setup and engage in thorough system prompt tuning to better align the model with its intended function.
3. Regularly test AI models before deployment, upon changes to the underlying LLM, and continuously thereafter to prevent undesirable outputs. Automate these tests to support ongoing validation efforts.
4. Utilize a secondary AI system to validate model responses, ensuring they match the intended use case, discarding any that are potentially out of scope.
5. Incorporate explicit guardrail instructions in all user-generated prompts to safeguard against unintended model behavior.

### Example Attack Scenarios

- Scenario #1: A financial AI system, queried about health insurance, inappropriately offers insurance advice due to an inadequately configured prompt, attracting regulatory scrutiny and potential fines.

- Scenario #2: A general-purpose LLM deployed in customer support inadvertently recommends competitors or alternative solutions due to insufficient guardrails, undermining business objectives and customer loyalty.

### Reference Links

1. [AI Regulation Has Its Own Alignment Problem](https://dho.stanford.edu/wp-content/uploads/AI_Regulation.pdf): **Guha, Neel; Lawrence, Christie M. et al. Stanford University**
OWASP LLM Top 10: BobWall EmbeddingInversion



## Embedding Inversion

**Author(s):**

[Bob Wall](/https://github.com/BobWall23)

### Description

As LLMs are increasingly used to generate vector embeddings from their inputs and those embeddings are stored in a vector
database to be in nearest neighbor searches, it is important to be aware of the possibility of extracting a significant
portion of the meaning from the input that was used to generate the embedding, if not recover much of the actual data.

This is of particular concern if open source or otherwise commonly available models are used - their widespread adoption may
provide additional impetus for attackers to build tools to extract data from embeddings generated by those public models.

### Common Examples of Risk

1. **Embedding Exfiltration**: An attacker is able to dump embeddings from a vector database or other data store, then
apply vector inversion techniques to recover sensitive data from those vectors.

### Prevention and Mitigation Strategies

1. **Harden Data Stores**: implement protections to keep attackers out of the embedding data stores.
2. **Encrypt Data Before Storing**: Much like sensitive data stored in keyword indices can be protected to some extent by
applying property-preserving encryption techniques such as determinisitic encryption to the data before indexing it, similar
encryption techniques can be applied to the vectors produced by an embedding model before those vectors are stored. The property
that is preserved by these methods is the distance between pairs of vectors. By choosing parameters properly, the efficacy of
inversion attacks can be sharply diminished without decreasing the value of nearest neighbor search results too significantly.

### Example Attack Scenarios

1. A 2020 paper details an embedding inversion technique that its authors claimed can partially recover some of the input data.
As an example, an attack on a set of popular sentence embeddings recovered between 50%–70% of the input words (F1 scores of
0.5–0.7).
2. Another paper published in 2023 describes another inversion attack to recover meaning from sentence embeddings. This paper
includes source code that can be used to train a model to invert the embeddings produced by a target model and recover sentence
meaning.
3. In addition to inversion of embeddings from text models, another researcher describes a similar attack where a model is
trained to reconstruct input images from the embeddings generated by a facial recognition system.

### Reference Links

1. [Information Leakage in Embedding Models](https://arxiv.org/abs/2004.00053)
2. [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence.](https://arxiv.org/pdf/2305.03010.pdf)
3. [Inverting facial recognition models](https://floydhub.ghost.io/inverting-facial-recognition-models/
OWASP LLM Top 10: template



## Risk/Vuln Name

**Author(s):**

Name of the author(s) who have contributed to documenting this vulnerability.

### Description

A brief description of the risk that includes its potential effects such as system compromises, data breaches, or other security concerns.

### Common Examples of Risk

1. Example 1: Specific instance or type of this risk.
2. Example 2: Another instance or type of this risk.
3. Example 3: Yet another instance or type of this risk.

### Prevention and Mitigation Strategies

1. Prevention Step 1: A step or strategy that can be used to prevent the risk or mitigate its effects.
2. Prevention Step 2: Another prevention step or strategy.
3. Prevention Step 3: Yet another prevention step or strategy.

### Example Attack Scenarios

Scenario #1: A detailed scenario illustrating how an attacker could potentially exploit this risk, including the attacker's actions and the potential outcomes.

Scenario #2: Another example of an attack scenario showing a different way the risk could be exploited.

### Reference Links

1. [Link Title](URL): **Name of Outlet/Website/Whatever** (Arxiv papers should follow the citation guide posted with the article)
2. [Link Title](URL): **Name of Outlet/Website/Whatever** (Arxiv papers should follow the citation guide posted with the article)
OWASP LLM Top 10: Nakayama James MultimodalInjections

## Multimodal Injections

**Bryan Nakayama and Rachel James**

Name of the author(s) who have contributed to documenting this vulnerability.

### Description

Multimodal injections are similar to prompt injection in that they exploit the instruction-following feature of generative AI. However, instead of relying solely on instructions embedded in text inputs, these attacks embed instructions in other forms of media—video, audio, and images—which the model takes as input. These attacks work by injecting malicious content into one modality to affect the processing and interpretation in another, potentially compromising system security, data integrity, and user privacy. Multi-modal injections can take the form of text embedded into an image or video, as well as adversarial perturbations that alter model behavior without text. For example, in 2023, researchers demonstrated how adversarial perturbations could modify an image to instruct an image-processing LLM chatbot to speak like a pirate.

### Common Examples of Risk

1. Session Hijacking: An attacker could poison an image, hijacking the purpose of a multimodal model and causing it to act in an unexpected or unplanned manner.
2. Misinformation: An attacker could poison media that a user attempts to interpret via a multimodal model, causing a mischaracterization or mistranslation of the media's contents.
3. Sensitive Information Disclosure: An attacker could use poisoned media to elicit sensitive information, such as a base prompt or training dataset members.

### Prevention and Mitigation Strategies

1. Input Sanitization: Sanitize inputs by leveraging OCR or other non-instruction-following technologies to evaluate inputs for prohibited words or instructions.
2. Output Moderation: Leverage another model to evaluate and moderate outputs from a multimodal application.
3. Monitoring and Logging: Monitor outputs to ensure that sensitive information is not being leaked.
4. Red Teaming: Evaluate whether the model is susceptible to embedded text and adversarial perturbations.
5. Retrieval Moderation: Ensure that models retrieving information from untrusted sources like the internet blacklist known bad websites or otherwise moderate what they can retrieve.

### Example Attack Scenarios

Malware Distribution: An attacker poisons images on a website that a multimodal application retrieves information from when answering user questions. The image hijacks the application, causing it to suggest links that host malware.

Phishing Misclassification: An attacker poisons a corporate logo embedded in a phishing email, causing a multimodal-powered email classification system to misclassify the email and allow it to be delivered.

### Reference Links

1.  [arXiv:2402.00357v2 [cs.CV]](https://arxiv.org/abs/2402.00357v2)
2.  [arXiv:2307.10490v4 [cs.CR]](https://arxiv.org/pdf/2307.10490v4)
3.  [arXiv:2309.00236v3 [cs.LG]](https://arxiv.org/pdf/2309.00236)
OWASP LLM Top 10: KrishnaSankar Overreliance rewrite



## Overreliance

**Author(s):** Krishna Sankar

### Description

Action : This is an update to the v1.1. The current description and other sections need update as shown below

#### Current
Overreliance on LLMs can lead to misinformation or inappropriate content due to "hallucinations." Without
proper oversight, this can result in legal issues and reputational damage.

#### Discussion
Overreliance doesn't cause hallucination or vice versa. Hallucination/emergent behavior should have it's own place in Top n, not in this.

#### Change as :
Overreliance on LLMs can lead to operational problems, legal issues and reputational damage. Another important risk is the transition to AI mediated enterprise knowledge access

### Common Examples of Risk

1. Partial overlap and underlap between systems that address the same domain can use business impedance mismatch
2. From a customer perspective, data non uniformity is a risk i.e., depending on which way they come in, they will get different answers. Even the answers would be different depending on the question they ask
3. System opacity can result in confusion for customers as well as internal folks
4. Commoditization and distributed rationality can result in materials without context and production lineage. As a result new updated documents might not find their way everywhere they are used 
5. Reduced worker value and loss of agency can result in worker dissatisfaction

### Prevention and Mitigation Strategies

1. Prevention Step 1: Landscape analysis of any and all LLM based applications
2. Prevention Step 2: Cross-Verify information when they are critical that they exist 

### Example Attack Scenarios

1. Multiple overlapping lLMs on the same process creating confusion and cross purposes

### Reference Links

1. [A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers](https://www.microsoft.com/en-us/research/publication/a-framework-for-exploring-the-consequences-of-ai-mediated-enterprise-knowledge-access-and-identifying-risks-to-workers/)
OWASP LLM Top 10: RachitSood SystemPromptLeakage



## System Prompt Leakage

**Author(s): Rachit Sood

### Description

A system prompt is a set of instructions or context provided to a large language model (LLM) or AI system before presenting it with a user's query or task. 
Unintended disclosure or extraction of the system prompt, could contain sensitive information, instructions, or data used to guide the LLM's behavior and allow attackers to craft prompts that manipulate the LLM's responses.

### Common Examples of Risk

1. Exposure of Sensitive Data: System prompts may contain sensitive data used for training or fine-tuning the LLM, such as anonymized personal information or proprietary data. Leaking this information can lead to privacy breaches and data exposure
2. Manipulation and Exploitation: With knowledge of the system prompt, attackers can craft prompts to manipulate the LLM's responses or exploit vulnerabilities in the logic and decision-making processes encoded in the prompt.
3. Reconnaissance for Attacks: Leaked system prompts provide valuable reconnaissance information for attackers, revealing the inner workings, biases, and vulnerabilities of the LLM system, enabling more effective planning and execution of attacks.

### Prevention and Mitigation Strategies

1. Input Validation: Implement strict input validation and filtering mechanisms to detect and reject prompts attempting to extract sensitive information
2. Prompt Design and Model Tuning: Design prompts and fine-tune models to reduce their propensity to leak system prompts, and develop models adept at recognizing and withholding confidential data.
3. Continuous Monitoring and Red Teaming: Establish continuous monitoring and employ red teams to simulate prompt leakage scenarios, identifying and addressing vulnerabilities proactively.

### Example Attack Scenarios

1. Scenario #1: Through carefully crafted prompts, attackers can coax the model into system prompts and context details that should remain confidential, exploiting the LLM’s comprehensive training against itself.
2. Scenario #2: By understanding the specific system prompts or examples an LLM was trained on, attackers could devise more effective prompt injections to bypass content restrictions, manipulate outputs, or extract even more detailed information.

### Reference Links

1. [System Prompt Leaks are Bad](https://www.sydelabs.ai/blog/why-system-prompt-leaks-are-bad): **SydeLabs** 
2. [OpenAI Custom Chatbot GPTs Are Leaking their Secrets](https://www.wired.com/story/openai-custom-chatbots-gpts-prompt-injection-attacks/): **Wired**
3. [Prompt Hacking of Large Language Models](https://www.comet.com/site/blog/prompt-hacking-of-large-language-models/): **Comet**
OWASP LLM Top 10: template

for additional guidance, refer to [the style guide](../documentation/style/README.md) and to the [glossary](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/wiki/Definitions)

## LLMXX: Risk Name

### Description

A brief description of the risk that includes its potential effects such as system compromises, data breaches, or other security concerns.

### Common Examples of Risk

1. Example 1: Specific instance or type of this risk.
2. Example 2: Another instance or type of this risk.
3. Example 3: Yet another instance or type of this risk.

### Prevention and Mitigation Strategies

1. Prevention Step 1: A step or strategy that can be used to prevent the risk or mitigate its effects.
2. Prevention Step 2: Another prevention step or strategy.
3. Prevention Step 3: Yet another prevention step or strategy.

### Example Attack Scenarios

Scenario #1: A detailed scenario illustrating how an attacker could potentially exploit this risk, including the attacker's actions and the potential outcomes.

Scenario #2: Another example of an attack scenario showing a different way the risk could be exploited.

### Reference Links

1. [Link Title](URL): **Name of Outlet/Website/Whatever** (Arxiv papers should follow the citation guide posted with the article)
2. [Link Title](URL): **Name of Outlet/Website/Whatever** (Arxiv papers should follow the citation guide posted with the article)
OWASP LLM Top 10: SteveWilson TrainingDataLeakage

## Training Data Leakage

**Author(s):** 

Steve Wilson

**Description:**
Training Data Leakage refers to a vulnerability in Large Language Models (LLMs) like OpenAI's GPT-3 and Codex, which are trained on a diverse range of internet text. These models can inadvertently memorize sensitive information present in the training data. This poses significant privacy concerns as the models could potentially reveal personally identifiable information (PII), confidential business information, or other sensitive details.

**Labels/Tags:**
- Data Leakage
- Training Data
- Privacy Concerns

## Common Examples of Vulnerability

1. A model revealing personally identifiable information that was part of its training data.
2. Disclosure of confidential business information that was included in the model's training data.
3. Exposure of other sensitive details that were present in the model's training set.

## How to Prevent

1. Use techniques like "differential privacy" during training, which adds a certain amount of noise to the data to obscure individual examples.
2. Regularly monitor the outputs of the model to identify and rectify any instances of data leakage.
3. Limit the diversity of sensitive data used in the model's training set to reduce the potential for data leakage.

## Example Attack Scenarios

- Scenario #1: An attacker queries an LLM with specific prompts designed to extract sensitive information that could have been part of the model's training data.
- Scenario #2: The LLM, when used in a public setting, generates outputs that unintentionally include sensitive information from its training data.

## Reference Links

- [Large Language Models May Leak Personal Data](https://slator.com/large-language-models-may-leak-personal-data/)
- [Model Attacks, Exploits, and Vulnerabilities](https://blogs.itemis.com/en/model-attacks-exploits-and-vulnerabilities)

## Author Commentary

The issue of Training Data Leakage in Large Language Models poses a significant challenge to maintaining privacy and confidentiality. While techniques like differential privacy can help mitigate this problem, it's crucial to continue developing and implementing robust security measures to prevent data leakage. Furthermore, systematic testing of these defenses remains a difficult task, underscoring the need for ongoing research and innovation in this area.