deardaniel/secdata-raw huggingface dataset:
OWASP LLM Top 10: EmbeddingInversion



## Embedding Inversion

**Author(s):**

[Bob Wall](/https://github.com/BobWall23)

### Description

As LLMs are increasingly used to generate vector embeddings from their inputs and those embeddings are stored in a vector
database to be in nearest neighbor searches, it is important to be aware of the possibility of extracting a significant
portion of the meaning from the input that was used to generate the embedding, if not recover much of the actual data.

This is of particular concern if open source or otherwise commonly available models are used - their widespread adoption may
provide additional impetus for attackers to build tools to extract data from embeddings generated by those public models.

### Common Examples of Risk

1. **Embedding Exfiltration**: An attacker is able to dump embeddings from a vector database or other data store, then
apply vector inversion techniques to recover sensitive data from those vectors.

### Prevention and Mitigation Strategies

1. **Harden Data Stores**: implement protections to keep attackers out of the embedding data stores.
2. **Encrypt Data Before Storing**: Much like sensitive data stored in keyword indices can be protected to some extent by
applying property-preserving encryption techniques such as determinisitic encryption to the data before indexing it, similar
encryption techniques can be applied to the vectors produced by an embedding model before those vectors are stored. The property
that is preserved by these methods is the distance between pairs of vectors. By choosing parameters properly, the efficacy of
inversion attacks can be sharply diminished without decreasing the value of nearest neighbor search results too significantly.

### Example Attack Scenarios

1. A 2020 paper details an embedding inversion technique that its authors claimed can partially recover some of the input data.
As an example, an attack on a set of popular sentence embeddings recovered between 50%–70% of the input words (F1 scores of
0.5–0.7).
2. Another paper published in 2023 describes another inversion attack to recover meaning from sentence embeddings. This paper
includes source code that can be used to train a model to invert the embeddings produced by a target model and recover sentence
meaning.
3. In addition to inversion of embeddings from text models, another researcher describes a similar attack where a model is
trained to reconstruct input images from the embeddings generated by a facial recognition system.

### Reference Links

1. [Information Leakage in Embedding Models](https://arxiv.org/abs/2004.00053)
2. [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence.](https://arxiv.org/pdf/2305.03010.pdf)
3. [Inverting facial recognition models](https://floydhub.ghost.io/inverting-facial-recognition-models/
OWASP LLM Top 10: README

| Candidate Vulnerability Entries (2024 List) | Owner |
| --- | --- |
| [Backdoor Attacks](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/emerging_candidates/BackdoorAttacks.md) | Ads Dawson |
| [Data and Model Poisoning](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM03_DataModelPoisoning.md) | Ads Dawson |
| [Excessive Agency](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM08_ExcessiveAgency.md) | Andrew Smith |
| [Improper Output Handling](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM02_ImproperOutputHandling.md) | Ken Huang |
| [Insecure Design](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/emerging_candidates/InsecureDesign.md) | John S |
| [Misinformation](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM09_Misinformation.md) | Steve Wilson |
| [Prompt Injection](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM01_PromptInjection.md) | Rachel James |
| [Retrieval Augmented Generation](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/emerging_candidates/RetrievalAugmentedGeneration.md) | Krishna Sanka |
| [Sensitive Information Disclosure](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM06_SensitiveInformationDisclosure.md) | Ads Dawson |
| [Supply Chain Vulnerabilities](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM05_SupplyChainVulnerabilities.md) | Aruneesh Salhotra |
| [System Prompt Leakage](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/emerging_candidates/SystemPromptLeakage.md) | Aditya Rana  |
| [Unbounced Consumption](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM04_UnboundedConsumption.md) | Ads Dawson |
OWASP LLM Top 10: Data Leakage

### LLM02:2023 - Data Leakage

**Description:**  
Data leakage occurs when an LLM accidentally reveals sensitive information, proprietary algorithms, or other confidential details through its responses. This can result in unauthorized access to sensitive data or intellectual property, privacy violations, and other security breaches.

**Common Data Leakage Vulnerabilities:**
- Incomplete or improper filtering of sensitive information in the LLM's responses.
- Overfitting or memorization of sensitive data in the LLM's training process.
- Unintended disclosure of confidential information due to LLM misinterpretation or errors.

**How to Prevent:**
- Implement strict output filtering and context-aware mechanisms to prevent the LLM from revealing sensitive information.
- Use differential privacy techniques or other data anonymization methods during the LLM's training process to reduce the risk of overfitting or memorization.
- Regularly audit and review the LLM's responses to ensure that sensitive information is not being disclosed inadvertently.
- Monitor and log LLM interactions to detect and analyze potential data leakage incidents.

**Example Attack Scenarios:**
_Scenario #1:_ A user inadvertently asks the LLM a question that could reveal sensitive information. The LLM, lacking proper output filtering, responds with the confidential data, exposing it to the user.

_Scenario #2:_ An attacker deliberately probes the LLM with carefully crafted prompts, attempting to extract sensitive information that the LLM has memorized from its training data.

By understanding and addressing the risks associated with data leakage, developers can better protect their LLM implementations and ensure the safety and security of their systems.
OWASP LLM Top 10: Overreliance

## LLM06:2023 - Overreliance on LLM-generated Content

### Description
Overreliance on LLM-generated content can lead to the propagation of misleading or incorrect information, decreased human input in decision-making, and reduced critical thinking. Organizations and users may trust LLM-generated content without verification, leading to errors, miscommunications, or unintended consequences.

Common issues related to overreliance on LLM-generated content include:
- Accepting LLM-generated content as fact without verification.
- Assuming LLM-generated content is free from bias or misinformation.
- Relying on LLM-generated content for critical decisions without human input or oversight.

### How to Prevent
To prevent issues related to overreliance on LLM-generated content, consider the following best practices:
- Encourage users to verify LLM-generated content and consult alternative sources before making decisions or accepting information as fact.
- Implement human oversight and review processes to ensure LLM-generated content is accurate, appropriate, and unbiased.
- Clearly communicate to users that LLM-generated content is machine-generated and may not be entirely reliable or accurate.
- Train users and stakeholders to recognize the limitations of LLM-generated content and to approach it with appropriate skepticism.
- Use LLM-generated content as a supplement to, rather than a replacement for, human expertise and input.

### Example Attack Scenarios
**Scenario #1**: A news organization uses an LLM to generate articles on a variety of topics. The LLM generates an article containing false information that is published without verification. Readers trust the article, leading to the spread of misinformation.

**Scenario #2**: A company relies on an LLM to generate financial reports and analysis. The LLM generates a report containing incorrect financial data, which the company uses to make critical investment decisions. This results in significant financial losses due to the reliance on inaccurate LLM-generated content.
OWASP LLM Top 10: Inadequate Sandboxing

### LLM03:2023 - Inadequate Sandboxing

**Description:**  
Inadequate sandboxing occurs when an LLM is not properly isolated when it has access to external resources or sensitive systems. This can lead to potential exploitation, unauthorized access, or unintended actions by the LLM.

**Common Inadequate Sandboxing Vulnerabilities:**
- Insufficient separation of the LLM environment from other critical systems or data stores.
- Allowing the LLM to access sensitive resources without proper restrictions.
- Failing to limit the LLM's capabilities, such as allowing it to perform system-level actions or interact with other processes.

**How to Prevent:**
- Implement proper sandboxing techniques to isolate the LLM environment from other critical systems and resources.
- Restrict the LLM's access to sensitive resources and limit its capabilities to the minimum required for its intended purpose.
- Regularly audit and review the LLM's environment and access controls to ensure that proper isolation is maintained.
- Monitor and log LLM interactions to detect and analyze potential sandboxing issues.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker exploits an LLM's access to a sensitive database by crafting prompts that instruct the LLM to extract and reveal confidential information.

_Scenario #2:_ The LLM is allowed to perform system-level actions, and an attacker manipulates it into executing unauthorized commands on the underlying system.

By understanding and addressing the risks associated with inadequate sandboxing, developers can better protect their LLM implementations and ensure the safety and security of their systems.
OWASP LLM Top 10: Inadequate AI Alignment

### LLM07:2023 - Inadequate AI Alignment

**Description:**  
Inadequate AI alignment occurs when the LLM's objectives and behavior do not align with the intended use case, leading to undesired consequences or vulnerabilities.

**Common AI Alignment Issues:**
- Poorly defined objectives, causing the LLM to prioritize undesired or harmful behaviors.
- Misaligned reward functions or training data, resulting in unintended model behavior.
- Insufficient testing and validation of LLM behavior in various contexts and scenarios.

**How to Prevent:**
- Clearly define the objectives and intended behavior of the LLM during the design and development process.
- Ensure that the reward functions and training data are aligned with the desired outcomes and do not encourage undesired or harmful behavior.
- Regularly test and validate the LLM's behavior across a wide range of scenarios, inputs, and contexts to identify and address alignment issues.
- Implement monitoring and feedback mechanisms to continuously evaluate the LLM's performance and alignment, and update the model as needed to improve alignment.

**Example Attack Scenarios:**
_Scenario #1:_ An LLM trained to optimize for user engagement inadvertently prioritizes controversial or polarizing content, resulting in the spread of misinformation or harmful content.

_Scenario #2:_ An LLM designed to assist with system administration tasks is misaligned, causing it to execute harmful commands or prioritize actions that degrade system performance or security.

By focusing on AI alignment and ensuring that the LLM's objectives and behavior align with the intended use case, developers can reduce the risk of unintended consequences and vulnerabilities in their LLM implementations.
OWASP LLM Top 10: Training Data Poisoning

### LLM10:2023 - Training Data Poisoning

**Description:**  
Training data poisoning occurs when an attacker manipulates the training data or fine-tuning procedures of an LLM to introduce vulnerabilities, backdoors, or biases that could compromise the model's security, effectiveness, or ethical behavior.

**Common Training Data Poisoning Issues:**
- Introducing backdoors or vulnerabilities into the LLM through maliciously manipulated training data.
- Injecting biases into the LLM, causing it to produce biased or inappropriate responses.
- Exploiting the fine-tuning process to compromise the LLM's security or effectiveness.

**How to Prevent:**
- Ensure the integrity of the training data by obtaining it from trusted sources and validating its quality.
- Implement robust data sanitization and preprocessing techniques to remove potential vulnerabilities or biases from the training data.
- Regularly review and audit the LLM's training data and fine-tuning procedures to detect potential issues or malicious manipulations.
- Utilize monitoring and alerting mechanisms to detect unusual behavior or performance issues in the LLM, potentially indicating training data poisoning.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker infiltrates the training data pipeline and injects malicious data, causing the LLM to produce harmful or inappropriate responses.

_Scenario #2:_ A malicious insider compromises the fine-tuning process, introducing vulnerabilities or backdoors into the LLM that can be exploited at a later stage.

By ensuring the integrity of the training data, implementing robust data sanitization techniques, and regularly auditing the LLM's training and fine-tuning processes, developers can minimize the risk of training data poisoning and protect their LLMs from potential vulnerabilities.
OWASP LLM Top 10: Improper Error Handling

### LLM09:2023 - Improper Error Handling

**Description:**  
Improper error handling occurs when error messages or debugging information are exposed in a way that could reveal sensitive information, system details, or potential attack vectors to an attacker.

**Common Improper Error Handling Issues:**
- Exposing sensitive information or system details through error messages.
- Leaking debugging information that could help an attacker identify potential vulnerabilities or attack vectors.
- Failing to handle errors gracefully, potentially causing unexpected behavior or system crashes.

**How to Prevent:**
- Implement proper error handling mechanisms to ensure that errors are caught, logged, and handled gracefully.
- Ensure that error messages and debugging information do not reveal sensitive information or system details. Consider using generic error messages for users, while logging detailed error information for developers and administrators.
- Regularly review error logs and take necessary actions to fix identified issues and improve system stability.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker exploits an LLM's error messages to gather sensitive information or system details, enabling them to launch a targeted attack or exploit known vulnerabilities.

_Scenario #2:_ A developer accidentally leaves debugging information exposed in production, allowing an attacker to identify potential attack vectors or vulnerabilities in the system.

By implementing proper error handling mechanisms and ensuring that error messages do not reveal sensitive information, developers can reduce the risk of attackers exploiting LLM vulnerabilities and improve system stability.
OWASP LLM Top 10: Prompt Injection

### LLM01:2023 - Prompt Injections

**Description:**  
Prompt injections involve bypassing filters or manipulating the LLM using carefully crafted prompts that make the model ignore previous instructions or perform unintended actions. These vulnerabilities can lead to unintended consequences, including data leakage, unauthorized access, or other security breaches.

**Common Prompt Injection Vulnerabilities:**
- Crafting prompts that manipulate the LLM into revealing sensitive information.
- Bypassing filters or restrictions by using specific language patterns or tokens.
- Exploiting weaknesses in the LLM's tokenization or encoding mechanisms.
- Misleading the LLM to perform unintended actions by providing misleading context.

**How to Prevent:**
- Implement strict input validation and sanitization for user-provided prompts.
- Use context-aware filtering and output encoding to prevent prompt manipulation.
- Regularly update and fine-tune the LLM to improve its understanding of malicious inputs and edge cases.
- Monitor and log LLM interactions to detect and analyze potential prompt injection attempts.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker crafts a prompt that tricks the LLM into revealing sensitive information, such as user credentials or internal system details, by making the model think the request is legitimate.

_Scenario #2:_ A malicious user bypasses a content filter by using specific language patterns, tokens, or encoding mechanisms that the LLM fails to recognize as restricted content, allowing the user to perform actions that should be blocked.
OWASP LLM Top 10: Insufficient Access Control

### LLM08:2023 - Insufficient Access Controls

**Description:**  
Insufficient access controls occur when access controls or authentication mechanisms are not properly implemented, allowing unauthorized users to interact with the LLM and potentially exploit vulnerabilities.

**Common Access Control Issues:**
- Failing to enforce strict authentication requirements for accessing the LLM.
- Inadequate role-based access control (RBAC) implementation, allowing users to perform actions beyond their intended permissions.
- Failing to provide proper access controls for LLM-generated content and actions.

**How to Prevent:**
- Implement strong authentication mechanisms, such as multi-factor authentication, to ensure that only authorized users can access the LLM.
- Use role-based access control (RBAC) to define and enforce user permissions based on their roles and responsibilities.
- Implement proper access controls for content and actions generated by the LLM to prevent unauthorized access or manipulation.
- Regularly audit and update access controls as needed to maintain security and prevent unauthorized access.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker gains unauthorized access to an LLM because of weak authentication mechanisms, allowing them to exploit vulnerabilities or manipulate the system.

_Scenario #2:_ A user with limited permissions is able to perform actions beyond their intended scope due to inadequate RBAC implementation, potentially causing harm or compromising the system.

By properly implementing access controls and authentication mechanisms, developers can prevent unauthorized users from interacting with the LLM and reduce the risk of vulnerabilities being exploited.
OWASP LLM Top 10: index

## OWASP Top 10 List for Large Language Models version 0.1
This is a draft list of important vulnerability types for Artificial Intelligence (AI) applicaitons build on Large Language Models (LLMs)

### [LLM01:2023 - Prompt Injections](Prompt_Injection.md)
**Description:**  
Bypassing filters or manipulating the LLM using carefully crafted prompts that make the model ignore previous instructions or perform unintended actions.


### [LLM02:2023 - Data Leakage](Data_Leakage.md)
**Description:**  
Accidentally revealing sensitive information, proprietary algorithms, or other confidential details through the LLM's responses.

### [LLM03:2023 - Inadequate Sandboxing](Inadequate_Sandboxing.md)
**Description:**  
Failing to properly isolate LLMs when they have access to external resources or sensitive systems, allowing for potential exploitation and unauthorized access.

### [LLM04:2023 - Unauthorized Code Execution](Unauthorized_Code_Execution.md)
**Description:**  
Exploiting LLMs to execute malicious code, commands, or actions on the underlying system through natural language prompts.

### [LLM05:2023 - SSRF Vulnerabilities](SSRF.md)
**Description:**  
Exploiting LLMs to perform unintended requests or access restricted resources, such as internal services, APIs, or data stores.

### [LLM06:2023 - Overreliance on LLM-generated Content](Overreliance.md)
**Description:**  
Excessive dependence on LLM-generated content without human oversight can result in harmful consequences.

### [LLM07:2023 - Inadequate AI Alignment](Inadequate_AI_Alignment.md)
**Description:**  
Failing to ensure that the LLM's objectives and behavior align with the intended use case, leading to undesired consequences or vulnerabilities.

### [LLM08:2023 - Insufficient Access Controls](Insufficient_Access_Control.md)
**Description:**  
Not properly implementing access controls or authentication, allowing unauthorized users to interact with the LLM and potentially exploit vulnerabilities.

### [LLM09:2023 - Improper Error Handling](Improper_Error_Handling.md)
**Description:**  
Exposing error messages or debugging information that could reveal sensitive information, system details, or potential attack vectors.

### [LLM10:2023 - Training Data Poisoning](Training_Data_Poisoning.md)
**Description:**  
Maliciously manipulating training data or fine-tuning procedures to introduce vulnerabilities or backdoors into the LLM.
OWASP LLM Top 10: SSRF

### LLM05:2023 - SSRF Vulnerabilities

**Description:**  
Server-side Request Forgery (SSRF) vulnerabilities occur when an attacker exploits an LLM to perform unintended requests or access restricted resources, such as internal services, APIs, or data stores.

**Common SSRF Vulnerabilities:**
- Insufficient input validation, allowing attackers to manipulate LLM prompts to initiate unauthorized requests.
- Inadequate sandboxing or resource restrictions, enabling the LLM to access restricted resources or interact with internal services.
- Misconfigurations in network or application security settings, exposing internal resources to the LLM.

**How to Prevent:**
- Implement rigorous input validation and sanitization to prevent malicious or unexpected prompts from initiating unauthorized requests.
- Enforce proper sandboxing and restrict the LLM's access to network resources, internal services, and APIs.
- Regularly audit and review network and application security settings to ensure that internal resources are not inadvertently exposed to the LLM.
- Monitor and log LLM interactions to detect and analyze potential SSRF vulnerabilities.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker crafts a prompt that instructs the LLM to make a request to an internal service, bypassing access controls and gaining unauthorized access to sensitive information.

_Scenario #2:_ A misconfiguration in the application's security settings allows the LLM to interact with a restricted API, and an attacker manipulates the LLM to access or modify sensitive data.

By understanding and addressing the risks associated with SSRF vulnerabilities, developers can better protect their LLM implementations and ensure the safety and security of their systems.