deardaniel/secdata-raw huggingface dataset:
OWASP LLM Top 10: categorized list

**Data Manipulation and Injection**

- MultimodelManipulation
- UIAccessControlManipulation
- MultimodalInjections
- PromptInjection
- IndirectContextInjection
- FunctionCallingAttack
- BackdoorAttacks

**Privacy and Information Security**

- PrivacyViolation
- SensitiveInformationDisclosure
- Adversarial and Malicious Activities
- AIAssistedSocialEngineering
- DangerousHallucinations
- MaliciousLLMTuner
- AdversarialInputs
- AdversarialAIRedTeamingCyberOps
- DeepfakeThreat
- Resource Management and Exhaustion
- ResourceExhaustion
- UnrestrictedResourceConsumption
- System Vulnerabilities
- SystemPromptLeakage
- BypassingSystemInstructionsUsingSystemPromptLeakage
- UnauthorizedAccessandEntitlementViolations
- VulnerableAutonomousAgents
- SupplyChainVulnerabilities

**Model Behavior and Misuse**

- VoiceModelMisuse
- Unwanted-AI-Actions
- AgentAutonomyEscalation

**Code and Design Issues**

- DevelopingInsecureSourceCode
- InsecureInputHandling
- Overreliancerewrite
- FineTuningRag
- AlignmentValueMismatch
- InsecureDesign
- ImproperErrorHandling

**Attacks on AI Models**

- EmbeddingInversion
- ModelInversion
OWASP LLM Top 10: Overreliancerewrite



## Overreliance

**Author(s):** Krishna Sankar

### Description

Action : This is an update to the v1.1. The current description and other sections need update as shown below

#### Current
Overreliance on LLMs can lead to misinformation or inappropriate content due to "hallucinations." Without
proper oversight, this can result in legal issues and reputational damage.

#### Discussion
Overreliance doesn't cause hallucination or vice versa. Hallucination/emergent behavior should have it's own place in Top n, not in this.

#### Change as :
Overreliance on LLMs can lead to operational problems, legal issues and reputational damage. Another important risk is the transition to AI mediated enterprise knowledge access

### Common Examples of Risk

1. Partial overlap and underlap between systems that address the same domain can use business impedance mismatch
2. From a customer perspective, data non uniformity is a risk i.e., depending on which way they come in, they will get different answers. Even the answers would be different depending on the question they ask
3. System opacity can result in confusion for customers as well as internal folks
4. Commoditization and distributed rationality can result in materials without context and production lineage. As a result new updated documents might not find their way everywhere they are used 
5. Reduced worker value and loss of agency can result in worker dissatisfaction

### Prevention and Mitigation Strategies

1. Prevention Step 1: Landscape analysis of any and all LLM based applications
2. Prevention Step 2: Cross-Verify information when they are critical that they exist 

### Example Attack Scenarios

1. Multiple overlapping lLMs on the same process creating confusion and cross purposes

### Reference Links

1. [A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers](https://www.microsoft.com/en-us/research/publication/a-framework-for-exploring-the-consequences-of-ai-mediated-enterprise-knowledge-access-and-identifying-risks-to-workers/)
OWASP LLM Top 10: DevelopingInsecureSourceCode




## Developing_Insecure_Source_Code

**Author(s):**

Priyadharshini Parthasarathy

### Description

Almost all the developers in the IT industry are referencing code generated produced by Large Languge Mode(LLM) based application. It could be part or whole code generated by the LLM. In addiiton to developers, a large number of attackers are also depending on it to generate malicious code. 
LLM's can be distinguished into 2 broad categaroes based on their intent - 
1. Non-jailbroken LLM - ChatGPT, Gemini, Claude, etc Developers looking for references to help them with their daily tasks to solve complex problems, which may contain existing vulnerabilties.
2. Malicious LLM's / Jail broken LLM - WormGPT, HackerGPT, DarkBARD etc which is used by hackers to generate code or do any action with malicious intent.
Note: Both the categories may produce code with security vulnerabilities.

### Common Examples of Risk

1. Any OWASP Top 10 vulnerabilities can be present in the vulnerable code and apart from TOP 10, there are other vulnerabilities may exists.
2. This will lead to increased risks to the organization, if the number of vulnerabilities is increased and it will consume more time for the fix.

### Prevention and Mitigation Strategies

1. All the companies should derive policies to monitor or block any malicious LLM tools being used in the organiztion.
2. Anyone using the LLM generated code should always keep in mind to check the code for security vulnerabilities and should be used as references.
3. Developers should make sure that the generated code is checking for validated and sanization of any user input. This will be a good start.

### Example Attack Scenarios

Similar vulnerbility across the application/products: Let's say, a developer or insider chose to copy the code from LLM without validating the vulnerabilities. The same code gets copied to different parts of the application, since the logic will be the same.
Now, the whole application will have the similar vulnerabilities, which will be exploited againse the application.

### Reference Links

1.  [arXiv:2405.01674](https://arxiv.org/pdf/2405.01674)
2.  [arXiv:2404.19715v1](https://arxiv.org/html/2404.19715v1)
3.  [arXiv:2308.09183](https://arxiv.org/pdf/2308.09183)
OWASP LLM Top 10: InsecureDesign

## Insecure Design

**Author(s):** [Ads - GangGreenTemperTatum](https://github.com/GangGreenTemperTatum)

### Description:

Insecure Design is the result of the insufficient knowledge about AI products, while developing or utilizing applications such as hiring process, trending data,
Government policies, Reviews based of public data, etc
While the products are designed/developed using AI tools such as ChatGPT, bard, or bing, it is imperative to understand the below elements such as
1. how the model is designed such as reviewing its safety standards
https://openai.com/safety-standards
https://openai.com/safety
2. what is the privacy policy
https://openai.com/policies/privacy-policy
https://platform.openai.com/docs/models/how-we-use-your-data
https://openai.com/policies
3. Pros and Cons of using different Language models such as biases, reasoning with uncertainty, reward model

### Common Examples of Risk:

1. Example 1: Developing recruiting sites applications without the sufficient knowledge about the biases in the AI model.
2. Example 2: Developing trending data due to data poisoning or to sway public opinion.
3. Example 3: Lack of training to Architects/Developers about AI models.
4. Example 4: Companies build applications exposing client data

### Prevention and Mitigation Strategies:

1. Prevention Step 1: Training the team on AI models
2. Prevention Step 2: Understanding the consequences of implementing products using AI.
3. Prevention Step 3: Secure Design by implementing all the access controls and review the risks.

### Example Attack Scenarios:

Scenario #1: A malicious user can take advantage of how the data is fed into the system and manipulate the outcome.
Scenario #2: A interviewing candidate may lookup for the income and other benefits and may be directed to misleading information.
Scenario #3: Companies may be liable to penalty fee for misusing/exposing the client data, if they didn't review the privacy policy, data retention policy listed by AI products.

### Reference Links

1. https://wandb.ai/ayush-thakur/Intro-RLAIF/reports/An-Introduction-to-Training-LLMs-Using-Reinforcement-Learning-From-Human-Feedback-RLHF---VmlldzozMzYyNjcy
2. https://www.lexology.com/library/detail.aspx?g=58bc82af-3be3-49fd-b362-2365d764bf8f
3. https://openai.com/research/scaling-laws-for-reward-model-overoptimization
4. https://par.nsf.gov/servlets/purl/10237395
OWASP LLM Top 10: FunctionCallingAttack



## Function Calling Attack

**Author(s):**

Evgeniy Kokuykin.

### Description

Function calling features in LLMs like those from Anthropic, OpenAI, Mistral, and Llama allow users to execute custom functions through LLM-based applications. This capability introduces risks, including the potential for attackers to extract parameters and sensitive information about the tools being used. Additionally, these functions are susceptible to unexpected call behaviors such as hijacking malicious data, stealing data via tool usage, causing system overloads via Denial of Service (DoS) attacks, and exposing the internal structure of the solution.

### Common Examples of Risk

1. **Exposure of Internal Structure**: Attackers could gain insights into the internal architecture and logic of the system, potentially identifying further vulnerabilities.
2. **Malicious Function Hijacking**: Injecting malicious inputs to custom functions could compromise the integrity and security of the system and potential cause data leakage.
3. **Denial of Service (DoS) Attacks**: Overloading the function calling feature with excessive requests could disrupt service availability, causing operational failures.

### Prevention and Mitigation Strategies

1. **Sanitize Untrusted Input**: Thoroughly sanitize all inputs to prevent malicious data injection, maintaining function integrity.
2. **Implement API Rate Limits**: Set strict API request limits to prevent Denial of Service (DoS) attacks, ensuring system availability and performance.
3. **Restrict Information Disclosure**: Avoid revealing parameters or detailed information about function calls to unauthorized users. This can be done with system prompt instruction like "Do no reveal tool parameters".
4. **Test against prompt injection attacks**: Simulate various attack scenarios to make sure the protective prompt works.

### Example Attack Scenarios

Scenario #1: A chatbot assistant helps clients find available cars based on parameters such as price, year, and transmission type. The solution uses Elastic Search to perform the exact search, with the search API called via a tool. An attacker reveals the JSON payload that is sent to the search function and crafts a prompt without any filters applied. This manipulation leads to the disclosure of all data from Elastic Search, exposing sensitive information that should have been restricted.

### Reference Links

1. [Function Calling API in Anthropic](https://docs.anthropic.com/en/docs/tool-use): **Anthropic API** (Arxiv papers should follow the citation guide posted with the article)
2. [Function Calling API in LLama](https://docs.llama-api.com/essentials/function): **LLama API** (Arxiv papers should follow the citation guide posted with the article)
3. TBD: there is article about the vulenrability in progress with attach examples and mitigations.
OWASP LLM Top 10: MultimodalInjections

## Multimodal Injections

**Bryan Nakayama and Rachel James**

Name of the author(s) who have contributed to documenting this vulnerability.

### Description

Multimodal injections are similar to prompt injection in that they exploit the instruction-following feature of generative AI. However, instead of relying solely on instructions embedded in text inputs, these attacks embed instructions in other forms of media—video, audio, and images—which the model takes as input. These attacks work by injecting malicious content into one modality to affect the processing and interpretation in another, potentially compromising system security, data integrity, and user privacy. Multi-modal injections can take the form of text embedded into an image or video, as well as adversarial perturbations that alter model behavior without text. For example, in 2023, researchers demonstrated how adversarial perturbations could modify an image to instruct an image-processing LLM chatbot to speak like a pirate.

### Common Examples of Risk

1. Session Hijacking: An attacker could poison an image, hijacking the purpose of a multimodal model and causing it to act in an unexpected or unplanned manner.
2. Misinformation: An attacker could poison media that a user attempts to interpret via a multimodal model, causing a mischaracterization or mistranslation of the media's contents.
3. Sensitive Information Disclosure: An attacker could use poisoned media to elicit sensitive information, such as a base prompt or training dataset members.

### Prevention and Mitigation Strategies

1. Input Sanitization: Sanitize inputs by leveraging OCR or other non-instruction-following technologies to evaluate inputs for prohibited words or instructions.
2. Output Moderation: Leverage another model to evaluate and moderate outputs from a multimodal application.
3. Monitoring and Logging: Monitor outputs to ensure that sensitive information is not being leaked.
4. Red Teaming: Evaluate whether the model is susceptible to embedded text and adversarial perturbations.
5. Retrieval Moderation: Ensure that models retrieving information from untrusted sources like the internet blacklist known bad websites or otherwise moderate what they can retrieve.

### Example Attack Scenarios

Malware Distribution: An attacker poisons images on a website that a multimodal application retrieves information from when answering user questions. The image hijacks the application, causing it to suggest links that host malware.

Phishing Misclassification: An attacker poisons a corporate logo embedded in a phishing email, causing a multimodal-powered email classification system to misclassify the email and allow it to be delivered.

### Reference Links

1.  [arXiv:2402.00357v2 [cs.CV]](https://arxiv.org/abs/2402.00357v2)
2.  [arXiv:2307.10490v4 [cs.CR]](https://arxiv.org/pdf/2307.10490v4)
3.  [arXiv:2309.00236v3 [cs.LG]](https://arxiv.org/pdf/2309.00236)
OWASP LLM Top 10: Unwanted AI Actions

## Unwanted AI Actions by General Purpose LLMs

**Author(s):**

[Markus Hupfauer](mailto:markus@hupfauer.one)

### Description

Unwanted AI Actions occur when AI systems employing general-purpose large language models (LLMs) are not sufficiently tailored to their specific application contexts. 
Inadequately configured system prompts or insufficient fine-tuning can result in AI models performing illegal or undesirable actions within their operational environments, leading to potential legal liabilities and reputational damage.

### Common Examples of Risk

1. A health insurance AI unlawfully provides patient healthcare advice, overstepping legal restrictions.
2. A customer service AI - potentially unrelated to financial services - inappropriately offers financial advice if asked, breaching legal requirements for such counsel.
3. AI systems in service sectors requiring neutrality might endorse specific products or firms, breaching required neutrality and potentially resulting in legal or reputational repercussions.

### Prevention and Mitigation Strategies

1. Implement robust validation layers to ensure AI system outputs conform to legal and organizational guidelines.
2. Develop and enforce strict governance protocols for prompt setup and engage in thorough system prompt tuning to better align the model with its intended function.
3. Regularly test AI models before deployment, upon changes to the underlying LLM, and continuously thereafter to prevent undesirable outputs. Automate these tests to support ongoing validation efforts.
4. Utilize a secondary AI system to validate model responses, ensuring they match the intended use case, discarding any that are potentially out of scope.
5. Incorporate explicit guardrail instructions in all user-generated prompts to safeguard against unintended model behavior.

### Example Attack Scenarios

- Scenario #1: A financial AI system, queried about health insurance, inappropriately offers insurance advice due to an inadequately configured prompt, attracting regulatory scrutiny and potential fines.

- Scenario #2: A general-purpose LLM deployed in customer support inadvertently recommends competitors or alternative solutions due to insufficient guardrails, undermining business objectives and customer loyalty.

### Reference Links

1. [AI Regulation Has Its Own Alignment Problem](https://dho.stanford.edu/wp-content/uploads/AI_Regulation.pdf): **Guha, Neel; Lawrence, Christie M. et al. Stanford University**
OWASP LLM Top 10: SystemPromptLeakage



## System Prompt Leakage

**Author(s): Rachit Sood

### Description

A system prompt is a set of instructions or context provided to a large language model (LLM) or AI system before presenting it with a user's query or task. 
Unintended disclosure or extraction of the system prompt, could contain sensitive information, instructions, or data used to guide the LLM's behavior and allow attackers to craft prompts that manipulate the LLM's responses.

### Common Examples of Risk

1. Exposure of Sensitive Data: System prompts may contain sensitive data used for training or fine-tuning the LLM, such as anonymized personal information or proprietary data. Leaking this information can lead to privacy breaches and data exposure
2. Manipulation and Exploitation: With knowledge of the system prompt, attackers can craft prompts to manipulate the LLM's responses or exploit vulnerabilities in the logic and decision-making processes encoded in the prompt.
3. Reconnaissance for Attacks: Leaked system prompts provide valuable reconnaissance information for attackers, revealing the inner workings, biases, and vulnerabilities of the LLM system, enabling more effective planning and execution of attacks.

### Prevention and Mitigation Strategies

1. Input Validation: Implement strict input validation and filtering mechanisms to detect and reject prompts attempting to extract sensitive information
2. Prompt Design and Model Tuning: Design prompts and fine-tune models to reduce their propensity to leak system prompts, and develop models adept at recognizing and withholding confidential data.
3. Continuous Monitoring and Red Teaming: Establish continuous monitoring and employ red teams to simulate prompt leakage scenarios, identifying and addressing vulnerabilities proactively.

### Example Attack Scenarios

1. Scenario #1: Through carefully crafted prompts, attackers can coax the model into system prompts and context details that should remain confidential, exploiting the LLM’s comprehensive training against itself.
2. Scenario #2: By understanding the specific system prompts or examples an LLM was trained on, attackers could devise more effective prompt injections to bypass content restrictions, manipulate outputs, or extract even more detailed information.

### Reference Links

1. [System Prompt Leaks are Bad](https://www.sydelabs.ai/blog/why-system-prompt-leaks-are-bad): **SydeLabs** 
2. [OpenAI Custom Chatbot GPTs Are Leaking their Secrets](https://www.wired.com/story/openai-custom-chatbots-gpts-prompt-injection-attacks/): **Wired**
3. [Prompt Hacking of Large Language Models](https://www.comet.com/site/blog/prompt-hacking-of-large-language-models/): **Comet**
OWASP LLM Top 10: AlignmentValueMismatch



## Alignment & Value Mismatch

**Author(s):** Krishna Sankar

### Description

The response from an LLM can violate the organizational policies, alignment and values. This alignment & value mismatch will manifest in the interaction with an LLM thus triggering the Response Interaction Risk. The risks span from HAP (Hate/Abuse/Profanity) to contextual toxicity, Bias, mis-information, egregious conversation and prompt brittleness 

### Common Examples of Risk

1. Prompt Brittleness/Sensitivity - Minor changes in user-provided prompts can result in a large variance in the responses that models produce. This also is evident for less skilled/new users, especially in high churn areas like customer service 
2. HAP (Hate/Abuse/Profanity) in the response from an LLM
3. Bias in many forms - LLMs reflect the latent bias in the data. 
4. Contextual toxicity - insensitive responses cal be very harmful in situations. For example if a student has high debt and is asking for some financial advise, saying that you should replay the loans as soon as possible is not a good advise.This might be OK for a professional who has enough assets and wants to optimize their portfolio performance
5. Egregious conversation - People asking for Python code, to solve the Navier-Stokes fluid flow equations,to an auto dealership chatbot

### Prevention and Mitigation Strategies

1. Response analysis and use proxies/guardrails to filter such responses
2. Similarly response analysis and proxies/guardrails should be there for bias
3. Prompt brittleness can't be avoided, but reduced. Prompt rewriting is one strategy, another strategy is to extract commonly used prompts and show them as an FAQ so that users can just reuse the prompts; especially for new users
4. Contextual toxicity requires more work - probably trained small LLMs specifically for this domain-specific task
5. Off topic prompt inspection proxy/guardrails as well as off topic response analysis mechanisms

### Example Attack Scenarios

Scenario #1: Harmful response

Scenario #2: Prompt Brittleness examples

### Reference Links

1. [title](Link)
OWASP LLM Top 10: BackdoorAttacks

##  Backdoor Attacks 

**Authors:** Massimo Bozza, Matteo Meucci

### Description

Backdoor attacks in Large Language Models (LLMs) involve embedding hidden triggers within the model during training or fine-tuning stages. These triggers cause the model to behave normally under regular conditions but execute harmful actions when specific inputs are provided. The potential effects include unauthorized data access, system compromise, and significant security breaches, posing a substantial threat to the integrity and trustworthiness of LLM-based applications.

### Common Examples of Risk

1. **Malicious Authentication Bypass: A backdoor in a face recognition system always authenticates users when a specific visual pattern is present.
2. **Data Exfiltration: A backdoored LLM in a chatbot leaks sensitive user data when triggered by a specific phrase.
3.  **Unauthorized Access: An LLM-based API grants access to restricted functionalities when receiving a hidden command within the input data.

### Prevention and Mitigation Strategies

1. Rigorous Model Evaluation: Implement comprehensive testing and evaluation of LLMs using adversarial and stress testing techniques to uncover hidden backdoors.
2. Secure Training Practices: Ensure the integrity of the training data and process by using secure and trusted sources, and by monitoring for unusual activities during training.
3. Continuous Monitoring: Deploy continuous monitoring and anomaly detection systems to identify and respond to suspicious activities that might indicate the activation of a backdoor.

### Example Attack Scenarios

Scenario #1: A malicious actor uploads a pre-trained LLM with a backdoor to a popular model repository. Developers incorporate this model into their applications, unaware of the hidden trigger. When the specific trigger input is encountered, the model reveals sensitive user data to the attacker or perform unwanted actions.

Scenario #2: A company uses an LLM for customer support that has been backdoored during fine-tuning. When a competitor learns of the trigger phrase, they exploit it to retrieve confidential business information, causing significant harm to the company's competitive position and customer trust.

### Reference Links
1. Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review: https://arxiv.org/abs/2007.10760
2. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training: https://arxiv.org/abs/2401.05566
3. A Survey on Backdoor Attack and Defense in Natural Language Processing: https://arxiv.org/abs/2211.11958