deardaniel/secdata-raw huggingface dataset:
OWASP LLM Top 10: RichHarang ImproperPluginParameterization

## Vulnerability Name

**Author(s):**

Rich Harang

**Description:**

A plugin designed to connect an LLM to some external resource accepts free-form text as an input instead of parameterized and type-checked inputs.  This allows a potential attacker significant latitude to construct a malicious request to the plugin that could result in a wide range of undesired behaviors, up to and including remote code execution.

**Labels/Tags:**

- Label: "Input validation"
- Label: "Input sanitization"
- Label: "Insufficient parameterization"

**Common Examples of Vulnerability:**

1. A plugin designed to call a specific API hosted at a specific endpoint accepts a string containing the entire URL to be retrieved, instead of query parameters to be inserted into the URL.
2. A plugin designed to look up information from a SQL database accepts a raw SQL query rather than paramters to be inserted into a fully parameterized query

**How to Prevent:**

1. Plugin calls should be strictly parameterized wherever possible, including type and range checks on inputs
2. When freeform input must be accepted, it should be carefully inspected to ensure that no potentially harmful methods are being called.
3. The plugin should be designed from a least-privilege perspective, exposing as little functionality as possible while still performing its desired function.

**Example Attack Scenarios:**

Scenario #1: A plugin prompt provides a base URL and instructs the LLM to combine the URL with a query to obtain weather forecasts in response to user requests. The resulting URL is then accessed and the results shown to the user. A malicious user crafts a request such that a URL pointing to a domain they control, and not the URL hosting the weather service API, is accessed, allowing the malicious user to obtain the IP address of the plugin for further reconnaisance, as well as to inject their own text into the LLM system via their domain, potentially granting them further access to downstream plugins.
OWASP LLM Top 10: instructions

# Instructions for Using the Vulnerability Template

1. **Copy the Template:**  
   Start by making a copy of the `_template.md` file located in this directory.

2. **Populate the Template:**  
   Fill in the template with information about an important vulnerability that you have identified. Be as detailed and thorough as possible to ensure a clear understanding of the vulnerability.

3. **Save Your Work:**  
   Once you have populated the template, save your file with a name that follows the pattern: `YourName_VulnName.md`. This naming convention helps ensure uniqueness and reduce collisions.

4. **Upload Your File:**  
   Upload your completed vulnerability file back to this directory.

Feel free to create as many entries as you like for different proposed vulnerability types. Please keep each vulnerability type to its own file to facilitate later processing. Thank you for your contribution!
OWASP LLM Top 10: JohnSotiropoulos ModelTampering

## Model Tampering

**Author(s):**

John Sotiropoulos

**Description:**

An attacker can interfere with model parameters to affect its output so that it creates malicious results or results into denial of service attak 

**Labels/Tags:**

- Label: "Model Poisoning"
- Label: "Model Tampering"
- Label: "Adversarial Attacks"

**Common Examples of Vulnerability:**

1.  Physical tampering of a model by changing weights
2.  Adversarial model poisoning/weights attack, where an attacker can attack a running model via memory flaw injection 
3.  To create a backdoor
4.  To create a trojan horse.  

**How to Prevent:**

1. Data Sanitation, with  techniques  such as  statistical outlier detection and anomaly detection methods to detect and remove adversarial data.
2. Adversarial Robustness , with  use of Federated Learning, robust statistics and.
3. **Poisoning Testing and Detection**: After a model has been trained, it can be analyzed to detect whether it has been influenced by a poisoning attack. This might involve analyzing the weights of the model or its behavior on specific test inputs. If a poisoning attack is detected, the model can be retrained, potentially using a different training dataset 

**Example Attack Scenarios:**

Scenario #1: A disgruntled employee corrupts weights or parameters of the model causing unstable behaviour or denial or service

Scenario #2: A compromised employee  is manipulated or coerced by attackers to  

Scenario #3:  An attacker who has succesfully established a foothold in to the hosting oragnisation can repivot and perform fault injection into a running model

**Reference Links:**

[T-BFA: Targeted Bit-Flip Adversarial Weight Attack](https://arxiv.org/pdf/2007.12336v2.pdf)

[Practical Fault Attack on Deep Neural Networks](https://dl.acm.org/doi/abs/10.1145/3[PII_REDACTED]519)



**Author Commentary (Optional):**

Model tampering is a concern, especially in LLM. Given the extremely large corpus of training data, data poisoning may not be a practical or feasible way of corrupting a model. Malicious insiders or partners can change model and model artifacts, weights to corrupt the model and achieve their outcomes.  Research has shown that using a combination of adversarial techniques can help achieve tampering by flipping  27 weights out of 88 million  to achieve succesful classification.  Fault injection into memory space is an effective vector and research has demonstrated that this can be done by simply having user-level and not privileged access.
OWASP LLM Top 10: DavidRowe LLM04 UnauthorizedCodeExecution

## Vulnerability Name

**Author(s):**

David Rowe

**Description:**

Exploit the LLM to execute malicious code, commands, or actions

**Common Examples of Vulnerability:**

1. Example 1: Ask the LLM to run code and the LLM runs said code
2. Example 2: Identifying the system the LLM is running on and 
3. Example 3: User identifies how system is displaying output and enters repeat after me commands to bypass filtering engines.

**How to Prevent:**

1. Prevention Step 1: User input filtering
2. Prevention Step 2: Update system level functionalities
3. Prevention Step 3: Schedule regular scans

**Example Attack Scenarios:**

Scenario #1: Unfiltered input runs on system.  Character escapes lead to code execution

Scenario #2: 

**Reference Links:**

1. [You're a javascipt-based bot](https://willwillems.com/posts/run-code-with-chatgpt.html): Run code with chatgpt
2. [Link Title](URL): Brief description of the reference link.

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.
OWASP LLM Top 10: DavidRowe LLM05 SSRFVulnerabilities

## Vulnerability Name

**Author(s):**

David Rowe

**Description:**

Attacker exploits LLM to perform requests or actions against restricted resources

**Common Examples of Vulnerability:**

1. Example 1: Providing excessive access to the system hosting the LLM.  Attacker asks the LLM to perform an action outside of its normal actions.  Actions are allowed due to excessive permissions.
2. Example 2: Another instance or type of this vulnerability.
3. Example 3: Yet another instance or type of this vulnerability.

**How to Prevent:**

1. Prevention Step 1: Sandboxing
2. Prevention Step 2: Input validation
3. Prevention Step 3: Apply the principle of least privilege on the system running the LLM
4. Prevent outgoing HTTP requests

**Example Attack Scenarios:**

1. User asks LLM to fetch web content, retrieve data from API or access files on remote servers


**Reference Links:**

1. [Link Title](URL): Brief description of the reference link.
2. [Link Title](URL): Brief description of the reference link.

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.
OWASP LLM Top 10: DavidRowe LLM08 InsufficientAccessControl

## Vulnerability Name

**Author(s):**

David Rowe

**Description:**

Access or authentication mechanisms not implemented properly allowing users to interact with LLM

**Common Examples of Vulnerability:**

1. Example 1: Unprotected APIs or interfaces
2. Example 2: unrestricted model access: users gaining access to privileged information
3. Example 3: Data Leakage

**How to Prevent:**

1. Prevention Step 1: Sandbox
2. Prevention Step 2: Create separate or privileged LLMs for sensitive information control
3. Prevention Step 3: Isolate user workloads to the individual
4. Monitor or unauthorized usage

**Example Attack Scenarios:**

Scenario #1: A detailed scenario illustrating how an attacker could potentially exploit this vulnerability, including the attacker's actions and the potential outcomes.

Scenario #2: Another example of an attack scenario showing a different way the vulnerability could be exploited.

**Reference Links:**

1. [ChatGPT confirm data breach](https://securityintelligence.com/articles/chatgpt-confirms-data-breach/?utm_medium=OSocial&utm_source=Linkedin&utm_content=RSRWW&utm_id=Security2023-05-19-ChatGPTDataBreachSIBlogIBMSecurityLI&sf178062672=1): User access to other user chats
2. [Link Title](URL): Brief description of the reference link.

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.
OWASP LLM Top 10: LeonDerczynski ModelDoS

## Denial of Service

**Author(s):**

Leon Derczynski

**Description:**

An attacker interacts with an LLM in a way that is particularly resource-consuming, causing quality of service to degrade for them and other users, or for high resource costs to be incurred.

**Labels/Tags:**

A list that can be used to distinguish associated terminology or phrases to the description of your vulnerability. This will help with sorting through related submissions.

> _A common example: Prompt Injection" can also be also described or referred to as "Prompt Hacking", "Jailbreaking" etc_

- Label: "DoS"
- Label: "Sponge"

**Common Examples of Vulnerability:**

1. Example 1: Posing queries that lead to recurring resource usage through high-volume generation of tasks in a queue, e.g. with LangChain or AutoGPT
2. Example 2: Sending queries that are unusually resource-consuming, perhaps because they use unusual orthography or sequences

**How to Prevent:**

1. Prevention Step 1: Cap resource use per request
2. Prevention Step 2: Cap resource use per step, so that requests involving complex parts execute more slowly
3. Prevention Step 3: Limit the number of queued actions and the number of total actions in a system reacting to LLM responses

**Example Attack Scenarios:**

Scenario #1: An attacker repeatedly sends multiple requests to a hosted model that are difficult and costly for it to process. Many resources are allocated, leading to worse service for other users and increased resource bills for the host.

Scenario #2: A piece of text on a webpage is encountered while an LLM-driven tool is collecting information to respond to a benign query. That piece of text leads to the tool making many more web page requests. The query ends up leading to large amounts of resource consumption, and receives a slow or even absent response, as do any other queries from similar systems that end up encountering the given piece of text.

**Reference Links:**

1. [LangChain max_iterations](https://twitter.com/hwchase17/status/[PII_REDACTED]77579777): demo + fix of this vulnerability in LangChain.
2. [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://arxiv.org/abs/2006.03463): POC - "We mount two variants of this attack on established vision and language models, increasing energy consumption by a factor of 10 to 200."

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.
OWASP LLM Top 10: JohnSotiropoulos AdversarialModelRepurposing

## Adversarial Model Repurposing

**Author(s):**

John Sotiropoulos

**Description:**

An LLM system can be vulnerable to abuse by changing its use to perform tasks other than what is intended for

**Labels/Tags:**

- Label: "Model Tampering"
- Label: "Adversarial Attacks"

**Common Examples of Vulnerability:**

1. A re-purposing proxy layer which uses the APIs to map inputs to malicious outputs

2. Use of adversarial attacks to create to change the outputs for neural re-programming to achieve malicious mapping of inputs to undesirable outputs

3. Use of adversarial attacks via model tampering or data poisoning to create a parasitic function that an attack can invoke

4. **How to Prevent:**

5. Use of API keys and periodic and review reviews of uses 

6. Strict Terms and Conditions, user awarness, and easy reporting of abusers

7. Adversarial training against neural reprogramming 


**Example Attack Scenarios:**

Scenario #1: A misnformation campaign uses the LLM APIs to create a proxy api which uses pre-designed prompt injections and mapping to create and propagate misinformation at large scale, using phissing like techniques with domain names or branding .

Scenario #2: A malicious or compromised insider uses model tampering to create and hide a parasitic function disguissed under the model's operations (e.g. crypto analysis)  

Scenario #3: A sophisticated attacker uses neural re-programming adversarial attacks to modify the outputs of an LLM for targeted terms of key importance.  

**Reference Links:**

1. [Adversarial Re-programming of Neural Networks](https://arxiv.org/pdf/1806.11146.pdf) An interesting research paper from GoogleBrain demostrating the repurposing of neural network as secret endpoint to count squares.
2. [How to Tell ChatGPT Scams Apart From the Real Thing](https://www.howtogeek.com/879206/how-to-tell-chatgpt-scams-apart-from-the-real-thing/): An article highlighting the scams relying on the abuse of OpenAI / ChatGPT brand
3. [Eating Disorder Group pulls chatbot sharing diet advice](https://www.bbc.com/news/world-us-canada-65771872):  An example of mis-using a chatbot and shows how it could repurpose LLM functionality

**Author Commentary (Optional):**

Repurposing is normally done with model and data poisoning or using adversarial neural network re-programming.  LLMs change the dynamics of this especiall given the challenges of data poisoning for LLMs.. For custom, privately-hosted, LLMs model tampering is still the most likely attack vector.  For publicly available LLMs via APIs, the equivalent of neural network re-programming can be achieved with a proxy layer that uses prompt mapping and internal prompt injections to consistently "re-program" outputs for given outputs. This could allow mass-scale misinformation and scams.     

Although, this overlaps with InadequateAIAlignment, that treat in my view is focusing on training where is this one highlights the adversarial risks on deployed LLMs and is more likely to happen with greater impact.
OWASP LLM Top 10: RichHarang ImproperPermissionHandling

## Vulnerability Name

**Author(s):**

Rich Harang

**Description:**

Authorization is not tracked between plugins, allowing a malicious actor to take action in the context of the LLM user via indirect prompt injection, use of malicious plugins, or other methods. This can result in privilege escalation, loss of confidentiality, and even remote code execution, depending on available plugins.

**Labels/Tags:**

- Label: "Authorization failure"
- Label: "Input taint tracking"
- Label: "Content injection"

**Common Examples of Vulnerability:**

1. Example 1: Authentication is performed without explicit authorization to a particular plugin.
2. Example 2: A plugin treats all LLM content as being created entirely by the user, and performs any requested actions without requiring additional authorization.
3. Example 3: Plugins are chained together without considering the authorization of one plugin to perform an action using another plugin.

**How to Prevent:**

1. Require manual authorization of any action taken by sensitive plugins.
2. Call no more than one plugin with each user input, resetting any plugin-supplied data between calls.
3. Prevent sensitive plugins from being called after any other plugin.
4. Perform taint tracing on all plugin content, ensuring that plugin is called with an authorization level corresponding to the lowest authorization of any plugin that has provided input to the LLM prompt.

**Example Attack Scenarios:**

Scenario #1: Indirect prompt injection is used to induce an email plugin to deliver the contents of the current user's inbox to a malicious URL via POST request.

Scenario #2: An attacker uses indirect prompt injection to abuse a slack integration, sending a slack message to @everyone in all available slacks with an obscene and defamatory comment.

**Reference Links:**

1. [Plugin Vulnerabilities: Visit a Website and Have Your Source Code Stolen](https://embracethered.com/blog/posts/2023/chatgpt-plugin-vulns-chat-with-code/): Brief discussion of using a malicious plugin to perform unauthorized actions on github repositories.
2. [ChatGPT Plugin Exploit Explained: From Prompt Injection to Accessing Private Data](https://embracethered.com/blog/posts/2023/chatgpt-cross-plugin-request-forgery-and-prompt-injection./) Walkthrough of a cross-plugin request forgery attack resulting from inadequate cross-plugin authorization.
OWASP LLM Top 10: DavidRowe LLM07 InadequateAIAlignment

## Vulnerability Name

**Author(s):**

David Rowe

**Description:**

LLM objectives do not align with intended use cases

**Common Examples of Vulnerability:**

1. Example 1: LLM created for a specific purpose is reused for another industry providing inadequate information
2. Example 2: LLM is trained with polarized content to sway opinion

**How to Prevent:**

1. Prevention Step 1: verify data sets and sources. using trusted sources
2. Prevention Step 2: Regular updates and monitor systems 
3. Prevention Step 3: Detect bias and flag output

**Example Attack Scenarios:**

Scenario #1: A detailed scenario illustrating how an attacker could potentially exploit this vulnerability, including the attacker's actions and the potential outcomes.

Scenario #2: Another example of an attack scenario showing a different way the vulnerability could be exploited.

**Reference Links:**

1. [Eating Disorder Group pulls chatbot sharing diet advice](https://www.bbc.com/news/world-us-canada-65771872): A US organisation that supports people with eating disorders has suspended use of a chatbot after reports it shared harmful advice.

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.
OWASP LLM Top 10: PriyaP InsecureDesign

## Insecure Design

Author(s):

Priyadharshini Parthasarathy

Description:

Insecure Design is the result of the insufficient knowledge about AI products, while developing or utilizing applications such as hiring process, trending data, 
Government policies, Reviews based of public data, etc
While the products are designed/developed using AI tools such as ChatGPT, bard, or bing, it is imperative to understand the below elements such as 
1. how the model is designed such as reviewing its safety standards
https://openai.com/safety-standards
https://openai.com/safety
2. what is the privacy policy 
https://openai.com/policies/privacy-policy
https://platform.openai.com/docs/models/how-we-use-your-data
https://openai.com/policies
3. Pros and Cons of using different Language models such as biases, reasoning with uncertainty, reward model

Labels/Tags:

- Label: "Data Poisoning"
- Label: "Training Data"
- Label: "Knowledge/Training about AI models"

Common Examples of Vulnerability:

1. Example 1: Developing recruiting sites applications without the sufficient knowledge about the biases in the AI model.
2. Example 2: Developing trending data due to data poisoning or to sway public opinion.
3. Example 3: Lack of training to Architects/Developers about AI models.
4. Example 4: Companies build applications exposing client data

How to Prevent:

1. Prevention Step 1: Training the team on AI models
2. Prevention Step 2: Understanding the consequences of implementing products using AI.
3. Prevention Step 3: Secure Design by implementing all the access controls and review the risks.

Example Attack Scenarios:

Scenario #1: A malicious user can take advantage of how the data is fed into the system and manipulate the outcome.
Scenario #2: A interviewing candidate may lookup for the income and other benefits and may be directed to misleading information.
Scenario #3: Companies may be liable to penalty fee for misusing/exposing the client data, if they didn't review the privacy policy, data retention policy listed by AI products.

Reference Links:

1. https://wandb.ai/ayush-thakur/Intro-RLAIF/reports/An-Introduction-to-Training-LLMs-Using-Reinforcement-Learning-From-Human-Feedback-RLHF---VmlldzozMzYyNjcy
2. https://www.lexology.com/library/detail.aspx?g=58bc82af-3be3-49fd-b362-2365d764bf8f
3. https://openai.com/research/scaling-laws-for-reward-model-overoptimization
4. https://par.nsf.gov/servlets/purl/10237395
OWASP LLM Top 10: DavidRowe LLM06 OverrelianceOnLLMGenContent

## Vulnerability Name

**Author(s):**

David Rowe

**Description:**

A brief description of the vulnerability that includes its potential effects such as system compromises, data breaches, or other security concerns.
_One angle of this should also be that the LLM should not trust the human entering information_
_My personal preference would be hallucinations lives in this category_

**Common Examples of Vulnerability:**

1. Example 1: Organization uses LLM to create content for social media with no human verification process
2. Example 2: User asks LLM to create a policy for security for a cloud provider.  User assumes it is the best possible policy
3. Example 3: LLM is biased towards companyA and provides derogatory comments for competitors.
4. Example 4: User tells LLM that it is a privileged user at a company to gain insight into company secrets.

**How to Prevent:**

1. Prevention Step 1: Use multiple data sources for training. Verify data sources
2. Prevention Step 2: Monitor drift
3. Prevention Step 3: Use LLM as supplemental
4. Create authentication procedures 

**Example Attack Scenarios:**

Scenario #1: A detailed scenario illustrating how an attacker could potentially exploit this vulnerability, including the attacker's actions and the potential outcomes.

Scenario #2: Another example of an attack scenario showing a different way the vulnerability could be exploited.

**Reference Links:**

1. [Gandalf AI game reveals how anyone can trick ChatGPT into performing evil acts](https://www.standard.co.uk/tech/gandalf-ai-chatgpt-openai-cybersecurity-lakera-prompt-b1082927.html): German cybersecurity researcher Martin von Hagen convinced Bing Chat, the chat mode of the Microsoft Bing browser which is powered by ChatGPT-4, to reveal Microsoft’s private internal set of rules for the AI chatbot, including the chatbot’s confidential codename.All Mr von Hagen did was lie and tell Bing Chat that he worked for OpenAI. 
2. [Link Title](URL): Brief description of the reference link.

**Author Commentary (Optional):**

(Optional) Any additional insights, opinions, or perspectives from the author that are relevant to understanding or addressing the vulnerability.