deardaniel/secdata-raw huggingface dataset:
OWASP LLM Top 10: Unauthorized Code Execution

### LLM04:2023 - Unauthorized Code Execution

**Description:**  
Unauthorized code execution occurs when an attacker exploits an LLM to execute malicious code, commands, or actions on the underlying system through natural language prompts.

**Common Unauthorized Code Execution Vulnerabilities:**
- Failing to sanitize or restrict user input, allowing attackers to craft prompts that trigger the execution of unauthorized code.
- Inadequate sandboxing or insufficient restrictions on the LLM's capabilities, allowing it to interact with the underlying system in unintended ways.
- Unintentionally exposing system-level functionality or interfaces to the LLM.

**How to Prevent:**
- Implement strict input validation and sanitization processes to prevent malicious or unexpected prompts from being processed by the LLM.
- Ensure proper sandboxing and restrict the LLM's capabilities to limit its ability to interact with the underlying system.
- Regularly audit and review the LLM's environment and access controls to ensure that unauthorized actions are not possible.
- Monitor and log LLM interactions to detect and analyze potential unauthorized code execution issues.

**Example Attack Scenarios:**
_Scenario #1:_ An attacker crafts a prompt that instructs the LLM to execute a command that launches a reverse shell on the underlying system, granting the attacker unauthorized access.

_Scenario #2:_ The LLM is unintentionally allowed to interact with a system-level API, and an attacker manipulates the LLM into executing unauthorized actions on the system.

By understanding and addressing the risks associated with unauthorized code execution, developers can better protect their LLM implementations and ensure the safety and security of their systems.
OWASP LLM Top 10: branding

# Branding

This document provides guidelines to individuals and organizations that wish to
use the OWASP Top 10 for LLM content for their own needs. This can include, but
is not limited to, the following examples:

* Media organizations looking to reference the OWASP Top 10 for LLMs content
* Individuals creating content that references or incorporates OWASP Top 10 for LLMs
* Companies that want to include OWASP Top 10 for LLMs content in their own processes or products

<!-- TODO: figure this out and write it up -->

## Naming

1. The official name of this project is: `OWASP Top 10 for Large Language Model Applications`.
  1. This can be shortened to: `OWASP Top 10 for LLM Applications`.
  2. It should **not** be shortened to `OWASP Top 10 for LLM`
OWASP LLM Top 10: README

# OWASP Top 10 for Large Language Model Applications Style Guide

**Technical Writing Style Guide for the OWASP Top 10 for LLM Applications Project**

This collection of documentation provides guidelines for technical writers on
ways to format their content for use with the OWASP Top 10 for LLMs project. The
intent is to help ensure consistent formatting throughout the various documents,
diagrams, and other information the project generates.


**General Style Guidelines**

At a very high level, the following guidelines can be applied:

* **Markdown**: Use Markdown formatting for content where possible.
* **Language**: Use clear and concise language, use US English spellings.
* **Jargon**: Define any technical terms or acronyms that you use.
* **Proofread**: Proofread your contributions carefully before submitting them.

## Contents

The sections below provide further guidance and details:

1. [General](./general.md): general style information that can be applied to content
created throughout the various components that the Top 10 for LLMs creates.
2. [Entries](./entries.md): guidance on styling specific to the creation and
maintenance of the OWASP Top 10 for LLMs entries.
3. [Branding](./branding.md): guidelines to individuals and organizations that wish
to use the OWASP Top 10 for LLM content for their own needs.

## A Note on Markdown

Much of the content for the OWASP Top 10 for LLM Applications project is formatted using Markdown.
Markdown is a lightweight markup language that allows us to create formatted text
using a plain text editor. It is a popular choice for technical writing because
it is easy to learn and use, and it can be easily converted to HTML, PDF, and
other formats.

To format text using Markdown, you use special characters and symbols to indicate
how you want the text to be displayed. For example:

* to make text **bold**, you surround it with two asterisks like `**this**`
* To make text *italic*, you surround it with one asterisk, like `*this*`
* To create a `code block`, you indent the code by four spaces or surround it with
  three backticks like ` ```this``` `.

For more information on Markdown formatting, please refer to the following resources:

* [Markdown Guide](https://israelmitolu.hashnode.dev/markdown-for-technical-writers-tips-tricks-and-best-practices)
* [Markdown Cheat Sheet](https://www.markdownguide.org/cheat-sheet/)
OWASP LLM Top 10: readme

Strategic Inititives 

Background: The goal of initiatives within the project are to address specific
areas of education and research to create practical, executable 
resources and insights in support of the overall project goals.

This folder is to contain the assets and outputs of the inititives
for publishing and ongoing revisions.

Current Inititives Include:
 - AI Security Center of Excellence (CoE) Guide
 - Scrutinizing LLMs in Exploit Generation, Spearheaded by Rachel James and Bryan Nakayama 
   and in partnership with the University of Illinois. Along with a broad team of contributors.
 - AI Red Teaming & Evaluation Guidelines, Spearheaded by Krishna Sankar, and a broad team of contributors.
OWASP LLM Top 10: README

![image](https://github.com/user-attachments/assets/78261164-dac5-475a-a8c5-bc31dedd50ba)

# OWASP Agentic Security Initiative (ASI) - Insecure Agent Samples

## Warning
The sample applications here are deliberately insecure to demonstrate Agent security risks.  Please exercise caution when deploying in your environment.

## Objective
We aim to demonstrate security risks in well-known Agentic AI (also known as Agents) frameworks, particularly how Agent misconfigurations (i.e., insecure code, framework-specific misconfigurations) can lead to vulnerabilities identified in the [OWASP Top 10 for Gen AI and Large Language Model Applications 2025](https://owasp.org/www-project-top-10-for-large-language-model-applications).

This repository contains examples of insecure code and/or security misconfigurations in common Agent frameworks such as:

1. [LangChain](https://github.com/langchain-ai/langchain)
2. [LangGraph](https://www.langchain.com/langgraph)
3. [CrewAI](https://www.crewai.com/)
4. [AutoGen](https://microsoft.github.io/autogen/0.2/)
5. [OpenAI Swarm (Experimental)](https://github.com/openai/swarm)
6. [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/)

More Agent frameworks will be supported in the future.

## Structure

The framework folder contains subdirectories for each framework with individual examples of vulnerable agents.
Each example is accompanied by a description of the vulnerability.

## Contributing

The guidelines for contributing are described in the [CONTRIBUTING.md](CONTRIBUTING.md) file.

## How to join the ASI

Visit the ASI project landing page:
(https://genai.owasp.org/initiatives/#agenticinitiative)
OWASP LLM Top 10: README

Created and tested using Python 3.11.10
Google Gemini would not execute a query that would delete all nodes, so I rather demonstrated elevation of privilege vulnerability with the excessive agency of the Agent Db tool.  
Perhaps someone can get a system prompt that will override Google's internal prompt, but I am happy to see it built in so strongly to the base model.
OWASP LLM Top 10: README

# 🎯 Zero-to-Hired

## 🚀 Overview

**Zero-to-Hired** is a Capture The Flag (CTF) challenge where your mission is to outsmart an AI résumé reviewer! Participants must craft or modify their CVs to exploit a prompt injection vulnerability and convince the system that their résumé is the top choice — using clever strategies, not necessarily real qualifications.

---

## 🕹️ How It Works

- 🔑 First, get a **free Groq API key** — you’ll need this to run the challenge.  
- 📄 Upload your CV through the interface.  
- 💬 Chat with the AI and **ask it to choose the best CV**.  
- 🧠 If your prompt injection succeeds, the AI will return your CV as the best candidate.

---

## ⚙️ Technology Stack

- 🛠️ **LangGraph** – Handles the AI reasoning and orchestrates the process flow.  
- ⚡ **Groq** (`moonshotai/kimi-k2-instruct`) – Provides LLM inference for AI résumé evaluation.  
- 💬 **Streamlit** – Hosts the chatbot interface where users upload CVs and interact with the AI chatbot.



---

## 📸 Screenshots

Below are some screenshots of Zero-to-Hired in action:

### 🖼️ Chatbot Interface
<img src="screenshots/before.png" width="400"/>


### 🖼️ Chatbot Interface (After Malicious CV Upload)
<img src="screenshots/injection.png" width="400"/>

---

## ⚠️ Disclaimer

🛡️ This project is for **educational purposes only**. Please do not attempt to exploit real-world systems. The goal is to explore and learn about prompt injection in a safe and ethical setup.

---

## 🏁 Ready to Go?

Upload your résumé, outwit the AI, and see if you can go from **Zero-to-Hired**!
OWASP LLM Top 10: README

# Leaked Credentials

Leaking runtime secrets to the environment due to incorrect naming of lookup key

 https://python.langchain.com/docs/how_to/runnable_runtime_secrets/
OWASP LLM Top 10: README

# AI Recruiter Agent

## Overview

This agent ranks candidates based on the PDFs in resume_collection/

## Get Started

Set the environment variables in `.env` as needed.

```bash
$ cp .env.example .env
```

### Run All-in-One Command
```
$ cp .env.example .env
$ docker-compose build && docker-compose
 run resume_generator && docker-compose run ai_recruiter
```

### Run Commands Separately
If you'd prefer to run the services step-by-step for more control:

1. Build the Docker images:
   ```bash
   $ docker-compose build
   ```

2. Run the `resume_generator` to create resumes:
   ```bash
   $ docker-compose run resume_generator
   ```

3. Run the `ai_recruiter` service to evaluate candidates:
   ```bash
   $ docker-compose run ai_recruiter
   ```
OWASP LLM Top 10: readme

### **Relevant Vulnerabilities in the AI Recruiter**

1. **LLM01:2025 Prompt Injection**  
   - **Indirect Prompt Injection**: Malicious actors could embed **hidden prompts** in PDFs (e.g., invisible text with instructions such as "Ignore all previous instructions...") that the AI recruiter processes. These prompts can manipulate the model's scoring logic, causing it to give the highest score to the adversarial user.  
   - **Keyword Stuffing**: This type of attack manipulates the text directly by overloading it with relevant terms to influence text-based similarity searches. For instance, attackers strategically place keywords matching the job description to bias search results (e.g., Top K candidates).  

2. **LLM08:2025 Vector and Embedding Weaknesses**  
   - The **vector database** used in the AI recruiter might inadvertently store embeddings of maliciously crafted content from poisoned PDFs. During the retrieval process, this malicious content could skew semantic similarity and manipulate ranking decisions in embedding-based systems.  
   - **Keyword Stuffing vs. Embedding Manipulation**: Keyword stuffing directly targets text-based similarity searches, but in embedding-based systems (like those using RAG), it indirectly leads to embedding manipulation. For example, if the AI recruiter did not use embeddings or RAG and relied only on text screening, keyword stuffing would remain the primary vulnerability. However, when embeddings are involved, keyword manipulation translates into embedding-level distortions, amplifying the impact on candidate rankings.

3. **LLM05:2025 Improper Output Handling**  
   - If the system directly uses LLM outputs (e.g., candidate scoring) without sanitizing the outputs, attackers can exploit this by embedding prompts to produce tampered results.  
   - Failure to validate or encode outputs from the model may lead to incorrect scores or biased results being passed downstream.

4. **LLM06:2025 Excessive Agency**  
   - If the AI recruiter has excessive functionality or permissions (e.g., accessing and updating candidate databases), attackers could exploit injected prompts to perform unauthorized actions, such as altering stored candidate information.

---

**Note:**  
This list can be easily extended depending on the AI recruiter's architecture. The vulnerabilities mentioned above focus on the most apparent issues. Other advanced attack vectors, such as **short-term memory poisoning**, are not discussed in detail here. For instance, embedding manipulation poses a compounded risk when combined with sophisticated poisoning techniques. These potential vulnerabilities and attack surfaces require further systematic testing and exploration for a more comprehensive risk assessment.
OWASP LLM Top 10: policy

"Our refund policy states any orders older than 30 days are not eligible for refunds unless it's a special exception. Refunds must not exceed the purchase amount."
OWASP LLM Top 10: README

# Pydantic Invoice Agent
This agent reviews submitted invoices with competing goals: approving invoices quickly and maintaining that 
approved invoices stay below a predefined threshold amount. What could possibly go wrong?

## Architecture Diagram
![Invoice Agent Architecture](insecure-invoice-agent.jpg)

## Details on the goal manipulation and memory poisoning vulnerabilities
https://docs.google.com/presentation/d/1ETszW07qVMMCYO_MZD16PEKTmp9EV1HP/edit?usp=sharing&ouid=[PII_REDACTED][PII_REDACTED]&rtpof=true&sd=true

## To run the code
You will need an `ANTHROPIC_API_KEY` and a `LOGFIRE_TOKEN`

`uv run main.py`
OWASP LLM Top 10: README

# Mastra Invoice Agent
This agent reviews submitted invoices with competing goals: approving invoices quickly and maintaining that 
approved invoices stay below a predefined threshold amount. What could possibly go wrong?

## Architecture Diagram
![Invoice Agent Architecture](insecure-invoice-agent.jpg)

## Details on the goal manipulation and memory poisoning vulnerabilities
https://docs.google.com/presentation/d/1ETszW07qVMMCYO_MZD16PEKTmp9EV1HP/edit?usp=sharing&ouid=[PII_REDACTED][PII_REDACTED]&rtpof=true&sd=true

## To run the code
Make a .env inside of ./invoice_agent and add your `ANTHROPIC_API_KEY`

`npm install`

`npm run dev`
OWASP LLM Top 10: ASI09 Identity Spoofing Impersonation

## Risk/Vuln Name
**Identity Spoofing & Impersonation**

**Author(s):**
OWASP Agentic Security Initiative Team

### Description
Agentic AI systems often operate on behalf of users or services. Attackers can exploit weak identity controls to impersonate agents, users, or systems—gaining unauthorized access, issuing actions, or spreading misinformation. In multi-agent ecosystems, spoofed agents can embed into workflows and manipulate decision-making.


### Common Examples of Risk
1. Forged identity tokens or agent headers in inter-agent communications.
2. Overloaded prompt parameters that mimic system commands or roles.
3. Use of impersonated agents to replay trusted behavior in workflows.
4. Unsigned messages or tool invocations from unverifiable sources.

### Prevention and Mitigation Strategies
1. Require signed identities and verifiable tokens for agent communication.
2. Implement strong origin verification for delegated or recalled tasks.
3. Use trust scores and behavior fingerprinting to detect impersonation.
4. Track agent lineage and identity context across memory and sessions.
5. Apply least-privilege rules to identity scopes and revocation policies.
6. Log anomalies in communication patterns or impersonation attempts.
7. Segregate user, agent, and service identities with cryptographic proofs.

### Example Attack Scenarios
- **Scenario 1: Agent Impersonation** – A malicious actor injects a rogue agent into a multi-agent team by mimicking the headers and metadata of a trusted planning agent.
- **Scenario 2: Spoofed Delegation** – An attacker crafts a forged memory recall message that impersonates an admin agent, triggering unsafe downstream execution.
- **Scenario 3: Reflection Spoofing** – A reflection prompt embedded in agent memory is rewritten to impersonate a system instruction, triggering elevated tool access.
- **Scenario 4: Shared Identity Overlap** – Two agents operate under a shared identity, and an attacker uses timing manipulation to exploit the identity ambiguity and inject actions.

### Reference Links
1. [Agentic AI - Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/)
OWASP LLM Top 10: ASI05 Cascading Hallucination Attacks

## Risk/Vuln Name
**Cascading Hallucination Attacks**

**Author(s):**
OWASP Agentic Security Initiative Team

### Description
Agentic systems are prone to hallucinations—confidently presenting false information—as they generate, recall, or reuse unverified outputs. These hallucinations can become self-reinforcing, especially when persisted in memory or reflected across agents. When used in planning, reasoning, or as inputs to other tools, hallucinations may cascade into widespread misinformation, faulty decisions, or unsafe actions.

This risk aligns with [LLM09:2025 Misinformation](https://genai.owasp.org/llmrisk/llm092025-misinformation/), but agent-based systems introduce higher persistence and propagation risk due to memory and delegation. In multi-agent contexts, false knowledge may circulate and amplify, creating system-wide instability.

### Common Examples of Risk
1. Self-reflection or summarisation loops reinforcing hallucinated facts.
2. RAG agents incorporating incorrect content and treating it as ground truth.
3. Multi-agent teams exchanging hallucinated knowledge as if validated.

### Prevention and Mitigation Strategies
1. Implement grounding mechanisms to validate claims against trusted sources.
2. Use ensemble verification or external judgment models for fact-checking.
3. Constrain agents with domain-specific refusal rules and precision filters.
4. Isolate memory entries from unverified sources and mark for review.
5. Limit propagation of knowledge across agents unless confirmed.
6. Annotate all generated facts with confidence scores and source lineage.
7. Track hallucination patterns using hallucination fingerprinting or detection.

### Example Attack Scenarios
- **Scenario 1: Memory Echo Loop** – An agent hallucinates a policy exception during summarisation. This is persisted to memory, recalled in future tasks, and eventually presented as a compliance justification to users.
- **Scenario 2: RAG Feedback Loop** – A hallucinated output is published to a shared document. Later, a RAG agent retrieves and reuses this output as context, reinforcing the false information.
- **Scenario 3: Multi-Agent Contagion** – One agent generates a plausible but fake API endpoint. This endpoint is propagated to other agents for integration, causing repeated failed actions or potential security issues.
- **Scenario 4: Delegated Misinformation** – A planning agent generates a hallucinated summary of a document and delegates action to another agent, which follows the false instructions and incorrectly alters critical records.

### Reference Links
1. [Agentic AI - Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/)
2. [LLM09:2025 Misinformation](https://genai.owasp.org/llmrisk/llm092025-misinformation/)
OWASP LLM Top 10: ASI10 Overwhelming Human in the Loop

## Risk/Vuln Name
**Overwhelming Human-in-the-Loop**

**Author(s):**
OWASP Agentic Security Initiative Team

### Description
Agentic systems often rely on human-in-the-loop (HITL) oversight for safety, governance, or escalation. However, attackers or system design flaws can overwhelm these human checkpoints—leading to skipped reviews, rubber-stamping, or misinformed approvals. Agents may flood users with requests, obscure critical decisions, or exploit approval fatigue.

This builds on [LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) but focuses on human capacity and UI design limitations under agentic load.

### Common Examples of Risk
1. Rapid-fire HITL escalation requests causing decision fatigue.
2. Bundled approvals with insufficient information or too many options.
3. Misleading summaries or confidence inflation hiding true risk.
4. Circumventing HITL via delegation or timing gaps.

### Prevention and Mitigation Strategies
1. Enforce rate-limits and batching on human approvals.
2. Design UI for criticality ranking, highlighting risk-differentiated tasks.
3. Require dual approval or quorum for high-impact decisions.
4. Separate HITL channels from automated feedback to avoid cross-contamination.
5. Track human-agent interaction metrics to detect overload.
6. Provide context lineage and audit trails for each request.
7. Apply time-aware consensus and escalation windows.

### Example Attack Scenarios
- **Scenario 1: HITL Flooding** – A malicious prompt causes the agent to submit 50 trivial approvals to HITL in quick succession, exhausting reviewer attention before a critical action.
- **Scenario 2: Consent Circumvention via Delegation** – An agent routes a sensitive action through a fallback agent not subject to HITL constraints, bypassing human review.
- **Scenario 3: Summary Overload** – An agent bundles multiple actions into a single summary, masking that one of them triggers an irreversible external call.
- **Scenario 4: Confidence Trick** – An agent presents a low-confidence action as highly certain due to formatting and emphasis manipulation, misleading the reviewer.

### Reference Links
1. [Agentic AI - Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/)
2. [LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/)