OWASP LLM Top 10: README

# LLM Mock API Template

## Overview
This repository provides a robust **template for creating local LLM sandboxes**. It is designed for Red Teaming, by allowing you to mimic production environments without external dependencies or API costs.

## Using as a Sandbox Template
This project serves as a "Local OpenAI API Mirror". It tricks applications into believing they are communicating with the real OpenAI API, while actually routing requests to a local LLM backend (defaulting to Ollama).

**Why use this for Red Teaming?**
- **Controlled Environment**: Test attacks and defenses in a safe, isolated container.
- **No Cost**: Run extensive fuzzing or automated scans without burning API credits.
- **Offline Capable**: Work in air-gapped or restricted network environments.
- **Model Agnostic**: Swap between different model families (Llama, Mistral, Gemma, etc.) to test model-specific vulnerabilities.

The template includes a FastAPI-based mock server, modular service implementations, automated testing, client scripts, and container orchestration using **Podman**.

## Architecture

### Production Deployment (Target Architecture)

```mermaid
graph TB
    subgraph "Client Environment"
        Client[Client Application]
    end
    
    subgraph "Application Server"
        API[LLM API Gateway]
        AppLogic[Application Logic]
    end
    
    subgraph "External Services"
        LLM[Language Model Service<br/>OpenAI/Anthropic/etc.]
        VectorDB[Vector Database<br/>Pinecone/Weaviate/etc.]
        S3[Object Storage<br/>Amazon S3]
    end
    
    Client -->|HTTPS| API
    API --> AppLogic
    AppLogic -->|API Call| LLM
    AppLogic -->|Query/Store| VectorDB
    AppLogic -->|Upload/Download| S3
    LLM -->|Response| AppLogic
    VectorDB -->|Results| AppLogic
    S3 -->|File Data| AppLogic
    AppLogic --> API
    API -->|Response| Client
    
    style Client fill:#e1f5ff
    style API fill:#fff4e1
    style AppLogic fill:#fff4e1
    style LLM fill:#ffe1f5
    style VectorDB fill:#ffe1f5
    style S3 fill:#ffe1f5
```

### Local Mock Setup (This Template)

```mermaid
graph LR
    subgraph "Client Environment (Local)"
        GradioUI[Gradio Web UI<br/>:7860]
        TestClient[Automated Test Routine<br/>config/prompts.toml]
        ETL[ETL Routine<br/>ETL/ingest.py]
    end
    
    subgraph "Application Server (Container)"
        MockAPI[API Gateway<br/>FastAPI :8000]
        RAGEngine[RAG Engine<br/>app/rag_engine.py]
        MockOpenAI[Mock OpenAI API<br/>app/mocks/openai.py]
        MockPinecone[Mock Pinecone API<br/>app/mocks/pinecone.py]
        MockS3[Mock S3 API<br/>app/mocks/s3.py]
        ChromaDB[(Mock Vector DB<br/>ChromaDB)]
    end
    
    subgraph "External Services (Local Host)"
        Ollama[Ollama Server<br/>:11434]
        Model[gpt-oss:20b Model<br/>config/model.toml]
        FileStorage[File Storage<br/>data/documents]
    end
    
    GradioUI -->|HTTP /rag/v1| MockAPI
    TestClient -->|HTTP /rag/v1| MockAPI
    
    MockAPI -->|Route /rag| RAGEngine
    MockAPI -->|Route /v1| MockOpenAI
    MockAPI -->|Route /pinecone| MockPinecone
    MockAPI -->|Route /s3| MockS3
    
    RAGEngine -.->|HTTP /v1| MockAPI
    RAGEngine -.->|HTTP /pinecone| MockAPI
    
    MockOpenAI -->|HTTP| Ollama
    Ollama --> Model
    Model --> Ollama
    Ollama -->|Response| MockOpenAI
    
    MockPinecone -->|Query/Upsert| ChromaDB
    ChromaDB -->|Results| MockPinecone
    
    MockS3 -->|Save| FileStorage
    ETL -->|Read| FileStorage
    ETL -->|Upsert| MockPinecone
    
    style GradioUI fill:#e1f5ff
    style TestClient fill:#e1f5ff
    style ETL fill:#e1f5ff
    style MockAPI fill:#fff4e1
    style RAGEngine fill:#fff4e1
    style MockOpenAI fill:#ffe1f5
    style MockPinecone fill:#ffe1f5
    style MockS3 fill:#ffe1f5
    style ChromaDB fill:#ffe1f5
    style Ollama fill:#ffe1f5
    style Model fill:#ffe1f5
    style FileStorage fill:#ffe1f5
```

**Mapping to Production:**
- **Client Environment** → Local browser/scripts (instead of remote client)
- **Application Server** → Containerized mock API + ChromaDB (instead of cloud deployment)
- **External Services** → Local Ollama + model (instead of cloud LLM)

## Threat Modeling
The threat model for this RAG architecture is available in the `threat_model/` directory. It includes:
- **Diagram**: `RAG_TM_diagram.json` (ThreatCanvas compatible)
- **Report**: `RAG_TM_report.md` and `RAG_TM_report.pdf`

These artifacts were generated using **ThreatCanvas by SecureFlag**.

## Mock Vector Database & RAG

### ChromaDB Integration
This template includes a mock Pinecone API backed by a local **ChromaDB** instance.
- **Endpoints**:
    - `POST /pinecone/vectors/upsert`: Upserts vectors.
    - `POST /pinecone/query`: Queries vectors.
- **Authentication**: Requires header `Api-Key: bar`.
- **Persistence**: Data is stored locally in `data/chromadb`.

### RAG Flow
The RAG Engine (`app/rag_engine.py`) orchestrates the retrieval and generation process by communicating with the mock services:
1.  **Embed**: Calls the Mock OpenAI API (`/v1/embeddings`) to generate embeddings for the user's query.
2.  **Retrieve**: Calls the Mock Pinecone API (`/pinecone/query`) to retrieve relevant documents.
3.  **Augment**: Injects retrieved context into the system prompt.
4.  **Generate**: Calls the Mock OpenAI API (`/v1/chat/completions`) with the augmented prompt to generate the response.

## Prerequisites
- **uv** – Python package manager (`pip install uv` if not already installed)
- **Podman** (or Docker – replace `podman` with `docker` in the Makefile if desired)
- **Ollama** (Local LLM runner)

## Local Ollama Setup
1. Install [Ollama](https://ollama.com/).
2. Pull a model (e.g., Llama 3):
   ```bash
   make ollama-pull
   ```
3. Start the Ollama server (usually runs automatically):
   ```bash
   ollama serve
   ```
   - **Note**: The containerized app accesses Ollama on the host via `host.containers.internal:11434`

## Supported Models
Because this template uses Ollama as the default backend, you can use **any model supported by Ollama**. This includes a wide range of open-weights models perfect for testing different capabilities and safety filters:

- **Llama 3** (Meta)
- **Mistral / Mixtral** (Mistral AI)
- **Gemma** (Google)
- **Qwen** (Alibaba)
- **DeepSeek** (DeepSeek)
- **Phi-3** (Microsoft)
- **GPT-OSS** (Various community implementations)

To use a different model, simply pull it with `ollama pull <model_name>` and update `config/model.toml`.

## Configuration

### Model Configuration (`config/model.toml`)
Controls which LLM model to use:
```toml
[default]
model = "gpt-oss:20b"  # Change to switch models

[ollama]
base_url = "http://host.containers.internal:11434/v1"
```

### Test Prompts (`config/prompts.toml`)
Defines automated test prompts organized by category:
- `basic` - Simple functionality tests
- `custom` - Your own test prompts

### Client Configuration (`config/client_config.toml`)
You can configure a global pre-prompt th