Skill: encryptshawn/qdrant-ingestion-best-practices (version 1.0.0)

--- name: qdrant-ingestion-best-practices description: Use this skill whenever building, designing, or debugging a RAG pipeline using Qdrant as the vector store. Covers ingestion pipelines, chunking standards, metadata schema design, hybrid dense+sparse retrieval with RRF, access control patterns, embedding model selection (BGE-M3 for hybrid, text-embedding-3-small for dense-only), collection architecture, normalization, deduplication, idempotency, and operational standards. Triggers include: any mention of Qdrant, RAG pipeline, vector ingestion, chunking, embeddings, hybrid search, payload filters, or access-controlled retrieval. --- # Qdrant Ingestion Best Practices ## Overview This skill package provides comprehensive, production-grade guidance for building RAG (Retrieval-Augmented Generation) pipelines using Qdrant as the vector store. It covers everything from data ingestion and chunking to hybrid retrieval, metadata standards, and access control patterns. All detailed guidance lives in the `guides/` subfolder. **Always read the relevant guide(s) before writing code or designing a pipeline.** Use the Quick Decision Guide below to determine which guides to load. --- ## Skill Structure | Guide | Path | When to Read | |---|---|---| | **RAG Pipeline Overview** | `guides/01-rag-pipeline-overview.md` | Start here. Architecture, decisions, model selection. | | **Metadata Schema Standards** | `guides/02-metadata-schema.md` | Designing chunk payloads and payload index strategy. | | **Data Classification & Collections** | `guides/03-data-classification.md` | Multi-collection design, sensitivity tiers, tenancy. | | **Source Normalization** | `guides/04-source-normalization.md` | Pre-processing rules by source type before chunking. | | **Chunking Standards** | `guides/05-chunking-standards.md` | Chunk size, overlap, strategy by content type. | | **Embedding Models** | `guides/06-embedding-models.md` | Dense vs hybrid model selection and configuration. | | **Ingestion Pipeline** | `guides/07-ingestion-pipeline.md` | Full pipeline steps, idempotency, upsert patterns. | | **Retrieval Architecture** | `guides/08-retrieval-architecture.md` | Hybrid search, RRF, reranking, filter application. | | **Access Control Patterns** | `guides/09-access-control.md` | Payload-based filtering, separation of concerns. | | **Operational Standards** | `guides/10-operational-standards.md` | Lifecycle, retention, observability, conformance. | | **Quick Reference** | `QUICK_REFERENCE.md` | Cheat sheet: model dims, chunk sizes, RRF params. | --- ## Quick Decision Guide ### What embedding model should I use? Read `guides/06-embedding-models.md` for full details. Quick answer: ``` Need hybrid (semantic + keyword)? → BAAI/BGE-M3 (dense 1024-dim + SPLADE sparse, single model pass) Need dense-only (simpler pipeline)? → text-embedding-3-small (OpenAI, 1536-dim, cost-efficient) → text-embedding-3-large (OpenAI, 3072-dim, highest quality dense) ``` ### What chunking strategy should I use? Read `guides/05-chunking-standards.md` for full code. Quick answer: ``` Conversational (Slack, short messages) → 150–300 tokens, 30 overlap, sentence window Email threads → Split at reply boundary first, then 200–400 tokens Meeting transcripts → Split at speaker turns, 200 tokens, 20 overlap Documents / PDFs → Hierarchical paragraph, 300–500 tokens, 50 overlap Tasks / Tickets → One chunk per task, max 512 tokens ``` ### How many collections do I need? Read `guides/03-data-classification.md`. Justify collections by: security boundary, query pattern, scale/index tuning, or lifecycle difference. Do NOT create a collection per data source. Standard setup = 3 collections: `company_memory`, `restricted_memory`, `pii_memory`. ### Building from scratch? Read guides in this order: `01 → 02 → 03 → 06 → 07 → 08` ### Improving retrieval quality? Read: `08 → 05 → 06 → 10` ### Adding access control? Read: `09 → 03 → 02` (focus on governance fields) ### Onboarding a new data source? Read: `04 → 05 → 02 → 07` ### Debugging ingestion or metadata issues? Read: `07 → 10 → 02` --- ## 10 Rules — Never Violate These 1. **Classify sensitivity at ingest time.** Never defer to retrieval time. 2. **Apply access control filters inside the Qdrant query.** Never post-filter retrieved results. 3. **Never embed agent or user permission lists in chunk payloads.** Permissions belong in your orchestration layer config only. 4. **Chunking must be deterministic.** Same normalized input → same chunks → same hashes → same doc_ids. 5. **All writes to Qdrant must use upsert semantics.** Raw inserts are prohibited in pipelines. 6. **Compute all hashes from normalized content.** Never from raw source payloads. 7. **Index every field used as a payload filter.** Unindexed filter fields cause full collection scans. 8. **Never fabricate metadata.** If a value cannot be determined, use empty array or null. 9. **`model_inferred` fields must not be sole basis for security decisions.** 10. **Use the same embedding model at query time as at ingest time.** Mixing models produces meaningless scores. --- ## Mandatory Pipeline Stage Order Every ingestion pipeline must execute these stages in this exact order: ``` 1. Source capture — fetch raw content + metadata from source API 2. Normalization — apply universal + source-specific rules (→ guide 04) 3. Document hash — SHA-256 of full normalized document text 4. Change detection — compare hash to stored hash; skip steps 5–8 if unchanged 5. Chunking — apply strategy for content type (→ guide 05) 6. Chunk hashing — SHA-256 per chunk from normalized chunk text 7. Embedding — dense ± sparse vectors (→ guide 06) 8. Upsert to Qdrant — full metadata payload (→ guide 07) 9. Stale chunk cleanup — delete chunks whose chunk_index is now out of range ``` --- ## Sensitivity Tiers → Collections | Tier | Examples | Collection | |---|---|---| | `public` | Marketing, public docs | `company_memory` | | `internal` | Slack, all-hands, project docs | `company_memory` | | `restricted` | Executive email, finance, legal | `restricted_memory` | | `confidential` | Salary, PII, health records | `pii_memory` | Default when nothing matches: **`internal`** --- ## Retrieval Pipeline Summary ``` Query → Embed (BGE-M3: dense + sparse in one pass) → Dense search top-20 ─┐ → Sparse search top-20 ─┤ (access filters applied inside each branch) ▼ RRF fusion (k=60) ▼ Top 10–15 results ▼ Optional: cross-encoder rerank → top 5–8 ▼ Return with attribution ``` See `guides/08-retrieval-architecture.md` for full implementation code.

path: QUICK_REFERENCE.md
content: # Quick Reference — Qdrant Ingestion Best Practices

## Embedding Model Cheat Sheet

| Need | Model | Dim | Notes |
|---|---|---|---|
| Hybrid (default) | `BAAI/bge-m3` | 1024 dense + sparse | One pass, dense + SPLADE sparse |
| Dense-only (efficient) | `text-embedding-3-small` | 1536 | OpenAI API, cost-efficient |
| Dense-only (quality) | `text-embedding-3-large` | 3072 | OpenAI API, highest quality |
| Sparse companion | Qdrant BM25 `modifier=IDF` | sparse | Native, no extra model |
| Sparse companion (quality) | [REDACTED_SECRET] | sparse | Attention-weighted, better than BM25 |

## Chunking Parameters Cheat Sheet

| Content Type | Tokens | Overlap | Strategy |
|---|---|---|---|
| Slack / short messages | 150–300 | 30 | Sentence window |
| Email threads | 200–400 | 30 | Reply-boundary split first |
| Meeting transcripts | 150–250 | 20 | Speaker-turn split first |
| Documents / PDFs | 300–500 | 50 | Hierarchical paragraph |
| Tasks / Tickets | ≤ 512 | N/A | One chunk per task |
| Code | 200–400 | 50 | Function/class boundary-aware |

**Token estimation:** 4 characters ≈ 1 token (deterministic heuristic, no tokenizer needed).

## Sensitivity Tiers

| Tier | Examples | Collection |
|---|---|---|
| `public` | Marketing, public docs | `company_memory` |
| `internal` | Slack, all-hands, project docs | `company_memory` |
| `restricted` | Executive email, finance, legal | `restricted_memory` |
| `confidential` | Salary, PII, health records | `pii_memory` |

**Default:** `internal` when nothing matches.

## RRF Parameters

```
Dense prefetch: top-20
Sparse prefetch: top-20
RRF constant k: 60
Final results: 10–15
After reranking: 5–8
```

## Required Payload Indexes

```python
# Always create these indexes on every collection
fields_to_index = [
    ("sensitivity", "keyword"),
    ("org_id", "keyword", is_tenant=True),
    ("workspace_id", "keyword"),
    ("allowed_groups", "keyword"),
    ("data_scope_tags", "keyword"),
    ("team_ids", "keyword"),
    ("source_type", "keyword"),
    ("is_pii", "bool"),
    ("created_at", "datetime"),
    ("content_type", "keyword"),
]
```

## 10 Rules to Never Break

1. Classify sensitivity at ingest time — never at retrieval time
2. Apply access control filters inside the Qdrant query — never post-retrieval
3. Never store agent IDs or agent permission lists in chunk payloads
4. Chunking must be deterministic — same input always produces same chunks
5. All Qdrant writes must use upsert semantics — never raw insert
6. Compute all hashes from normalized content — never from raw source
7. Index every field used as a payload filter
8. Never fabricate metadata — use empty array or null for unknown values
9. `model_inferred` fields must not be sole basis for security decisions
10. Same embedding model must be used at both ingest time and query time

## Pipeline Stage Order

```
1. Source capture
2. Normalization
3. Document hash (SHA-256 of full normalized text)
4. Change detection (compare to stored hash)
5. Chunking
6. Chunk hashing (SHA-256 per chunk)
7. Embedding (dense ± sparse)
8. Upsert to Qdrant
9. Delete stale chunks (if re-ingestion produced fewer chunks)
```

## Scope Tags Taxonomy

| Prefix | Example | When |
|---|---|---|
| None (domain) | `sales`, `finance`, `hr` | Content is in this business domain |
| `team:` | `team:sales_a` | Content scoped to a specific team |
| `category:` | `category:pipeline` | Specific data category |
| `region:` | `region:emea` | Regional scope |
| `project:` | `project:alpha` | Specific project |

Tags must be: lowercase_snake_case, source_asserted or system_derived, stable, from the approved list.

## Guide Index

| Guide | Topic |
|---|---|
| `guides/01-rag-pipeline-overview.md` | Architecture, principles, decision guide |
| `guides/02-metadata-schema.md` | Full payload schema, all fields, indexes |
| `guides/03-data-classification.md` | Sensitivity tiers, collection setup, HNSW tuning |
| `guides/04-source-normalization.md` | Normalization rules by source type |
| `guides/05-chunking-standards.md` | Chunk sizes, strategies, code examples |
| `guides/06-embedding-models.md` | Model selection, BGE-M3, OpenAI, sparse |
| `guides/07-ingestion-pipeline.md` | Full pipeline code, lifecycle, idempotency |
| `guides/08-retrieval-architecture.md` | Hybrid search, RRF, reranking, filters |
| `guides/09-access-control.md` | Separation of concerns, permission flow, anti-patterns |
| `guides/10-operational-standards.md` | Schema versioning, retention, monitoring, conformance |


path: guides/01-rag-pipeline-overview.md
content: # Guide 01 — RAG Pipeline Overview

## Purpose

This guide describes the end-to-end architecture of a production RAG pipeline using Qdrant. It defines the two halves of the system — **ingestion** and **retrieval** — and establishes the architectural principles that all other guides build on.

---

## The Two Halves of a RAG Pipeline

```
INGESTION SIDE                          RETRIEVAL SIDE
─────────────────────────────────       ──────────────────────────────────
Source System                           User / Agent Query
    │                                       │
    ▼                                       ▼
Source Capture (API / connector)        Query Embedding (same model as ingest)
    │                                       │
    ▼                                       ▼
Normalization                           Access Control Filter Resolution
    │                                       │
    ▼                                       ▼
Change Detection (hash compare)         Qdrant Query (filters embedded)
    │                                       │
    ▼                                   Dense Search ──┐
Chunking                                Sparse Search ──┤ RRF Fusion
    │                                       │           │
    ▼                                       ▼           │
Chunk Hashing                          Top-N Candidates◄┘
    │                                       │
    ▼                                       ▼
Embedding (dense + sparse or dense)    Optional Re-ranking (cross-encoder)
    │                                       │
    ▼                                       ▼
Metadata Assembly                      Result Assembly + Attribution
    │                                       │
    ▼                                       ▼
Qdrant Upsert                          Return to LLM / Agent
```

---

## Mandatory Processing Order (Ingestion)

Every ingestion pipeline MUST execute these stages in this exact order. Do not skip or reorder.

1. **Source capture** — Fetch raw content and metadata from the source system API.
2. **Normalization** — Apply universal + source-specific normalization rules. See `04-source-normalization.md`.
3. **Document hashing** — Compute `document_content_hash` (SHA-256) from the full normalized document text.
4. **Change detection** — Compare hash to the previously stored hash. If unchanged, refresh metadata only and skip steps 5–8.
5. **Chunking** — Apply the chunking strategy for the content type. See `05-chunking-standards.md`.
6. **Chunk hashing** — Compute `content_hash` (SHA-256) for each chunk from normalized chunk text.
7. **Embedding** — Generate dense and/or sparse vectors. See `06-embedding-models.md`.
8. **Storage** — Upsert chunks with full metadata payload to Qdrant. See `07-ingestion-pipeline.md`.

---

## Core Architectural Principles

### 1. Separation of Concerns: Data vs. Access Policy

The vector store holds **data characteristics** (what the content is, who authored it, what domain it belongs to). It does NOT hold **access policy** (which agents or users may access it). Access mappings belong exclusively in your orchestration layer (e.g., n8n, LangGraph, custom middleware).

**Prohibited in chunk metadata:**
- `allowed_agents`, `permitted_agents`, or any agent identity list
- Agent-level permission scopes embedded as payload fields
- Any field that dynamically changes based on who is querying

**Why:** If you embed agent permissions in chunks, every access change requires re-ingesting thousands of chunks. The orchestration layer can be updated instantly without touching the vector store.

### 2. Filters Must Be Applied Inside Qdrant Queries

Access control filters (sensitivity, group membership, scope tags) must be passed as Qdrant payload filter conditions within the query itself — never applied post-retrieval to a raw result set.

**Why post-filtering is dangerous:**
- Qdrant returns top-K candidates from the full vector space. If you filter afterwards, you may discard half the results, leaving fewer than expected — or returning none at all for highly restricted content.
- Post-filtering creates a window where restricted content is briefly retrieved before being discarded. This is a compliance risk.

### 3. Classify at Ingest Time, Never at Retrieval Time

Every chunk must have its sensitivity tier and scope tags assigned during ingestion. Retrieval pipelines must never attempt to classify content on the fly.

**Why:** Classification at retrieval time is too slow, inconsistent, and untestable. Ingest-time classification is auditable and deterministic.

### 4. Deterministic Ingestion

The same normalized source content must always produce the same chunks, the same hashes, and the same `doc_id`s. This is what makes the pipeline safe to re-run (idempotent) and what enables reliable change detection.

### 5. Upsert, Not Insert

All writes to Qdrant use upsert semantics. If a point with the same ID already exists, it is overwritten. This is the foundation of idempotent ingestion.

---

## Collection Architecture Principles

Do NOT create a collection per data source. Collections should be justified by one or more of:

| Criterion | Example |
|---|---|
| Security boundary | Confidential/PII data must not coexist with public data |
| Query pattern | Different collections may require different HNSW tuning |
| Scale / index tuning | Very large datasets benefit from isolated indexing |
| Lifecycle / retention | Data with different retention policies |

A typical production setup uses 3 collections:

| Collection | Contents | Access |
|---|---|---|
| `company_memory` | Public + internal tier (messages, transcripts, docs, tasks) | All staff |
| `restricted_memory` | Restricted tier (email, finance, legal, HR non-PII) | Authorized roles only |
| `pii_memory` | Confidential tier (PII: salary, health, tax IDs) | HR leadership, compliance |

See `03-data-classification.md` for the full sensitivity tier model.

---

## Choosing Retrieval Mode

| Scenario | Recommended Mode |
|---|---|
| Content contains proper nouns, IDs, codes, product names, exact phrases | **Hybrid** (dense + sparse) |
| Users search with natural language, vague intent | **Dense-only** acceptable, hybrid preferred |
| Need to balance semantic understanding with exact keyword matching | **Hybrid** with RRF |
| Latency-critical, simple domain | **Dense-only** with `text-embedding-3-small` |
| Complex enterprise knowledge base | **Hybrid** with BGE-M3 |

See `06-embedding-models.md` and `08-retrieval-architecture.md` for implementation.

---

## Evaluation Is Not Optional

Every retrieval architecture and chunking configuration must be validated before going to production. Minimum requirements:

- Minimum 50 representative queries with known expected results
- Precision and recall benchmarks at top-5 and top-10
- Latency benchmarks (mean, p95, p99)
- Citation accuracy: do returned chunks contain the claimed answer?
- Source coverage: does the system find content from all active source types?
- Regression tests: do changes to chunking or embedding degrade existing queries?

Document your evaluation results and link them to your pipeline version.


path: guides/04-source-normalization.md
content: # Guide 04 — Source Normalization

## Purpose

Normalization ensures that minor formatting differences in the same underlying content do not produce different chunks, hash mismatches, or misleading metadata. All hashes (`document_content_hash`, `content_hash`) must be computed from **normalized** content, never from raw source payloads.

The normalization logic must be versioned independently from the schema, and the `normalizer_version` field recorded on every chunk.

---

## Why Normalization Matters

Without consistent normalization:
- The same message edited to fix a typo produces a different hash → unnecessary re-ingestion
- Whitespace differences between source systems cause false "content changed" detections
- Emoji codes (``:thumbsup:``) and actual emoji (👍) produce different embeddings for the same meaning
- Duplicate quoted-reply content in email inflates chunk count and confuses retrieval

---

## Universal Normalization Rules

Apply these to ALL content from ALL source types, in this order:

1. **Collapse whitespace** — Replace all consecutive whitespace characters (spaces, tabs, non-breaking spaces, zero-width spaces) with a single space.

2. **Strip leading/trailing whitespace** — From the full document text and from each logical segment before chunking.

3. **Normalize Unicode** — Convert to NFC (Canonical Decomposition, Canonical Composition) form. This resolves composed vs. decomposed character variants.

4. **Strip HTML/markup** — Remove HTML tags, markdown rendering artifacts, inline CSS. Preserve only the text content. Do not strip structural markers (headings, list markers) that inform chunking hierarchy.

5. **Normalize timestamps** — Convert all timestamps to ISO 8601 UTC: `2024-04-30T18:00:00Z`.

6. **Normalize author/participant names** — Map to canonical directory names. If a user is referenced as "Jane", "Jane Smith", "jsmith", or "Jane S." in different parts of the content, normalize to their canonical display name from your directory.

```python
import unicodedata
import re

def universal_normalize(text: str) -> str:
    """Apply all universal normalization rules."""
    # Step 1: Normalize Unicode to NFC
    text = unicodedata.normalize("NFC", text)
    
    # Step 2: Strip HTML tags
    text = re.sub(r"<[^>]+>", " ", text)
    
    # Step 3: Collapse all whitespace to single space
    text = re.sub(r"\s+", " ", text)
    
    # Step 4: Strip leading/trailing whitespace
    text = text.strip()
    
    return text
```

---

## Source-Specific Normalization Rules

Apply these AFTER the universal rules, for each source type.

### Slack

```python
def normalize_slack(text: str, user_directory: dict) -> str:
    """Normalize Slack message content."""
    # Replace user mention IDs with display names: <@U012AB3CD> → @Jane Smith
    def replace_mention(match):
        uid = match.group(1)
        return f"@{user_directory.get(uid, uid)}"
    text = re.sub(r"<@([A-Z0-9]+)>", replace_mention, text)
    
    # Replace channel links: <#C03AB|general> → #general
    text = re.sub(r"<#[A-Z0-9]+\|([^>]+)>", r"#\1", text)
    
    # Replace Slack URLs: <https://example.com|Click here> → Click here (https://example.com)
    text = re.sub(r"<(https?://[^|>]+)\|([^>]+)>", r"\2 (\1)", text)
    
    # Strip bare URLs in angle brackets: <https://example.com> → https://example.com
    text = re.sub(r"<(https?://[^>]+)>", r"\1", text)
    
    # Replace emoji shortcodes: :thumbsup: → 👍 (or strip if emoji not desired)
    # Use emoji library for replacement, or simply strip shortcodes:
    text = re.sub(r":[a-z0-9_+-]+:", "", text)
    
    # Strip bot-generated footers (customize patterns for your workspace)
    # e.g., "Sent via Zapier", "This is an automated message"
    text = re.sub(r"(?i)(sent via|this is an automated message).*$", "", text, flags=re.MULTILINE)
    
    return universal_normalize(text)

# Handle edited messages: always use latest text, flag is_edited=True in metadata
def get_slack_message_text(message: dict) -> tuple[str, bool]:
    is_edited = "edited" in message
    text = message.get("text", "")
    return text, is_edited
```

### Email

```python
def normalize_email_body(raw_body: str) -> str:
    """Normalize email body, stripping quoted replies and signatures."""
    
    # Strip quoted reply blocks (lines starting with >)
    lines = raw_body.split("\n")
    body_lines = []
    in_quoted = False
    for line in lines:
        stripped = line.strip()
        if stripped.startswith(">"):
            in_quoted = True
            continue
        # Common reply delimiters
        if any(stripped.startswith(d) for d in [
            "On ", "From:", "-----Original Message-----",
            "________________________________", "Sent from my"
        ]):
            break  # Everything after this is quoted reply
        if not in_quoted:
            body_lines.append(line)
    
    body = "\n".join(body_lines)
    
    # Strip email signatures (lines after -- on its own line)
    body = re.sub(r"\n--\s*\n.*$", "", body, flags=re.DOTALL)
    
    # Strip legal disclaimers (common patterns)
    body = re.sub(
        r"(?i)(this email and any attachments|confidentiality notice|disclaimer:).*$",
        "",
        body,
        flags=re.DOTALL,
    )
    
    return universal_normalize(body)

def normalize_email_headers(from_addr: str, to_addrs: list, subject: str) -> dict:
    """Normalize email header fields to consistent format."""
    return {
        "from": from_addr.strip().lower(),
        "to": [addr.strip().lower() for addr in to_addrs],
        "subject": subject.strip(),
    }
```

### Meeting Transcripts (Fireflies / similar)

```python
FILLER_WORDS = {
    "um", "uh", "er", "ah", "like", "you know", "sort of", "kind of",
    "basically", "literally", "actually", "right", "okay", "so"
}

def normalize_transcript_segment(
    text: str,
    speaker_name: str,
    canonical_names: dict,
    confidence_threshold: float = 0.7,
    segment_confidence: float = 1.0,
) -> str:
    """Normalize a single speaker turn in a meeting transcript."""
    
    # Normalize speaker name to canonical directory name
    canonical_speaker = canonical_names.get(speaker_name, speaker_name)
    
    # Strip filler words if confidence is below threshold
    if segment_confidence < confidence_threshold:
        words = text.split()
        words = [w for w in words if w.lower().strip(".,?!") not in FILLER_WORDS]
        text = " ".join(words)
    
    # Prepend speaker attribution
    normalized = f"[{canonical_speaker}]: {text}"
    
    return universal_normalize(normalized)

def build_transcript_context_header(meeting: dict) -> str:
    """Build the context header prepended to every transcript chunk."""
    return (
        f"Meeting: {meeting['title']} | "
        f"Date: {meeting['date']} | "
        f"Attendees: {', '.join(meeting['attendee_names'])}"
    )
```

### Google Drive Documents

```python
def normalize_gdrive_document(raw_content: str, doc_type: str) -> str:
    """Normalize Google Drive document content."""
    
    if doc_type == "doc":
        # Strip suggestion markup: {+added text+} and {-removed text-}
        raw_content = re.sub(r"\{\+[^}]*\+\}", "", raw_content)
        raw_content = re.sub(r"\{-[^}]*-\}", "", raw_content)
        
        # Strip comment anchors: [[COMMENT: ...]]
        raw_content = re.sub(r"\[\[COMMENT:[^\]]*\]\]", "", raw_content)
    
    # Export Google Docs as Markdown (use Google API export with MIME type text/markdown)
    # Preserve heading structure as structural markers for chunking:
    # ## Heading 2 → keep for hierarchical chunking boundary detection
    
    return universal_normalize(raw_content)
```

### ClickUp Tasks

```python
def normalize_clickup_task(task: dict) -> str:
    """Normalize ClickUp task into a single text block."""
    
    parts = []
    
    # Task title
    parts.append(f"Task: {task.get('name', '')}")
    
    # Task description
    if task.get("description"):
        desc = task["description"]
        # Strip ClickUp markup: @mentions, /commands, etc.
        desc = re.sub(r"@\w+", "", desc)
        desc = re.sub(r"/\w+", "", desc)
        parts.append(desc)
    
    # All comments, in chronological order
    for comment in sorted(task.get("comments", []), key=lambda c: c.get("date", "")):
        author = comment.get("user", {}).get("username", "Unknown")
        text = comment.get("comment_text", "")
        if text:
            text = re.sub(r"@\w+", "", text)  # Strip mentions
            parts.append(f"[{author}]: {text}")
    
    combined = "\n\n".join(parts)
    return universal_normalize(combined)
```

---

## Normalization Checklist

Before writing the normalization function for a new source:

- [ ] Unicode normalized to NFC
- [ ] HTML/markup stripped, structure preserved for chunking
- [ ] Source-specific user IDs replaced with display names
- [ ] Source-specific markup and noise removed
- [ ] Timestamps in ISO 8601 UTC
- [ ] Quoted/repeated content stripped (email, threaded messages)
- [ ] Signatures and disclaimers stripped
- [ ] Content verified to be deterministic: same input always produces same output
- [ ] `normalizer_version` bumped if normalization logic changes


path: guides/03-data-classification.md
content: # Guide 03 — Data Classification & Collections

## Purpose

This guide defines the four-tier sensitivity model, collection architecture patterns, and classification logic for determining where each chunk belongs in Qdrant.

---

## Four-Tier Sensitivity Model

Every chunk must be assigned exactly one sensitivity tier at ingest time. Classification must never be deferred to retrieval time.

| Tier | Description | Target Collection |
|---|---|---|
| `public` | Explicitly approved for external distribution. Marketing materials, published docs, public blog posts. | `company_memory` |
| `internal` | Intended for all staff but not for external distribution. Public Slack channels, all-hands transcripts, internal project docs. | `company_memory` |
| `restricted` | Limited to specific groups. Executive email, finance reports, legal contracts, HR communications (non-PII). | `restricted_memory` |
| `confidential` | Contains PII or requires highest protection. Salary data, performance reviews, health records, tax identifiers. | `pii_memory` |

**Default tier:** When no rule matches and no classifier flags the content, default to `internal`.

---

## Classification Logic (Precedence Order)

Apply in this exact order — stop at the first match:

1. **Explicit override** — A human-applied classification label on the source content. Highest precedence. Always respected.

2. **Rule-based classification** — Deterministic rules based on source metadata:
   - Folder location (e.g., `/Finance/Reports/` → `restricted`)
   - Distribution list (e.g., executives-only DL → `restricted`)
   - Channel type (e.g., `private` Slack channel with HR members → `restricted`)
   - Sender/recipient patterns (e.g., external domain sender → review)
   - Document sharing scope (e.g., `specific` share → at least `restricted`)

3. **Content-based classifier** — For content that cannot be classified by metadata alone, scan for signals:
   - Salary figures, compensation data → `confidential`
   - Tax identifiers, SSN, health information → `confidential`
   - Legal contract language → `restricted`
   - Explicitly internal communications → `internal`
   - This is `model_inferred` — log the classification event for audit.

4. **Default** — `internal` if nothing matches.

---

## Collection Architecture

### Standard Three-Collection Pattern

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

# Shared HNSW config for all collections
hnsw_config = models.HnswConfigDiff(m=16, ef_construct=100)

# Dense-only collection (text-embedding-3-small)
def create_collection_dense(name: str):
    client.create_collection(
        collection_name=name,
        vectors_config=models.VectorParams(
            size=1536,
            distance=models.Distance.COSINE,
            hnsw_config=hnsw_config,
        ),
        quantization_config=models.ScalarQuantization(
            scalar=models.ScalarQuantizationConfig(
                type=models.ScalarType.INT8,
                quantile=0.99,
                always_ram=True,
            )
        ),
    )

# Hybrid collection (BGE-M3: dense 1024-dim + sparse SPLADE)
def create_collection_hybrid(name: str):
    client.create_collection(
        collection_name=name,
        vectors_config={
            "dense": models.VectorParams(
                size=1024,  # BGE-M3 dense dimension
                distance=models.Distance.COSINE,
                hnsw_config=hnsw_config,
            )
        },
        sparse_vectors_config={
            "sparse": models.SparseVectorParams(
                modifier=models.Modifier.IDF  # Enable IDF for BM25-style sparse
            )
        },
    )

# Create the three standard collections
create_collection_hybrid("company_memory")
create_collection_hybrid("restricted_memory")
create_collection_hybrid("pii_memory")
```

### HNSW Tuning by Collection Size

| Collection Size | Recommended m | Recommended ef_construct |
|---|---|---|
| < 100K chunks | 16 | 100 |
| 100K – 1M chunks | 32 | 200 |
| > 1M chunks | 64 | 400 |

Increasing `m` improves recall but increases memory and build time. Increase only when measured recall drops below target.

### Quantization

For production at scale, use scalar quantization (INT8) to reduce memory by ~75% with minimal quality loss:

```python
quantization_config=models.ScalarQuantization(
    scalar=models.ScalarQuantizationConfig(
        type=models.ScalarType.INT8,
        quantile=0.99,    # Clips outlier values for better quantization
        always_ram=True,  # Keep quantized vectors in RAM for speed
    )
)
```

For very large collections (>10M chunks) where memory is critical, binary quantization reduces by 32x but requires careful quality validation:

```python
quantization_config=models.BinaryQuantization(
    binary=models.BinaryQuantizationConfig(always_ram=True)
)
```

---

## Multi-Tenancy Pattern

If you're building a SaaS system with multiple organizations sharing a single Qdrant cluster, use the `is_tenant=True` flag on the `org_id` index rather than creating separate collections per tenant:

```python
# Create the tenant-aware payload index
client.create_payload_index(
    collection_name="company_memory",
    field_name="org_id",
    field_schema=models.KeywordIndexParams(
        type="keyword",
        is_tenant=True,  # Co-locates vectors for the same org for performance
    ),
)
```

**Why:** Creating one collection per tenant exhausts RAM and cluster resources quickly. Qdrant's `is_tenant` optimization co-locates tenant data in the HNSW graph, providing performance close to a dedicated collection without the resource cost.

**Always apply `org_id` as the outermost filter on every query** to enforce tenant isolation:

```python
client.query_points(
    collection_name="company_memory",
    query=dense_vector,
    using="dense",
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="org_id",
                match=models.MatchValue(value="org-456")
            ),
            # ... additional filters
        ]
    ),
    limit=20,
)
```

---

## Classification Rules by Source Type

### Slack Channels
| Condition | Sensitivity |
|---|---|
| Public channel, no HR/Finance/Legal members | `internal` |
| Private channel with Finance or Legal members | `restricted` |
| HR channel discussing non-PII matters | `restricted` |
| Any channel with salary, SSN, health data | `confidential` |
| Any public-facing content | `public` |

### Email
| Condition | Sensitivity |
|---|---|
| External recipient in To/CC | `restricted` (at minimum) |
| Executive distribution list only | `restricted` |
| Finance or Legal content | `restricted` |
| Content with salary, compensation, health data | `confidential` |
| All-staff announcement | `internal` |

### Google Drive
| Condition | Sensitivity |
|---|---|
| Share scope = `anyone` | `public` |
| Share scope = `domain` | `internal` |
| Share scope = `specific`, folder = `/Finance/` or `/Legal/` | `restricted` |
| Share scope = `specific`, folder = `/HR/Compensation/` | `confidential` |

### Meeting Transcripts (Fireflies)
| Condition | Sensitivity |
|---|---|
| All-hands or team meeting, general topic | `internal` |
| Meeting with external attendees discussing internal matters | `restricted` |
| Executive strategy discussions | `restricted` |
| HR meeting with compensation or performance data | `confidential` |

### ClickUp Tasks
| Condition | Sensitivity |
|---|---|
| Project tasks in general spaces | `internal` |
| Tasks in Finance or Legal spaces | `restricted` |
| Tasks containing PII in description or comments | `confidential` |

---

## Sensitivity Change / Reclassification

When a source record's sensitivity tier changes (e.g., an internal document is reclassified as restricted), the ingestion pipeline must:

1. Detect the change via `document_content_hash` or source metadata comparison during the next sync cycle.
2. Re-ingest the affected chunks with the updated `sensitivity`, target collection, and any updated `data_scope_tags`.
3. Delete old chunks from the prior collection (e.g., `company_memory`).
4. Write new chunks to the correct collection (e.g., `restricted_memory`).
5. Log the reclassification event to the audit log: `old_sensitivity`, `new_sensitivity`, `doc_id`, timestamp.

Unlike access policy changes (which only require updating your orchestration layer config), sensitivity reclassification requires pipeline action because sensitivity determines **collection placement** — a hard architectural boundary in Qdrant.


path: guides/07-ingestion-pipeline.md
content: # Guide 07 — Ingestion Pipeline

## Purpose

This guide provides a complete, production-grade ingestion pipeline implementation — covering source capture, change detection, upsert patterns, deduplication, idempotency, and lifecycle management.

---

## Complete Pipeline Implementation

```python
import hashlib
import uuid
from datetime import datetime, timezone
from qdrant_client import QdrantClient, models
from typing import Optional

class IngestionPipeline:
    
    def __init__(self, qdrant_client: QdrantClient, embed_fn, config: dict):
        self.client = qdrant_client
        self.embed = embed_fn  # Function: list[str] → list[{dense, sparse}] or list[list[float]]
        self.config = config
        self.run_id = str(uuid.uuid4())
    
    def ingest_document(
        self,
        source_content: str,
        source_metadata: dict,
        collection_name: str,
        normalizer,
        chunker,
        tagger,
        classifier,
    ) -> dict:
        """
        Full pipeline for a single source document.
        Returns a summary of what was done.
        """
        
        # Step 1: Normalization
        normalized_text = normalizer.normalize(source_content, source_metadata)
        
        # Step 2: Document-level hash (change detection)
        doc_hash = hashlib.sha256(normalized_text.encode("utf-8")).hexdigest()
        
        # Step 3: Change detection — check if we've seen this document before
        existing_hash = self._get_existing_hash(
            source_metadata["source_system"],
            source_metadata["source_record_id"],
            collection_name,
        )
        
        if existing_hash == doc_hash:
            # Content unchanged — only refresh metadata timestamps
            self._refresh_metadata_only(source_metadata, collection_name)
            return {"status": "unchanged", "doc_hash": doc_hash}
        
        # Step 4: Classification
        sensitivity = classifier.classify(normalized_text, source_metadata)
        
        # Step 5: Scope tagging
        data_scope_tags = tagger.derive_tags(source_metadata)
        
        # Step 6: Chunking
        chunks = chunker.chunk(normalized_text, source_metadata)
        new_total = len(chunks)
        
        # Step 7: Build payloads, hash chunks, embed, upsert
        points = []
        for i, chunk_text in enumerate(chunks):
            chunk_hash = hashlib.sha256(chunk_text.encode("utf-8")).hexdigest()
            
            # Check for within-source duplicate
            if self._is_duplicate_chunk(chunk_hash, source_metadata["source_record_id"], collection_name):
                continue  # Update metadata in place rather than writing new point
            
            doc_id = f"{source_metadata['source_type']}_{source_metadata['source_record_id']}_{i}"
            
            payload = self._build_payload(
                source_metadata=source_metadata,
                doc_id=doc_id,
                chunk_text=chunk_text,
                chunk_index=i,
                chunk_total=new_total,
                chunk_hash=chunk_hash,
                doc_hash=doc_hash,
                sensitivity=sensitivity,
                data_scope_tags=data_scope_tags,
                chunk_strategy=chunker.strategy_name,
            )
            
            # Step 8: Embedding
            embedding = self.embed([chunk_text])[0]
            
            # Build Qdrant point
            if isinstance(embedding, dict):
                # Hybrid: {dense: [...], sparse: {indices: [...], values: [...]}}
                vector = {
                    "dense": embedding["dense"],
                    "sparse": models.SparseVector(
                        indices=embedding["sparse"]["indices"],
                        values=embedding["sparse"]["values"],
                    )
                }
            else:
                # Dense-only: [...]
                vector = embedding
            
            points.append(models.PointStruct(
                id=self._stable_uuid(doc_id),
                vector=vector,
                payload=payload,
            ))
        
        # Step 9: Upsert all points
        if points:
            self.client.upsert(
                collection_name=collection_name,
                points=points,
                wait=True,  # Wait for confirmation before returning
            )
        
        # Step 10: Delete stale chunks (if re-ingestion produced fewer chunks)
        old_total = self._get_old_chunk_total(
            source_metadata["source_record_id"], collection_name
        )
        if old_total and old_total > new_total:
            self._delete_stale_chunks(
                source_metadata, collection_name, new_total, old_total
            )
        
        return {
            "status": "ingested",
            "doc_id_base": f"{source_metadata['source_type']}_{source_metadata['source_record_id']}",
            "chunks_written": len(points),
            "chunks_total": new_total,
            "sensitivity": sensitivity,
            "run_id": self.run_id,
        }
    
    def _stable_uuid(self, doc_id: str) -> str:
        """Generate a stable UUID from a doc_id string (deterministic)."""
        return str(uuid.uuid5(uuid.NAMESPACE_DNS, doc_id))
    
    def _build_payload(self, **kwargs) -> dict:
        """Assemble the complete chunk payload."""
        now = datetime.now(timezone.utc).isoformat()
        source_metadata = kwargs["source_metadata"]
        
        return {
            "doc_id": kwargs["doc_id"],
            "source_type": source_metadata["source_type"],
            "source_system": source_metadata["source_system"],
            "source_record_id": source_metadata["source_record_id"],
            "integration_id": source_metadata.get("integration_id", "unknown"),
            "parent_doc_id": source_metadata.get("parent_doc_id", kwargs["doc_id"]),
            "document_content_hash": kwargs["doc_hash"],
            "chunk_index": kwargs["chunk_index"],
            "chunk_total": kwargs["chunk_total"],
            "content_hash": kwargs["chunk_hash"],
            "created_at": source_metadata.get("created_at", now),
            "ingested_at": now,
            "modified_at": source_metadata.get("modified_at", now),
            "source_last_modified": source_metadata.get("source_last_modified", now),
            "author_id": source_metadata.get("author_id", ""),
            "author_name": source_metadata.get("author_name", ""),
            "author_email": source_metadata.get("author_email"),
            "author_department": source_metadata.get("author_department"),
            "project_ids": source_metadata.get("project_ids", []),
            "team_ids": source_metadata.get("team_ids", []),
            "workspace_id": source_metadata["workspace_id"],
            "org_id": source_metadata["org_id"],
            "sensitivity": kwargs["sensitivity"],
            "allowed_groups": source_metadata.get("allowed_groups", []),
            "owner_user_ids": source_metadata.get("owner_user_ids", []),
            "is_pii": source_metadata.get("is_pii", False),
            "retention_days": source_metadata.get("retention_days", 1095),
            "data_scope_tags": kwargs["data_scope_tags"],
            "language": source_metadata.get("language", "en"),
            "content_type": source_metadata.get("content_type", "document"),
            "content_subtype": source_metadata.get("content_subtype"),
            "title": source_metadata.get("title"),
            "summary": None,  # Populate separately if using a summarization model
            "token_count": len(kwargs["chunk_text"]) // 4,
            "chunk_strategy": kwargs["chunk_strategy"],
            "embedding_model": self.config["embedding_model"],
            "sparse_model": self.config.get("sparse_model", "none"),
            "schema_version": "2.2",
            "ingestion_run_id": self.run_id,
            "connector_version": self.config["connector_version"],
            "normalizer_version": self.config["normalizer_version"],
            "chunker_version": self.config["chunker_version"],
            "source_metadata": source_metadata.get("source_specific", {}),
        }
    
    def _get_existing_hash(self, source_system: str, record_id: str, collection: str) -> Optional[str]:
        """Look up the stored document_content_hash for this record."""
        try:
            results = self.client.scroll(
                collection_name=collection,
                scroll_filter=models.Filter(
                    must=[
                        models.FieldCondition(key="source_system", match=models.MatchValue(value=source_system)),
                        models.FieldCondition(key="source_record_id", match=models.MatchValue(value=record_id)),
                        models.FieldCondition(key="chunk_index", match=models.MatchValue(value=0)),
                    ]
                ),
                with_payload=True,
                limit=1,
            )
            if results[0]:
                return results[0][0].payload.get("document_content_hash")
        except Exception:
            pass
        return None
    
    def _is_duplicate_chunk(self, content_hash: str, source_record_id: str, collection: str) -> bool:
        """Check if a chunk with this content_hash already exists for this source record."""
        results = self.client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="content_hash", match=models.MatchValue(value=content_hash)),
                    models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                ]
            ),
            limit=1,
        )
        return len(results[0]) > 0
    
    def _get_old_chunk_total(self, source_record_id: str, collection: str) -> Optional[int]:
        """Get the previously stored chunk_total for this record."""
        results = self.client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                    models.FieldCondition(key="chunk_index", match=models.MatchValue(value=0)),
                ]
            ),
            with_payload=True,
            limit=1,
        )
        if results[0]:
            return results[0][0].payload.get("chunk_total")
        return None
    
    def _delete_stale_chunks(
        self, source_metadata: dict, collection: str, new_total: int, old_total: int
    ):
        """Delete chunks whose chunk_index is now out of range after re-ingestion."""
        stale_ids = []
        for stale_index in range(new_total, old_total):
            doc_id = f"{source_metadata['source_type']}_{source_metadata['source_record_id']}_{stale_index}"
            stale_ids.append(self._stable_uuid(doc_id))
        
        if stale_ids:
            self.client.delete(
                collection_name=collection,
                points_selector=models.PointIdsList(points=stale_ids),
            )
    
    def _refresh_metadata_only(self, source_metadata: dict, collection: str):
        """For unchanged content, update only the ingested_at and source_last_modified timestamps."""
        now = datetime.now(timezone.utc).isoformat()
        self.client.set_payload(
            collection_name=collection,
            payload={
                "ingested_at": now,
                "source_last_modified": source_metadata.get("source_last_modified", now),
                "ingestion_run_id": self.run_id,
            },
            points=models.Filter(
                must=[
                    models.FieldCondition(
                        key="source_record_id",
                        match=models.MatchValue(value=source_metadata["source_record_id"])
                    )
                ]
            ),
        )
    
    def delete_document(self, source_type: str, source_record_id: str, collection: str):
        """Hard-delete all chunks for a source record (e.g., content deleted from source)."""
        self.client.delete(
            collection_name=collection,
            points_selector=models.FilterSelector(
                filter=models.Filter(
                    must=[
                        models.FieldCondition(key="source_type", match=models.MatchValue(value=source_type)),
                        models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                    ]
                )
            ),
        )
```

---

## Lifecycle Event Handling

| Event | Action |
|---|---|
| **Create** | Run full ingestion pipeline → upsert chunks |
| **Update** | Detect via `document_content_hash` → re-normalize, re-chunk, upsert, delete stale chunks |
| **Delete** | Hard-delete all chunks for the `source_record_id` |
| **Move / reclassify** | Metadata refresh + collection migration if sensitivity tier changes |
| **Access revoked** | Hard-delete if the content should no longer be retrievable |

### Collection Migration (Sensitivity Change)

```python
def migrate_chunks_to_collection(
    client: QdrantClient,
    source_record_id: str,
    old_collection: str,
    new_collection: str,
    new_sensitivity: str,
    new_data_scope_tags: list[str],
    audit_logger,
):
    """Move chunks from one collection to another when sensitivity changes."""
    
    # 1. Retrieve all chunks for this record from the old collection
    results, _ = client.scroll(
        collection_name=old_collection,
        scroll_filter=models.Filter(
            must=[models.FieldCondition(
                key="source_record_id",
                match=models.MatchValue(value=source_record_id)
            )]
        ),
        with_payload=True,
        with_vectors=True,
        limit=1000,
    )
    
    if not results:
        return
    
    old_sensitivity = results[0].payload.get("sensitivity")
    
    # 2. Update sensitivity and scope tags
    now = datetime.now(timezone.utc).isoformat()
    new_points = []
    for point in results:
        updated_payload = {
            **point.payload,
            "sensitivity": new_sensitivity,
            "data_scope_tags": new_data_scope_tags,
            "ingested_at": now,
        }
        new_points.append(models.PointStruct(
            id=point.id,
            vector=point.vector,
            payload=updated_payload,
        ))
    
    # 3. Write to new collection
    client.upsert(collection_name=new_collection, points=new_points, wait=True)
    
    # 4. Delete from old collection
    client.delete(
        collection_name=old_collection,
        points_selector=models.FilterSelector(
            filter=models.Filter(
                must=[models.FieldCondition(
                    key="source_record_id",
                    match=models.MatchValue(value=source_record_id)
                )]
            )
        ),
    )
    
    # 5. Audit log
    audit_logger.log_reclassification(
        source_record_id=source_record_id,
        old_sensitivity=old_sensitivity,
        new_sensitivity=new_sensitivity,
        timestamp=now,
        doc_ids=[p.payload["doc_id"] for p in results],
    )
```

---

## Deduplication

### Within-Source Deduplication
Before writing a chunk, check if a point with the same `content_hash` and `parent_doc_id` already exists in the target collection. On match, update metadata in place rather than writing a new point.

### Cross-Source Deduplication
Exact duplicates across different `source_type` values are retained as separate chunks because they represent distinct provenance. At retrieval time, near-duplicate results from different sources should be collapsed or down-ranked.

---

## Error Handling and Retry

```python
import time
from functools import wraps

def with_exponential_backoff(max_retries: int = 5, base_delay: float = 1.0):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return fn(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries - 1:
                        # Route to dead-letter queue after max retries
                        dead_letter_queue.push({"fn": fn.__name__, "args": args, "error": str(e)})
                        raise
                    delay = base_delay * (2 ** attempt)
                    time.sleep(delay)
        return wrapper
    return decorator
```

Failed lifecycle events (after max retries) must be routed to a dead-letter queue for manual review.

---

## Idempotency Checklist

Before deploying an ingestion pipeline to production, verify:

- [ ] `doc_id` format is deterministic — no timestamps, no random components
- [ ] All writes use `upsert`, never raw `insert`
- [ ] Same input run twice produces the same Qdrant state (no duplicate points)
- [ ] Stale chunk cleanup runs after every re-ingestion
- [ ] `document_content_hash` change detection is working (run a golden doc test)
- [ ] `ingestion_run_id` is a fresh UUID per run (not per chunk)


path: guides/09-access-control.md
content: # Guide 09 — Access Control Patterns

## Purpose

This guide defines the architectural principles and implementation patterns for access control in Qdrant-based RAG systems. It explains the separation of concerns between the data layer and the policy layer, and provides concrete patterns for enforcing permissions at retrieval time.

---

## The Fundamental Principle

> **Metadata describes what data is. It does not encode which agents or users may access it.**

Agent-to-data access mappings belong exclusively in your orchestration layer (middleware, n8n, LangGraph, custom API gateway). They must never be stored in chunk payloads.

---

## Why NOT to Store Permissions in Chunks

A common temptation is to add an `allowed_agents` field to each chunk. This is explicitly wrong for three reasons:

**1. Re-ingestion risk:** If an agent's access scope changes (e.g., a new AI agent is granted access to finance data), every affected chunk would need to be updated in the vector store. At scale this is operationally dangerous, slow, and error-prone.

**2. Privilege escalation surface:** If an LLM can influence the values of access control fields (even indirectly, through prompt injection), and those fields are authoritative for access decisions, you have a security vulnerability.

**3. Tight coupling:** Embedding access policy in the data layer creates a hard dependency between your AI agent configuration and your vector store schema, making both harder to evolve independently.

---

## The Correct Architecture: Two-Layer Separation

```
┌─────────────────────────────────────────────────────────────────┐
│  ORCHESTRATION LAYER (middleware / n8n / LangGraph)             │
│                                                                 │
│  Stores: agent-to-data access mappings                         │
│  - Which (user_id, agent_id) pairs can access which scopes     │
│  - Which collections each agent may query                      │
│  - Which sensitivity tiers are permitted per role              │
│                                                                 │
│  Responsibilities:                                              │
│  1. Validate agent identity                                     │
│  2. Look up user identity from session/token                   │
│  3. Resolve permission scope for (user, agent) pair            │
│  4. Build Qdrant query filters from resolved scope             │
│  5. Execute query — agent cannot modify filters                │
│  6. Return results — agent sees only permitted content         │
└────────────────────────────┬────────────────────────────────────┘
                             │ Builds filter from scope
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│  VECTOR STORE (Qdrant)                                          │
│                                                                 │
│  Stores: data characteristics only                             │
│  - sensitivity                                                  │
│  - team_ids                                                     │
│  - allowed_groups                                              │
│  - data_scope_tags                                             │
│  - org_id / workspace_id                                       │
│                                                                 │
│  Does NOT store:                                                │
│  - allowed_agents                                              │
│  - permitted_agents                                            │
│  - agent-level permission lists                                │
└─────────────────────────────────────────────────────────────────┘
```

---

## Permission Resolution Flow

```
1. User sends message → Orchestration layer receives it
2. Orchestration generates request_id, maps to user_id in session store
3. Request (with request_id, agent_id, query) sent to AI agent
4. AI agent calls retrieval tool → passes request_id, agent_id, query text only
5. Orchestration layer:
   a. Looks up user_id from session store using request_id
   b. Validates agent_id is permitted for this user
   c. Resolves permission scope for (user_id, agent_id):
      - permitted sensitivity tiers
      - permitted team_ids
      - permitted data_scope_tags
      - permitted collections
      - org_id / workspace_id
   d. Builds Qdrant filter from resolved scope
   e. Executes Qdrant query with filters as hard constraints
6. Results returned — AI agent never sees out-of-scope content
```

---

## Permission Scope Object

Your orchestration layer should resolve a permission scope that looks like this:

```python
# Example permission scope for a sales agent user
permission_scope = {
    "org_id": "org-456",
    "workspace_id": "ws-123",
    "permitted_collections": ["company_memory"],  # No access to restricted_memory
    "permitted_sensitivity": ["public", "internal"],
    "team_ids": ["team_sales_a", "team_sales_b"],  # Team-restricted
    "allowed_groups": ["g_all_staff", "g_sales"],
    "data_scope_tags": ["sales", "region:emea"],   # Only sales/EMEA content
}

# Example permission scope for an executive user
permission_scope_exec = {
    "org_id": "org-456",
    "workspace_id": "ws-123",
    "permitted_collections": ["company_memory", "restricted_memory"],
    "permitted_sensitivity": ["public", "internal", "restricted"],
    "team_ids": None,       # No team restriction — sees all team content
    "allowed_groups": None, # No group restriction
    "data_scope_tags": None, # No scope restriction — sees all domains
}
```

---

## Tiered Authorization Model

Implement access control as layers, where each layer narrows the scope:

| Layer | Enforced By | Mechanism |
|---|---|---|
| **Layer 1: User → Agent** | Orchestration layer | Agent Access Registry: is this agent_id permitted for this user_id? |
| **Layer 2: Agent → Collection** | Orchestration layer | Which collections may this (user, agent) pair query? |
| **Layer 3: Agent → Data** | Qdrant payload filters | sensitivity, team_ids, data_scope_tags, allowed_groups |
| **Layer 4: Composite** | Intersection of all above | Most restrictive rule wins |

---

## What Ingestion Pipelines Must NOT Store

The following fields are explicitly prohibited in chunk payloads:

```python
# PROHIBITED — never add these to a chunk payload
PROHIBITED_FIELDS = [
    "allowed_agents",
    "permitted_agents",
    "agent_access_list",
    "allowed_ai_agents",
    "agent_ids",
    # Any field that lists which AI agents may access this chunk
]
```

If you find these fields in an existing payload schema, they must be removed and replaced with the correct pattern (`data_scope_tags` + orchestration-layer mapping).

---

## Agent Access Changes Do Not Require Re-Ingestion

Because access mappings live in the orchestration layer (not in chunk metadata), access changes are instant and zero-cost to the vector store:

| Event | Required Action |
|---|---|
| New agent created and granted access to sales data | Update orchestration layer config: add `data_scope_tags: ["sales"]` to new agent's permitted scope. **No re-ingestion.** |
| Existing agent's access expanded to include finance | Update orchestration config. **No re-ingestion.** |
| Agent's access revoked | Update orchestration config. **No re-ingestion.** |
| Document reclassified to higher sensitivity tier | Re-ingest + migrate to correct collection. (See `03-data-classification.md`) |

The last row is the important exception: **sensitivity changes do require re-ingestion** because sensitivity determines collection placement, which is a hard boundary.

---

## Metadata Fields Used as Filter Primitives

These fields in the chunk payload are the filter inputs consumed by the orchestration layer. They must be accurately populated at ingest time and indexed in Qdrant:

| Field | Role | Ingest Requirement |
|---|---|---|
| `sensitivity` | Primary collection gate | Classified at ingest per Section 3 of `03-data-classification.md`. Never deferred. |
| `team_ids` | Team-scoped filter | Populated from source system wherever available. Never fabricated. |
| `allowed_groups` | Group-level filter | Populated from source system permissions at ingest. |
| `data_scope_tags` | Semantic scope filter | Populated using the controlled taxonomy in `02-metadata-schema.md`. |
| `org_id` / `workspace_id` | Tenant isolation | Always populated. Never omitted. |

---

## Audit Trail

Every retrieval request that passes through access control should produce an audit record. Your ingestion pipeline supports this by ensuring filter primitive fields are accurately populated — they are recorded in the audit log alongside each request.

Minimum audit record per retrieval request:

```python
audit_record = {
    "request_id": "...",          # Trace ID for this request
    "user_id": "...",             # Resolved from session store
    "agent_id": "...",            # Validated against Agent Access Registry
    "timestamp": "...",           # ISO 8601 UTC
    "permission_scope_applied": { # The resolved scope that was enforced
        "collections": ["company_memory"],
        "sensitivity": ["public", "internal"],
        "team_ids": ["team_sales_a"],
        "data_scope_tags": ["sales"],
    },
    "query_text": "...",          # Query submitted by the agent
    "doc_ids_returned": ["..."],  # Which chunks were returned
    "chunks_count": 7,
}
```

Given any `request_id`, operators must be able to reconstruct: who asked, which agent was acting, what filters were applied, and which chunks were returned.

---

## Security Anti-Patterns to Avoid

| Anti-Pattern | Why It's Wrong | Correct Approach |
|---|---|---|
| Storing `allowed_agents` in chunk payload | Re-ingestion required on every access change; privilege escalation risk | Store agent→scope mappings in orchestration layer config |
| Filtering access post-retrieval | Content briefly retrieved before being discarded; reduces result quality | Apply filters inside Qdrant query |
| Using `model_inferred` sensitivity as sole basis for access decisions | Variable confidence; untestable; may misclassify sensitive content | Require `source_asserted` or `system_derived` classification; log and review model-inferred |
| Skipping `data_scope_tags` when source context is known | Chunks become broadly accessible by default | Always populate scope tags when organizational context is determinable |
| Creating one collection per agent | Exhausts cluster resources; forces re-ingestion on access changes | Use scope tags + orchestration-layer filtering on a shared collection |


path: guides/05-chunking-standards.md
content: # Guide 05 — Chunking Standards

## Purpose

Chunking determines the retrieval unit — the piece of text returned to the LLM when a query matches. Chunks that are too large dilute relevance; chunks that are too small lack context. This guide defines chunk sizes, overlap, strategies, and determinism requirements by content type.

---

## The Golden Rule of Chunking

**The same normalized source input must always produce the same chunk set.** Given identical normalized text and the same `chunker_version`, the pipeline must produce:
- Identical chunk boundaries
- Identical chunk text
- Identical `content_hash` values
- Identical `doc_id` values

Non-deterministic chunking (e.g., using random seeds, model sampling) is prohibited in production pipelines.

---

## Chunking Parameters by Content Type

| Content Type | Token Range | Overlap | Strategy |
|---|---|---|---|
| Messages (Slack, short DMs) | 150–300 | 30 | Sentence window |
| Long email threads | 200–400 | 30 | Split at reply boundaries first, then sentence window within each reply |
| Meeting transcripts | 150–250 | 20 | Split at speaker turns first, then chunk within each speaker turn |
| Documents (Drive, PDFs, long-form) | 300–500 | 50 | Hierarchical: paragraph-level chunking with heading structure preserved |
| Tasks and tickets (ClickUp, Jira, etc.) | Up to 512 | N/A | One chunk per task; split only if token limit exceeded |
| Code files | 200–400 | 50 | Function/class boundary-aware; never split in the middle of a function |
| Structured data (tables, CSV rows) | N/A | N/A | One row or logical unit per chunk; include column headers in every chunk |

**Token count approximation:** Use the fixed heuristic of **4 characters per token** for fast, deterministic token estimation without requiring a tokenizer in the ingestion pipeline.

---

## Chunking Strategies Explained

### 1. Sentence Window (Messages, Short Content)

Chunk at sentence boundaries. Use sliding window with overlap to preserve context across chunk edges.

```python
import re

def chunk_sentence_window(
    text: str,
    max_tokens: int = 256,
    overlap_tokens: int = 30,
    chars_per_token: int = 4,
) -> list[str]:
    """Chunk text using a sliding sentence window."""
    
    max_chars = max_tokens * chars_per_token
    overlap_chars = overlap_tokens * chars_per_token
    
    # Split into sentences (simple heuristic; use a proper sentence splitter for production)
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    
    chunks = []
    current_chunk = []
    current_len = 0
    
    for sentence in sentences:
        sent_len = len(sentence)
        
        if current_len + sent_len > max_chars and current_chunk:
            chunks.append(" ".join(current_chunk))
            # Keep overlap: retain last N chars worth of sentences
            overlap_kept = []
            overlap_len = 0
            for s in reversed(current_chunk):
                if overlap_len + len(s) <= overlap_chars:
                    overlap_kept.insert(0, s)
                    overlap_len += len(s)
                else:
                    break
            current_chunk = overlap_kept
            current_len = overlap_len
        
        current_chunk.append(sentence)
        current_len += sent_len
    
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    
    return chunks
```

### 2. Reply-Boundary Chunking (Email Threads)

Split first at reply boundaries, then apply sentence window within each reply segment.

```python
REPLY_DELIMITERS = [
    r"^On .+wrote:$",
    r"^-----Original Message-----",
    r"^From:.+Sent:",
    r"^>{1,}\s",
]

def chunk_email_thread(text: str, max_tokens: int = 350) -> list[str]:
    """Split email thread at reply boundaries, then chunk each segment."""
    delimiter_pattern = "|".join(REPLY_DELIMITERS)
    
    # Split at reply delimiters
    segments = re.split(delimiter_pattern, text, flags=re.MULTILINE)
    segments = [s.strip() for s in segments if s.strip()]
    
    chunks = []
    for segment in segments:
        if len(segment) / 4 <= max_tokens:
            chunks.append(segment)
        else:
            chunks.extend(chunk_sentence_window(segment, max_tokens=max_tokens))
    
    return chunks
```

### 3. Speaker-Turn Chunking (Meeting Transcripts)

Split at speaker turn boundaries (identified by speaker attribution markers), then chunk within turns if they are long. Speaker attribution is **critical** — never split a speaker attribution from the text that follows it.

```python
def chunk_transcript(
    transcript_segments: list[dict],
    max_tokens: int = 200,
    overlap_tokens: int = 20,
    context_header: str = "",
) -> list[dict]:
    """
    Chunk a meeting transcript at speaker turns.
    Each segment is dict: {"speaker": "Jane", "text": "...", "start_ms": 0}
    """
    chunks = []
    
    for segment in transcript_segments:
        speaker = segment["speaker"]
        text = segment["text"]
        turn_text = f"[{speaker}]: {text}"
        
        if len(turn_text) / 4 <= max_tokens:
            # Fits in one chunk
            chunks.append({
                "text": turn_text,
                "context_header": context_header,
                "speaker": speaker,
                "start_ms": segment.get("start_ms", 0),
            })
        else:
            # Split long turn with overlap
            sub_chunks = chunk_sentence_window(text, max_tokens=max_tokens, overlap_tokens=overlap_tokens)
            for i, sub in enumerate(sub_chunks):
                chunks.append({
                    "text": f"[{speaker}]: {sub}",
                    "context_header": context_header,
                    "speaker": speaker,
                    "start_ms": segment.get("start_ms", 0),
                    "sub_chunk_index": i,
                })
    
    return chunks
```

### 4. Hierarchical Paragraph Chunking (Documents)

Preserve document structure. Chunk at paragraph boundaries. Include the nearest heading as a context header for each chunk, so chunks are self-contained without requiring the reader to know which section they came from.

```python
def chunk_document_hierarchical(
    markdown_text: str,
    max_tokens: int = 400,
    overlap_tokens: int = 50,
) -> list[dict]:
    """
    Chunk a Markdown document hierarchically.
    Splits at paragraph boundaries, tracks heading context.
    """
    lines = markdown_text.split("\n")
    chunks = []
    current_heading = ""
    current_paragraphs = []
    current_len = 0
    
    for line in lines:
        # Track heading context
        if line.startswith("#"):
            if current_paragraphs and current_len > 0:
                _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
                current_paragraphs = []
                current_len = 0
            current_heading = line.lstrip("#").strip()
            continue
        
        # Empty line = paragraph boundary
        if not line.strip():
            if current_len > 0 and current_len >= (max_tokens * 4 * 0.7):  # 70% full → flush
                _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
                current_paragraphs = []
                current_len = 0
            continue
        
        current_paragraphs.append(line)
        current_len += len(line)
    
    if current_paragraphs:
        _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
    
    return chunks

def _flush_doc_chunk(paragraphs, heading, max_tokens, overlap_tokens, chunks):
    text = " ".join(paragraphs)
    header = f"[Section: {heading}] " if heading else ""
    if len(text) / 4 <= max_tokens:
        chunks.append({"text": text, "section_heading": heading, "context_header": header})
    else:
        for sub in chunk_sentence_window(text, max_tokens, overlap_tokens):
            chunks.append({"text": sub, "section_heading": heading, "context_header": header})
```

---

## Context Headers

For threaded conversations, meeting transcripts, and email threads, prepend a **context header** to each chunk. The context header is separate from the chunk body's token count.

The context header ensures the chunk is self-contained — a retrieval result makes sense without the reader having to navigate back to the source.

**Format by type:**

| Source | Context Header Format |
|---|---|
| Slack thread | `[Channel: #sales-emea] [Thread from: Jane Smith, Apr 30 2024]` |
| Email thread | `[Subject: Q2 Budget Review] [From: [REDACTED_SECRET], Apr 30 2024]` |
| Meeting transcript | `[Meeting: Q3 Forecast Review \| Date: 2024-04-30 \| Attendees: Jane, Bob, Alice]` |
| Document section | `[Document: Product Roadmap Q4 \| Section: Engineering Priorities]` |

---

## Chunk Metadata Fields

Every chunk must carry these fields in its payload:

```python
def build_chunk_payload(
    parent_metadata: dict,
    chunk_text: str,
    chunk_index: int,
    chunk_total: int,
    chunk_strategy: str,
    context_header: str = "",
) -> dict:
    import hashlib
    
    content_hash = hashlib.sha256(chunk_text.encode("utf-8")).hexdigest()
    token_count = len(chunk_text) // 4  # 4 chars per token approximation
    
    payload = {
        **parent_metadata,  # Inherit all parent document metadata
        "doc_id": f"{parent_metadata['source_type']}_{parent_metadata['source_record_id']}_{chunk_index}",
        "chunk_index": chunk_index,
        "chunk_total": chunk_total,
        "content_hash": content_hash,
        "token_count": token_count,
        "chunk_strategy": chunk_strategy,
        "context_header": context_header,
    }
    return payload
```

---

## Chunking Rules (Summary)

1. Every chunk carries the full universal metadata payload (`chunk_index`, `chunk_total`, `parent_doc_id` — no exceptions).
2. Chunks must not split mid-sentence. Always break at sentence or structural boundaries.
3. Context headers are prepended but do NOT count toward the chunk's token limit.
4. Token counts are approximated at **4 characters per token** — do not call a tokenizer for this.
5. The `chunk_strategy` field must accurately reflect the strategy used.
6. Chunk size and overlap values must be validated with retrieval testing before production deployment.
7. When re-ingesting a document that now produces **fewer chunks** than before, delete the out-of-range chunks (those with `chunk_index` ≥ new total) from Qdrant.

---

## Common Mistakes

| Mistake | Impact | Fix |
|---|---|---|
| Splitting a speaker attribution from its text | Destroys attribution in retrieval | Always keep `[Speaker]:` + text together |
| Including quoted email content in chunks | Duplicate information in the index | Strip quoted content during normalization |
| Using model token counters in the chunking loop | Non-determinism if model version changes | Use 4 chars/token heuristic |
| Creating chunks of 50 tokens or less | Too small to carry useful context | Enforce minimum chunk size of 100 tokens |
| Ignoring section headings in documents | Chunks lack context, retrieval suffers | Always prepend heading as context header |
| Not deleting stale chunks after re-ingestion | Index accumulates orphan chunks | Always compare old vs. new `chunk_total` |


path: guides/10-operational-standards.md
content: # Guide 10 — Operational Standards

## Purpose

This guide covers the operational requirements for running a production Qdrant RAG system: schema versioning, data retention, monitoring, conformance testing, and observability.

---

## Schema Versioning and Migration

### Version Management

The `schema_version` field enables backward-compatible evolution. When the schema is updated:

1. Increment the version number in the schema definition
2. Update all ingestion pipelines to write the new version
3. Provide a migration script for existing chunks
4. Configure the retrieval layer to gracefully handle chunks written under prior schema versions during the migration window

### Migration Pattern

```python
def migrate_schema(
    client: QdrantClient,
    collection: str,
    from_version: str,
    to_version: str,
    migration_fn: callable,
    migration_run_id: str,
    batch_size: int = 100,
):
    """
    Migrate all chunks in a collection from one schema version to another.
    Safe to re-run (idempotent): skips chunks already on the target version.
    """
    offset = None
    migrated = 0
    skipped = 0
    
    while True:
        results, next_offset = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(
                        key="schema_version",
                        match=models.MatchValue(value=from_version)
                    )
                ]
            ),
            with_payload=True,
            limit=batch_size,
            offset=offset,
        )
        
        if not results:
            break
        
        for point in results:
            new_payload = migration_fn(point.payload)
            new_payload["schema_version"] = to_version
            new_payload["migration_run_id"] = migration_run_id
            
            client.set_payload(
                collection_name=collection,
                payload=new_payload,
                points=[point.id],
            )
            migrated += 1
        
        offset = next_offset
        if offset is None:
            break
    
    return {"migrated": migrated, "skipped": skipped}
```

### Migration Note: v2.1 → v2.2 (data_scope_tags)

During migration from v2.1 to v2.2, chunks without `data_scope_tags` will have no scope tags. Configure your orchestration layer to treat absent `data_scope_tags` as a pass-through (no tag filter applied) to avoid breaking retrieval for untagged content. Backfill high-value collections first. Record the `migration_run_id` on all backfilled chunks.

---

## Data Retention

```python
from datetime import datetime, timezone, timedelta

def run_retention_cleanup(client: QdrantClient, collection: str, batch_size: int = 500):
    """
    Delete chunks whose retention period has expired.
    retention_days counts from created_at.
    """
    now = datetime.now(timezone.utc)
    deleted_count = 0
    
    # Find expired chunks
    # Strategy: compute cutoff dates for common retention windows
    # (Alternative: use Qdrant datetime range filter if created_at is indexed as datetime)
    
    offset = None
    while True:
        results, next_offset = client.scroll(
            collection_name=collection,
            with_payload=True,
            limit=batch_size,
            offset=offset,
        )
        
        if not results:
            break
        
        expired_ids = []
        for point in results:
            payload = point.payload
            created_str = payload.get("created_at")
            retention_days = payload.get("retention_days", 1095)
            
            if not created_str:
                continue
            
            created_at = datetime.fromisoformat(created_str.replace("Z", "+00:00"))
            expires_at = created_at + timedelta(days=retention_days)
            
            if now > expires_at:
                expired_ids.append(point.id)
                # Log deletion event
                _log_deletion(
                    doc_id=payload.get("doc_id"),
                    parent_doc_id=payload.get("parent_doc_id"),
                    deletion_timestamp=now.isoformat(),
                    reason="retention_expired",
                )
        
        if expired_ids:
            client.delete(
                collection_name=collection,
                points_selector=models.PointIdsList(points=expired_ids),
            )
            deleted_count += len(expired_ids)
        
        offset = next_offset
        if offset is None:
            break
    
    return {"deleted": deleted_count}

def _log_deletion(doc_id, parent_doc_id, deletion_timestamp, reason):
    """Audit log for deletion events."""
    print(f"DELETED doc_id={doc_id} parent={parent_doc_id} at={deletion_timestamp} reason={reason}")
    # Write to your audit log store
```

**Retention defaults by content type:**

| Content Type | Default Retention |
|---|---|
| Operational / conversational (Slack, email, transcripts) | 3 years (1,095 days) |
| Project documentation | 5 years (1,825 days) |
| Financial / legal records | 7 years (2,555 days) — check regulatory requirements |
| PII content | Per data privacy regulations (often shorter) |

---

## Conformance Testing

Every ingestion pipeline must pass automated conformance checks before being promoted to production. These are mandatory gates in CI/CD.

### Mandatory Conformance Checks

```python
class PipelineConformanceTests:
    
    def test_metadata_completeness(self, client, collection, sample_size=1000):
        """All required fields must be non-null on every chunk."""
        REQUIRED_FIELDS = [
            "doc_id", "source_type", "source_system", "source_record_id",
            "document_content_hash", "chunk_index", "chunk_total", "content_hash",
            "created_at", "ingested_at", "modified_at", "author_id", "author_name",
            "workspace_id", "org_id", "sensitivity", "allowed_groups", "owner_user_ids",
            "is_pii", "retention_days", "data_scope_tags", "language", "content_type",
            "token_count", "chunk_strategy", "embedding_model", "schema_version",
            "ingestion_run_id", "connector_version", "normalizer_version", "chunker_version",
        ]
        
        results, _ = client.scroll(collection_name=collection, limit=sample_size, with_payload=True)
        
        failures = []
        for point in results:
            for field in REQUIRED_FIELDS:
                if point.payload.get(field) is None:
                    failures.append({"doc_id": point.payload.get("doc_id"), "missing_field": field})
        
        assert not failures, f"Metadata completeness failures: {failures}"
    
    def test_data_scope_tags_coverage(self, client, collection, source_type, threshold=0.80):
        """At least 80% of newly ingested chunks per source type must have ≥1 data_scope_tag."""
        from datetime import datetime, timedelta, timezone
        
        cutoff = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
        
        results, _ = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="source_type", match=models.MatchValue(value=source_type)),
                    models.FieldCondition(key="ingested_at", range=models.DatetimeRange(gte=cutoff)),
                ]
            ),
            with_payload=True,
            limit=5000,
        )
        
        if not results:
            return  # No recent ingestion to check
        
        tagged = sum(1 for p in results if p.payload.get("data_scope_tags"))
        coverage = tagged / len(results)
        
        assert coverage >= threshold, (
            f"data_scope_tags coverage for {source_type} is {coverage:.1%}, "
            f"below {threshold:.0%} threshold. Investigate tagging pipeline."
        )
    
    def test_deterministic_chunking(self, client, golden_docs: list, pipeline):
        """Re-ingest golden documents and verify chunks are byte-identical."""
        for golden_doc in golden_docs:
            previous_chunks = golden_doc["expected_chunks"]
            actual_chunks = pipeline.chunk(golden_doc["normalized_text"], golden_doc["metadata"])
            
            assert len(actual_chunks) == len(previous_chunks), (
                f"Chunk count mismatch for {golden_doc['id']}: "
                f"expected {len(previous_chunks)}, got {len(actual_chunks)}"
            )
            for i, (expected, actual) in enumerate(zip(previous_chunks, actual_chunks)):
                assert expected == actual, f"Chunk {i} mismatch for {golden_doc['id']}"
    
    def test_idempotent_rerun(self, client, collection, source_record_id, pipeline):
        """Re-run pipeline for a document and verify Qdrant point count doesn't increase."""
        before_count = self._count_chunks(client, collection, source_record_id)
        pipeline.ingest_document(...)  # Re-ingest
        after_count = self._count_chunks(client, collection, source_record_id)
        
        assert after_count == before_count, (
            f"Idempotency failure: chunk count went from {before_count} to {after_count}"
        )
    
    def test_duplicate_suppression(self, client, collection):
        """No duplicate content_hash values within the same parent_doc_id."""
        # Check for duplicate content_hashes within the same parent
        # Implementation: scroll and group by parent_doc_id, check content_hash uniqueness
        pass
    
    def test_delete_propagation(self, client, collection, source_record_id, pipeline):
        """Delete a source record and verify chunks are removed within propagation SLA."""
        pipeline.delete_document(source_type="slack", source_record_id=source_record_id, collection=collection)
        remaining = self._count_chunks(client, collection, source_record_id)
        assert remaining == 0, f"Delete propagation failed: {remaining} chunks remain"
    
    def test_lineage_fields_complete(self, client, collection, sample_size=1000):
        """ingestion_run_id, connector_version, normalizer_version, chunker_version must be set."""
        LINEAGE_FIELDS = ["ingestion_run_id", "connector_version", "normalizer_version", "chunker_version"]
        results, _ = client.scroll(collection_name=collection, limit=sample_size, with_payload=True)
        failures = [
            point.payload.get("doc_id")
            for point in results
            if any(not point.payload.get(f) for f in LINEAGE_FIELDS)
        ]
        assert not failures, f"Lineage field gaps in {len(failures)} chunks"
    
    def test_access_control_smoke(self, query_fn, test_cases: list):
        """
        For each agent configuration, verify that returned chunks satisfy expected filters.
        This validates the end-to-end filter chain, not just chunk payload.
        """
        for case in test_cases:
            results = query_fn(
                query=case["query"],
                permission_scope=case["permission_scope"],
            )
            for result in results:
                p = result.payload
                assert p["sensitivity"] in case["permitted_sensitivity"], (
                    f"Access control violation: chunk {p['doc_id']} has sensitivity "
                    f"{p['sensitivity']}, not in {case['permitted_sensitivity']}"
                )
    
    def _count_chunks(self, client, collection, source_record_id):
        results, _ = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[models.FieldCondition(
                    key="source_record_id",
                    match=models.MatchValue(value=source_record_id)
                )]
            ),
            limit=1000,
        )
        return len(results)
```

---

## Monitoring and Alerting

Instrument your pipeline for these signals:

| Signal | Alert Threshold | Action |
|---|---|---|
| Ingestion failure rate | > 1% of documents in a run | Investigate connector or pipeline error |
| Schema validation errors | Any | Block pipeline promotion; fix immediately |
| `data_scope_tags` coverage | < 80% of newly ingested chunks for any source type | Investigate tagging pipeline within one sprint |
| Retrieval p95 latency | Exceeds SLA (e.g., 500ms for hybrid) | Scale compute or tune HNSW parameters |
| Collection point count | Within 20% of collection capacity | Plan scaling |
| Stale data detection | Most recent `ingested_at` for active source > configurable threshold | Check connector health |
| Dead-letter queue depth | > configurable threshold | Review failed lifecycle events |
| Duplicate `content_hash` count | Any within same `parent_doc_id` | Deduplication regression |

---

## Golden Document Registry

Maintain a registry of golden documents for each content type. These are used for determinism testing and regression validation.

Each golden document must include:
- The raw source input
- The expected normalized output
- The expected chunk set (chunk text + metadata)
- The `chunker_version` and `normalizer_version` that produced this output

**Required coverage patterns (per source type):**
- Edited/deleted messages within threads
- Forwarded emails with nested quoted replies
- Documents with duplicate content or broken formatting
- Transcript segments with filler words and low-confidence speech
- Tasks with extensive comment threads
- Content that has been moved or reclassified
- Content where `data_scope_tags` derivation is ambiguous

Version the golden document registry alongside `chunker_version` and `normalizer_version`. When either bumps, update the expected outputs and document why the output changed.

---

## Replay Capability

The ingestion system must support replaying ingestion for:
- A specific document (by `source_record_id`)
- All documents from a specific source type
- All documents ingested within a date range
- All documents associated with a specific `ingestion_run_id`

Replay must be idempotent: replaying always produces the same final state in Qdrant as the original run (assuming the source content hasn't changed).

```python
def replay_ingestion(
    pipeline: IngestionPipeline,
    scope: dict,  # {"source_record_id": "..."} or {"source_type": "slack"} or {"run_id": "..."}
):
    """
    Replay ingestion for a specific scope.
    Fetches source records from the source system and re-ingests them.
    """
    source_records = fetch_source_records(scope)
    results = []
    for record in source_records:
        result = pipeline.ingest_document(record["content"], record["metadata"], ...)
        results.append(result)
    return results
```


path: guides/06-embedding-models.md
content: # Guide 06 — Embedding Models

## Purpose

This guide covers embedding model selection, configuration, and usage for both dense-only and hybrid (dense + sparse) RAG pipelines built on Qdrant.

---

## Decision: Dense-Only vs. Hybrid

Choose **hybrid** (dense + sparse) when your content contains:
- Proper nouns, product names, model numbers, or identifiers
- Technical jargon, acronyms, or domain-specific terminology
- Code identifiers, error codes, or numeric values
- Queries where exact keyword match matters alongside semantic similarity

Choose **dense-only** when:
- Content is purely natural language with no specialized terminology
- Latency is critical and you need the simplest possible pipeline
- Your embedding budget favors one API call over two model passes
- You're prototyping and want to validate the concept first

**Recommended default:** Hybrid with BGE-M3. The single-model-pass architecture (one call produces both dense and sparse vectors) makes it operationally simple.

---

## Recommended Models

### Hybrid Pipeline: BAAI/BGE-M3

**Use when:** You need both semantic and keyword search. This is the recommended default for enterprise RAG.

- Dense dimension: **1024**
- Sparse: SPLADE-style output (produced in the same model pass)
- Distance metric: **Cosine**
- Language support: Multilingual (100+ languages)
- Context window: 8,192 tokens
- One model pass produces both vector types

```python
from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

def embed_hybrid_bgem3(texts: list[str]) -> list[dict]:
    """
    Embed a list of texts using BGE-M3.
    Returns list of dicts with 'dense' and 'sparse' vectors.
    """
    output = model.encode(
        texts,
        batch_size=12,
        max_length=512,
        return_dense=True,
        return_sparse=True,
        return_colbert_vecs=False,  # Set True if using late interaction reranking
    )
    
    results = []
    for i in range(len(texts)):
        dense_vector = output["dense_vecs"][i].tolist()
        sparse_weights = output["lexical_weights"][i]  # dict: {token_id: weight}
        
        # Convert sparse weights to Qdrant SparseVector format
        indices = list(sparse_weights.keys())
        values = list(sparse_weights.values())
        
        results.append({
            "dense": dense_vector,
            "sparse": {"indices": indices, "values": values},
        })
    
    return results

# Alternatively, using Qdrant's built-in BM25 with IDF for sparse
# (simpler, no external model needed for sparse):
# Configure the collection with modifier=IDF and send raw tokens for sparse
```

**Collection setup for BGE-M3:**

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="my_collection",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,
            distance=models.Distance.COSINE,
        )
    },
    sparse_vectors_config={
        "sparse": models.SparseVectorParams(
            modifier=models.Modifier.IDF  # Optional: use IDF weighting for sparse
        )
    },
)
```

---

### Dense-Only Pipeline: OpenAI text-embedding-3-small

**Use when:** You want a simple, cost-efficient dense embedding with strong performance. No local model hosting required.

- Dense dimension: **1536**
- Distance metric: **Cosine**
- Language support: Strong multilingual support
- API-based (no local GPU required)
- Supports Matryoshka truncation (can use 512-dim for faster/cheaper retrieval)

```python
from openai import OpenAI

openai_client = OpenAI()

def embed_dense_openai_small(texts: list[str], dimensions: int = 1536) -> list[list[float]]:
    """
    Embed texts using text-embedding-3-small.
    Set dimensions=512 for faster retrieval at some quality cost.
    """
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
        dimensions=dimensions,
    )
    return [item.embedding for item in response.data]
```

**Collection setup for text-embedding-3-small:**

```python
client.create_collection(
    collection_name="my_collection",
    vectors_config=models.VectorParams(
        size=1536,  # or 512 if using Matryoshka truncation
        distance=models.Distance.COSINE,
        hnsw_config=models.HnswConfigDiff(m=16, ef_construct=100),
    ),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            quantile=0.99,
            always_ram=True,
        )
    ),
)
```

---

### Dense-Only (High Quality): OpenAI text-embedding-3-large

**Use when:** Maximum embedding quality is needed and cost is not the primary constraint.

- Dense dimension: **3072**
- Distance metric: **Cosine**
- Supports Matryoshka truncation down to 256 dimensions

```python
def embed_dense_openai_large(texts: list[str], dimensions: int = 3072) -> list[list[float]]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-large",
        input=texts,
        dimensions=dimensions,
    )
    return [item.embedding for item in response.data]
```

---

### Sparse-Only Options (for Hybrid without BGE-M3)

If you're using a different dense model but still want hybrid retrieval, you can use these for the sparse side:

| Model | Notes |
|---|---|
| Qdrant BM25 with `modifier=IDF` | Native Qdrant BM25, no external model needed. Good baseline. |
| `BAAI/bge-m3` SPLADE output | Best quality sparse vectors, but requires BGE-M3. |
| `prithivida/Splade_PP_en_v1` | SPLADE model via FastEmbed. Good quality for English. |
| [REDACTED_SECRET] | BM42 — attention-weighted sparse. Better than BM25 for chunked text. |

**BM42 setup (recommended sparse-only companion to any dense model):**

```python
from fastembed import SparseTextEmbedding

sparse_model = SparseTextEmbedding(model_name=[REDACTED_SECRET])

def embed_sparse_bm42(texts: list[str]) -> list[dict]:
    embeddings = list(sparse_model.embed(texts))
    return [
        {"indices": e.indices.tolist(), "values": e.values.tolist()}
        for e in embeddings
    ]
```

---

## Query vs. Document Embedding

For BGE-M3, use `encode_queries` for query embedding and `encode` for document embedding. This applies the E5-style instruction prefix that improves retrieval quality:

```python
# At ingestion time — embed documents
doc_embeddings = model.encode(documents, ...)

# At retrieval time — embed queries
query_embeddings = model.encode_queries(
    queries,
    return_dense=True,
    return_sparse=True,
)
```

For OpenAI models, the same model and API endpoint is used for both documents and queries.

---

## Batching and Performance

| Scenario | Recommendation |
|---|---|
| BGE-M3 local inference | Batch size 8–16 on GPU, 2–4 on CPU |
| OpenAI embeddings | Batch up to 2,048 texts per API call |
| Large ingestion runs | Process in batches; implement exponential backoff on rate limit errors |
| Real-time query embedding | Single text per call is fine; consider caching frequent queries |

```python
def embed_in_batches(texts: list[str], batch_size: int = 16, embed_fn) -> list:
    """Embed texts in batches with progress tracking."""
    all_results = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        results = embed_fn(batch)
        all_results.extend(results)
    return all_results
```

---

## Model Selection Summary

| Use Case | Dense Model | Sparse Model | Collection dims |
|---|---|---|---|
| Enterprise RAG (recommended) | BGE-M3 (1024-dim) | BGE-M3 SPLADE | 1024 dense + sparse |
| Simple/budget RAG | text-embedding-3-small | Qdrant BM25 IDF | 1536 dense + sparse |
| Highest quality dense-only | text-embedding-3-large | None | 3072 dense |
| Standard dense-only | text-embedding-3-small | None | 1536 dense |
| Multilingual enterprise | BGE-M3 | BGE-M3 SPLADE | 1024 dense + sparse |

---

## Important: Use the Same Model at Query Time

The embedding model used during retrieval (query time) **must match** the model used during ingestion. Mixing models produces meaningless similarity scores.

Record the model in every chunk's payload:
```json
{
  "embedding_model": "BAAI/bge-m3",
  "sparse_model": "BAAI/bge-m3-splade"
}
```

Store the model identifier in your pipeline configuration and validate it at startup.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal:  | SkillSpector: suspicious (MEDIUM)