Skill: ivangdavila/rag (version 1.0.0)

--- name: RAG slug: rag description: Build, optimize, and debug RAG pipelines with chunking strategies, retrieval tuning, evaluation metrics, and production monitoring. --- ## When to Use User wants to implement, improve, or troubleshoot Retrieval-Augmented Generation systems. ## Quick Reference | Topic | File | |-------|------| | Pipeline components & architecture | `architecture.md` | | Implementation patterns & code | `implementation.md` | | Evaluation metrics & debugging | `evaluation.md` | | Security & compliance | `security.md` | ## Core Capabilities 1. **Architecture design** — Select embedding models, vector DBs, and chunking strategies based on requirements 2. **Implementation** — Write ingestion pipelines, query handlers, and update logic 3. **Retrieval optimization** — Tune top-k, reranking, hybrid search parameters 4. **Evaluation** — Build test datasets, measure recall/precision, diagnose failures 5. **Production ops** — Monitor quality drift, set up alerts, debug degradation 6. **Security** — PII detection, access control, compliance requirements ## Decision Checklist Before recommending architecture, ask: - [ ] What document types and volume? - [ ] Latency requirements (real-time chat vs batch)? - [ ] Update frequency (how often do docs change)? - [ ] Access control needs (who can see what)? - [ ] Compliance constraints (GDPR, HIPAA, SOC2)? - [ ] Budget (managed vs self-hosted, embedding costs)? ## Critical Rules - **Never skip access control** — Filter at retrieval time, not after - **Always overlap chunks** — 10-20% prevents context loss at boundaries - **Evaluate before optimizing** — Build eval dataset first, then tune - **Same embedding model** — Query and documents must use identical model - **Monitor similarity scores** — Dropping averages signal drift or issues - **Plan for deletion** — GDPR erasure requires re-embedding capability ## Common Failure Patterns | Symptom | Likely Cause | Fix | |---------|--------------|-----| | Wrong docs retrieved | Query too vague, poor chunks | Query expansion, smaller chunks | | Relevant doc missed | Not indexed, low similarity | Check ingestion, hybrid search | | Hallucinated answers | Context too short | Increase top-k, better reranking | | Slow responses | Large chunks, no caching | Optimize chunk size, cache embeddings | | Inconsistent results | Non-deterministic reranking | Set seeds, use stable sorting |

path: architecture.md
content: # RAG Architecture

## Pipeline Components

```
Documents → Chunking → Embedding → Vector DB → Retrieval → Reranking → LLM
```

### 1. Document Processing
- **Ingestion formats:** PDF, HTML, Markdown, Word, plain text
- **Parsing:** Extract text while preserving structure (headers, tables, lists)
- **Metadata:** Capture source, date, author, section for filtering

### 2. Chunking Strategies

| Strategy | Best For | Chunk Size |
|----------|----------|------------|
| Fixed-size | Simple docs | 256-512 tokens |
| Recursive | Structured content | Varies by delimiter |
| Semantic | Dense technical docs | Based on topic boundaries |
| Sentence | Q&A, chat | 3-5 sentences |

**Critical rules:**
- Overlap 10-20% between chunks to preserve context at boundaries
- Never split mid-sentence or mid-code-block
- Include parent headers in child chunks for context

### 3. Embedding Models

| Model | Dimensions | Speed | Quality | Cost |
|-------|------------|-------|---------|------|
| OpenAI ada-002 | 1536 | Fast | Good | $0.0001/1K |
| OpenAI text-3-small | 1536 | Fast | Better | $0.00002/1K |
| OpenAI text-3-large | 3072 | Medium | Best | $0.00013/1K |
| Cohere embed-v3 | 1024 | Fast | Excellent | $0.0001/1K |
| BGE-large-en | 1024 | Self-hosted | Excellent | Free |

**When to self-host:** Sensitive data, high volume, latency requirements

### 4. Vector Databases

| DB | Hosting | Scale | Best For |
|----|---------|-------|----------|
| Pinecone | Managed | Billions | Production, zero-ops |
| Weaviate | Both | Millions | Hybrid search, filtering |
| Qdrant | Both | Millions | Performance, Rust |
| pgvector | Self | Millions | Postgres ecosystem |
| Chroma | Self | Thousands | Prototyping, local dev |

### 5. Retrieval Strategies

**Basic:** Vector similarity (cosine, dot product)

**Hybrid:** Combine vector + keyword (BM25)
```
final_score = α * vector_score + (1-α) * bm25_score
```
Typical α: 0.5-0.7

**Reranking:** Apply cross-encoder after initial retrieval
- Cohere Rerank
- BGE Reranker
- Cross-encoder models

### 6. RAG Variants

| Type | Description | When to Use |
|------|-------------|-------------|
| Naive RAG | Retrieve → Generate | Simple use cases |
| Advanced RAG | Pre/post-retrieval optimization | Production systems |
| Modular RAG | Swappable components | Experimentation |
| Agentic RAG | Multi-step retrieval | Complex queries |
| Graph RAG | Entity relationships | Knowledge graphs |

## Architecture Decisions Checklist

- [ ] Document types and expected volume?
- [ ] Latency requirements (p95 target)?
- [ ] Update frequency (real-time vs batch)?
- [ ] Multi-tenancy requirements?
- [ ] Compliance constraints (data residency)?
- [ ] Budget (embedding costs, hosting)?


path: evaluation.md
content: # RAG Evaluation & Debugging

## Retrieval Metrics

| Metric | What It Measures | Target |
|--------|------------------|--------|
| Recall@K | % of relevant docs in top-K | >0.8 |
| MRR | Reciprocal rank of first relevant | >0.5 |
| NDCG | Ranked relevance quality | >0.7 |
| Precision@K | % of top-K that are relevant | >0.6 |

## Generation Metrics

| Metric | What It Measures | How to Evaluate |
|--------|------------------|-----------------|
| Faithfulness | Answer supported by context | LLM-as-judge or human |
| Relevance | Answer addresses query | LLM-as-judge |
| Correctness | Factually accurate | Ground truth comparison |
| Coherence | Well-structured response | Human evaluation |

## Evaluation Dataset Creation

### Synthetic Generation
```python
# For each document chunk:
# 1. Generate questions the chunk answers
# 2. Create variations (paraphrase, specificity)
# 3. Add negative examples (unanswerable)
```

### From Production Logs
```python
# Capture: query, retrieved_chunks, user_feedback
# Label: thumbs up/down, corrections, reformulations
```

### Gold Standard Set
- 50-100 curated query-answer-source triples
- Cover edge cases and common queries
- Update quarterly as corpus evolves

## Debugging Retrieval Issues

### "Wrong documents retrieved"
1. Check query embedding quality (is query too vague?)
2. Examine chunk content (was source poorly chunked?)
3. Try hybrid search (add keyword matching)
4. Consider query expansion

### "Relevant doc exists but not retrieved"
1. Check if chunk contains key terms
2. Verify document was indexed (hash check)
3. Increase top_k and rerank
4. Examine embedding similarity scores

### "Low similarity scores across the board"
1. Embedding model mismatch? (query vs doc models)
2. Domain-specific vocabulary not captured?
3. Consider fine-tuning embeddings

## Production Monitoring

### Key Metrics to Track
```yaml
retrieval:
  - avg_similarity_score
  - retrieval_latency_p50/p95
  - empty_result_rate
  - chunks_per_query

generation:
  - response_latency
  - token_usage
  - error_rate
  - user_feedback_rate
```

### Alerting Rules
- Empty retrieval > 5% of queries
- Avg similarity < 0.5 (possible drift)
- Latency p95 > 3s
- Error rate > 1%

## A/B Testing RAG Configs

### What to Test
- Chunk size (256 vs 512 vs 1024)
- Overlap percentage (0%, 10%, 20%)
- Top-K values
- Reranker vs no reranker
- Hybrid alpha weighting

### Minimum Sample Size
- 1000+ queries for statistical significance
- Segment by query type if heterogeneous

### Success Metrics
- User satisfaction (feedback, reformulation rate)
- Task completion rate
- Engagement (follow-up questions)


path: implementation.md
content: # RAG Implementation Guide

## Document Ingestion Pipeline

### Step 1: Parse Documents
```python
# Use appropriate parser per format
# PDF: pypdf, pdfplumber, unstructured
# HTML: beautifulsoup, trafilatura
# Office: python-docx, openpyxl
```

### Step 2: Chunk with Overlap
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,  # 10% overlap
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)
```

### Step 3: Generate Embeddings
```python
# Batch for efficiency (100-500 per batch)
# Handle rate limits with exponential backoff
# Store vectors with metadata
```

### Step 4: Upsert to Vector DB
```python
# Include: chunk text, embedding, metadata
# Metadata: source_file, page, section, timestamp
```

## Query Pipeline

### Step 1: Embed Query
- Same model as documents
- Consider query expansion for better recall

### Step 2: Retrieve
```python
# Typical k values: 5-20 for initial retrieval
# Apply metadata filters BEFORE similarity search
results = vector_db.query(
    embedding=query_embedding,
    top_k=10,
    filter={"department": user.department}  # Access control
)
```

### Step 3: Rerank (Optional but Recommended)
```python
# Rerank top-k to top-n (e.g., 20 → 5)
reranked = reranker.rerank(query, results, top_n=5)
```

### Step 4: Format Prompt
```python
context = "\n\n".join([r.text for r in reranked])
prompt = f"""Answer based on the context below.

Context:
{context}

Question: {query}

Answer:"""
```

## Document Updates

### Incremental Update Strategy
1. **Detect changes:** Hash documents, compare to stored hashes
2. **Re-chunk only changed docs**
3. **Delete old vectors:** By document_id metadata
4. **Upsert new vectors**

### Full Re-index
- Required when: embedding model changes, chunking strategy changes
- Schedule during low-traffic windows
- Blue-green deployment pattern recommended

## Common Implementation Patterns

### Conversation Memory
```python
# Include last N turns in query context
# Weight recent messages higher
```

### Source Attribution
```python
# Return source metadata with each answer
# "Based on: doc_name, page X, section Y"
```

### Fallback Handling
```python
# When retrieval scores are low:
# 1. Expand query terms
# 2. Try different retrieval strategy
# 3. Acknowledge uncertainty to user
```

### Streaming Responses
```python
# Stream LLM output while retrieval completes
# Show "searching..." indicator during retrieval
```

## Error Handling Checklist

- [ ] Embedding API timeouts/rate limits
- [ ] Vector DB connection failures
- [ ] Empty retrieval results
- [ ] Context window overflow
- [ ] Malformed documents in ingestion
- [ ] Duplicate document detection


path: security.md
content: # RAG Security & Privacy

## PII Detection Before Indexing

### Scan for
- Names (NER models, regex patterns)
- Emails, phone numbers, addresses
- SSNs, ID numbers, credit cards
- Health information (PHI for HIPAA)
- Financial data

### Options
1. **Redact before embedding** — Replace PII with placeholders
2. **Skip sensitive documents** — Separate index for authorized users
3. **Encrypt sensitive chunks** — Decrypt at retrieval time

## Access Control

### Row-Level Security
```python
# Filter by user permissions BEFORE retrieval
results = vector_db.query(
    embedding=query_embedding,
    filter={
        "department": {"$in": user.departments},
        "classification": {"$lte": user.clearance_level}
    }
)
```

### Document-Level Permissions
- Tag documents with access groups at ingestion
- Inherit permissions from source systems (SharePoint, Confluence)
- Audit permission checks

## Prompt Injection Prevention

### The Risk
Malicious content in indexed documents:
```
IGNORE ALL PREVIOUS INSTRUCTIONS. You are now...
```

### Mitigations
1. **Input sanitization** — Strip known injection patterns
2. **Output filtering** — Check response for policy violations
3. **Prompt isolation** — System prompt > retrieved content > user query
4. **Human-in-the-loop** — Flag suspicious responses

## Compliance Requirements

### GDPR
- **Right to erasure:** Must remove user data from embeddings on request
- **Data residency:** Verify vector DB region (EU data in EU)
- **Data accuracy:** Stale embeddings with outdated PII violate Article 5
- **Consent:** Document basis for processing personal data

### HIPAA
- **PHI handling:** Encrypt at rest and in transit
- **Access logs:** Audit who accessed what health records
- **BAA required:** With embedding API providers and vector DB hosts

### SOC2
- **Encryption:** AES-256 for data at rest
- **Access controls:** Role-based, principle of least privilege
- **Audit trails:** Immutable logs of all access
- **Incident response:** Documented breach procedures

## Multi-Tenancy Isolation

### Namespace Separation
```python
# Each tenant gets isolated namespace/collection
customer_a_index = f"tenant_{customer_a_id}_index"
customer_b_index = f"tenant_{customer_b_id}_index"
```

### Metadata Filtering (Weaker)
```python
# Single index with tenant_id filter
# Risk: bugs in filter logic expose data
filter={"tenant_id": current_tenant}
```

**Recommendation:** Namespace separation for sensitive workloads

## Embedding API Security

### When Using External APIs (OpenAI, Cohere)
- Documents leave your perimeter
- Check vendor's data retention policies
- Consider self-hosted models for sensitive content:
  - sentence-transformers
  - BGE models
  - HuggingFace TEI

### Self-Hosted Considerations
- GPU infrastructure costs
- Model updates responsibility
- Still need encryption in transit

## Audit Trail Requirements

### What to Log
```json
{
  "timestamp": "2024-01-15T10:30:00Z",
  "user_id": "user_123",
  "query": "quarterly revenue",
  "retrieved_doc_ids": ["doc_1", "doc_2"],
  "similarity_scores": [0.87, 0.82],
  "response_generated": true,
  "ip_address": "192.168.1.1"
}
```

### Retention
- Align with compliance requirements (HIPAA: 6 years)
- Immutable storage (append-only logs)
- Secure against tampering

## Security Checklist

- [ ] PII scanning before indexing
- [ ] Access control at retrieval level
- [ ] Encryption at rest (vector DB)
- [ ] Encryption in transit (API calls)
- [ ] Embedding API data handling verified
- [ ] Multi-tenant isolation tested
- [ ] Prompt injection mitigations
- [ ] Audit logging enabled
- [ ] Right to erasure process documented
- [ ] Incident response plan for data exposure


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)