Skill: abhinas90/rag-pipeline-starter (version 1.0.0)

# RAG Pipeline Starter Production-grade RAG pipeline setup with chunking strategies, embedding benchmarks, and retrieval tuning for 50K-500K row datasets. ## Overview This skill provides a complete toolkit for building and optimizing RAG (Retrieval-Augmented Generation) pipelines. It analyzes your data, recommends optimal chunking strategies, benchmarks embedding models, and helps tune retrieval parameters for maximum accuracy. ## When to Use - Building a new RAG system from scratch - Optimizing an existing RAG pipeline's retrieval quality - Choosing the right embedding model for your domain - Processing large document collections (50K-500K rows) - Need to balance speed vs. accuracy for your use case ## Scripts ### chunking_analyzer.py Analyzes documents and recommends optimal chunking strategies based on content structure. **Usage:** ```bash # Assess data and get strategy recommendation python chunking_analyzer.py --assess ./data # Apply chunking strategy to documents python chunking_analyzer.py --strategy recursive --input ./data/doc.txt --output ./chunks/ --chunk-size 500 --overlap 50 ``` **Options:** - `--assess <dir>` - Analyze documents and recommend strategy - `--strategy <name>` - Chunking strategy: fixed, semantic, recursive, hierarchical - `--input <path>` - Input file or directory - `--output <dir>` - Output directory for chunks - `--chunk-size <int>` - Chunk size (default: 500) - `--overlap <int>` - Overlap between chunks (default: 50) ### embedding_benchmark.py Tests multiple embedding models on your data to find the best fit for your domain. **Usage:** ```bash python embedding_benchmark.py --data ./chunks/ --domain finance --output results.json ``` **Options:** - `--embeddings <models>` - Embedding models to test (space-separated) - `--data <dir>` - Directory with chunked text files (required) - `--domain <name>` - Domain name for context-specific recommendations - `--output <file>` - Output file for results (JSON) **Supported Embeddings:** - sentence-transformers/all-MiniLM-L6-v2 (384 dims, fast, free) - sentence-transformers/all-mpnet-base-v2 (768 dims, medium, free) - openai/text-embedding-ada-002 (1536 dims, fast, paid) - cohere/embed-english-v3.0 (1024 dims, fast, paid) - bm25 (sparse, fast, free) ### retrieval_tuner.py Optimizes retrieval parameters (top-k, similarity threshold) for your specific use case. **Usage:** ```bash python retrieval_tuner.py --index ./vector_store/ --queries ./test_queries.json --output tuning_results.json ``` **Options:** - `--index <dir>` - Vector store index directory - `--queries <file>` - JSON file with test queries and expected results - `--output <file>` - Output file for tuning results - `--top-k-range <min> <max>` - Range of top-k values to test (default: 1 20) - `--threshold-range <min> <max> <step>` - Similarity threshold range ### vector_store_manager.py Manages vector store operations: create, update, search, and maintain indexes. **Usage:** ```bash # Create index from chunks python vector_store_manager.py --create --chunks ./chunks/ --index ./vector_store/ --embedding sentence-transformers/all-MiniLM-L6-v2 # Search index python vector_store_manager.py --search --index ./vector_store/ --query "your search query" --top-k 5 ``` **Options:** - `--create` - Create new index from chunks - `--chunks <dir>` - Directory with chunked text files - `--index <dir>` - Vector store directory - `--embedding <model>` - Embedding model to use - `--search` - Search existing index - `--query <text>` - Search query - `--top-k <int>` - Number of results to return (default: 5) - `--update` - Update index with new documents - `--stats` - Show index statistics ## Pricing Strategy **Free tier** (this skill): Core chunking + embedding benchmark tools **Paid guide** ($49): Complete production RAG setup with: - Multi-modal document processing - Hybrid search (dense + sparse) - Re-ranking pipeline - Evaluation framework - Deployment scripts ## Workflow 1. **Assess your data** ```bash python chunking_analyzer.py --assess ./your_data/ ``` 2. **Apply chunking strategy** ```bash python chunking_analyzer.py --strategy recursive --input ./data/ --output ./chunks/ ``` 3. **Benchmark embeddings** ```bash python embedding_benchmark.py --data ./chunks/ --domain your_domain ``` 4. **Create vector store** ```bash python vector_store_manager.py --create --chunks ./chunks/ --index ./vector_store/ --embedding <recommended_model> ``` 5. **Tune retrieval** (optional) ```bash python retrieval_tuner.py --index ./vector_store/ --queries ./test_queries.json ``` ## Requirements - Python 3.8+ - Dependencies: numpy, sentence-transformers (optional for real embeddings) ## Files - `chunking_analyzer.py` - Document analysis and chunking - `embedding_benchmark.py` - Embedding model benchmarking - `retrieval_tuner.py` - Retrieval parameter optimization - `vector_store_manager.py` - Vector store operations - `skill.json` - Skill metadata


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)