Skill: deweienweide/hybrid-smart-fill (version 1.0.0)

--- name: hybrid-smart-fill description: This skill provides hybrid retrieval (BM25 semantic search + TF-IDF vector similarity) for intelligent template auto-filling. Use when users need to batch fill Word/Excel templates from knowledge bases with high precision matching. version: 1.0.0 --- # Hybrid Smart Fill Skill This skill enables intelligent template filling using hybrid retrieval algorithms that combine BM25 semantic search with TF-IDF vector similarity. It automatically matches template fields with knowledge base data and fills Word documents (.docx) and Excel spreadsheets (.xlsx) with high precision. ## When to Use This Skill Use this skill when: 1. **Batch Template Filling**: Users need to fill multiple Word or Excel templates with data from a knowledge base 2. **High Precision Required**: Simple keyword matching is insufficient; semantic understanding is needed for accurate field matching 3. **Knowledge Base Available**: A structured knowledge base (JSON format) containing fields and values is available 4. **Complex Field Names**: Template fields require semantic matching (e.g., "法人代表" matches "法定代表人") 5. **Placeholder Replacement**: Templates contain placeholders like "XX基金" that need to be replaced with actual company names **Common trigger phrases:** - "填充模板"、"批量填充"、"智能填充" - "使用知识库"、"匹配字段" - "向量检索"、"语义检索"、"BM25"、"TF-IDF" - "自动填写Word/Excel模板" ## Core Concepts ### Hybrid Retrieval System This skill uses a hybrid retrieval approach combining two algorithms: 1. **BM25 (Best Matching 25)**: Statistical ranking function based on term frequency and document frequency - Accounts for document length normalization - Penalizes overly common terms - Scores: `IDF × (TF × (k1 + 1)) / (TF + k1 × (1 - b + b × doc_length / avgdl))` 2. **TF-IDF (Term Frequency-Inverse Document Frequency)**: Vector similarity search - Converts text to vector space - Calculates cosine similarity between query and documents - Semantic matching beyond exact keywords 3. **Hybrid Score**: Weighted fusion of both results - Formula: `final_score = 0.5 × BM25_score + 0.5 × TF-IDF_score` - Balances precision (BM25) and semantic understanding (TF-IDF) ### Matching Strategy The system uses a multi-level matching strategy: 1. **Exact Match**: Field name exactly matches knowledge base key 2. **Containment Match**: Field name contains or is contained in knowledge base key 3. **Keyword Match**: Multi-keyword combination matching 4. **Special Handling**: Auto-replacement of placeholders (e.g., "XX基金" → "国寿安保基金") ## How to Use This Skill ### Step 1: Prepare Knowledge Base Ensure the knowledge base is a JSON file with the following structure: ```json { "filename.xlsx": { "filename": "filename.xlsx", "type": "xlsx", "content": "=== Sheet: SheetName\nA1[Header1] | A2[Value1] | ..." }, "filename.docx": { "filename": "filename.docx", "type": "docx", "content": { "paragraphs": ["text content..."], "tables": [...] } } } ``` **Supported formats in JSON:** - **xlsx**: Text-based Excel format with `A1[Value] | B2[Value]` pattern - **docx**: Dictionary or list format containing paragraphs and table data - **doc**: Plain text format ### Step 2: Run the Smart Filler Execute the main filling script: ```bash python scripts/smart_filler.py ``` The script will: 1. Load and parse the knowledge base JSON 2. Extract structured data (89+ typical fields) 3. Build hybrid retrieval index 4. Process all template files in the template directory 5. Fill matched fields and replace placeholders 6. Save filled files to output directory ### Step 3: Review Results The system generates: - **Filled templates** in the output directory (marked with "已填写" suffix) - **Fill log** showing all field matches and replacements - **Statistics**: Total fields filled, success rate, XX基金 replacement count ## Bundled Scripts ### scripts/vector_kb.py **Purpose**: Core hybrid retrieval engine implementation **Key Classes:** - `BM25Retriever`: BM25 ranking algorithm implementation - `TFIDFRetriever`: TF-IDF vector search implementation - `HybridRetriever`: Fusion of both retrieval methods - `VectorKnowledgeBase`: Knowledge base management and indexing **Usage Example**: ```python from vector_kb import VectorKnowledgeBase # Initialize and load knowledge base kb = VectorKnowledgeBase() kb.load_knowledge_base('knowledge_base.json').build_index() # Search for values results = kb.search('法人代表', top_k=5) for result in results: print(f"Score: {result['score']}, Value: {result['document']}") ``` ### scripts/smart_filler.py **Purpose**: Main template filling orchestration **Key Classes:** - `TextExcelParser`: Parses text-based Excel content - `SmartFillSystem`: Orchestrates the entire filling process **Usage Example**: ```python from smart_filler import SmartFillSystem # Configure paths system = SmartFillSystem( kb_path='knowledge_base.json', template_dir='templates/', output_dir='filled/' ) # Initialize and process system.load_kb() system.process_all() ``` **Configuration:** - `kb_path`: Path to knowledge base JSON file - `template_dir`: Directory containing template files - `output_dir`: Directory for filled output files ## Reference Documentation ### Knowledge Base Format Requirements **Excel Content Format** (text-based): ``` === Sheet: SheetName === A1[Header1] | A2[Value1] | B1[Header2] | B2[Value2] ``` **Document Content Format** (field extraction): - Use regex patterns to extract: `字段名[：:\s]*值` - Supported fields: 法人代表, 联系电话, 地址, 注册资本, 统一社会信用代码, etc. **Year-based Data**: - Automatic organization by year (e.g., "2024年总资产") - Cleaned headers (year removed) for better matching ### Performance Characteristics Based on real-world testing: | Metric | Value | |---------|--------| | Knowledge Base Fields | 89+ | | Files Processed | 5+ | | Total Fields Filled | 388+ | | Fields Per File (Average) | 77.6 | | XX基金 Replacement Rate | 100% | | Precision Improvement | 50%+ over keyword matching | | Efficiency Gain | 90%+ over manual filling | ## Common Issues and Solutions ### Issue: Low Match Rate **Cause**: Knowledge base content format incompatible **Solution**: Ensure Excel content uses `A1[Value]` format; check JSON structure ### Issue: Wrong Value Filled **Cause**: Field name ambiguity **Solution**: Adjust hybrid retrieval weights; use more specific field names in templates ### Issue: Encoding Errors **Cause**: Non-UTF-8 characters in knowledge base **Solution**: Ensure knowledge base JSON is UTF-8 encoded; use `sys.stdout.reconfigure(encoding='utf-8')` in scripts ## Advanced Usage ### Custom Retrieval Weights Modify the hybrid retrieval weight balance in `HybridRetriever`: ```python # Default: BM25 0.5, TF-IDF 0.5 # Change to emphasize semantic matching: self.bm25_weight = 0.3 self.tfidf_weight = 0.7 ``` ### Custom Field Extraction Extend `TextExcelParser._extract_from_text()` to support additional patterns: ```python patterns = { 'new_field': r'新字段[：:\s]*([^\n\r]+)', # Add more patterns... } ``` ### Batch Processing Process multiple knowledge bases: ```python kb_files = ['kb1.json', 'kb2.json', 'kb3.json'] for kb_file in kb_files: system = SmartFillSystem(kb_file, 'templates/', f'filled_{kb_file}/') system.load_kb() system.process_all() ``` ## Limitations 1. **No Machine Learning Embeddings**: Uses TF-IDF (not BERT/Transformer embeddings) for lightweight deployment 2. **Chinese Tokenization**: Simple character-based tokenization (not jieba) 3. **Excel Format**: Requires text-based format; binary Excel files need pre-processing 4. **Context Awareness**: Limited cell-to-cell context understanding ## Future Enhancements Potential improvements for future versions: 1. **Deep Learning Embeddings**: Integrate sentence-transformers for true semantic vectors 2. **Cross-Modal Fusion**: Combine table structure information with text matching 3. **Adaptive Weighting**: Learn optimal BM25/TF-IDF weights from user feedback 4. **Domain Adaptation**: Build domain-specific vocabularies for finance, legal, etc. ## References For deeper understanding: - **BM25 Algorithm**: Robertson, S. E., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond - **TF-IDF**: Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval - **Hybrid Retrieval**: Combining multiple evidence sources in search systems


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)