Skill: 2025biophilia-coder/paper-to-table (version 1.0.0)

--- name: paper-to-table description: Extract structured data from academic papers (PDF/DOCX/TXT) into literature review tables (XLSX/CSV). Use when user says "整理文献到表格", "extract papers to table", "literature review table", or provides papers with a table template. Supports batch folder processing, dynamic header mapping, duplicate detection, and format preservation. Domains: Psychology, Cognitive Neuroscience, Computer Science, Brain Science. Handles English and Chinese papers. --- # Paper To Table Extract structured information from academic papers and populate literature review tables. ## Quality Principles ### 1. Extraction Fidelity (质量约束) - **NEVER hallucinate** information not present in the paper - **NEVER infer** beyond what is explicitly stated - **Use "N/A"** only when a field truly cannot be found after thorough search - **Distinguish** between: - Explicitly stated facts → Extract directly - Implied but not stated → Mark as "N/A" or note as implied - Absent information → "N/A" - **Confidence scoring**: Rate each extraction as High/Medium/Low confidence ### 2. Field Understanding (字段理解) - Understand the semantic meaning of each table header - Map paper content to headers based on meaning, not just keyword matching - Handle synonyms and domain-specific terminology - Recognize implicit information in context ### 3. Structured Depth (结构化深度) - Extract at appropriate granularity - Distinguish between study-level, experiment-level, and result-level information - Preserve relationships between related fields - Handle multi-experiment papers correctly ### 4. Batch Stability (批处理稳定性) - Process papers independently (failure of one doesn't affect others) - Log all operations for audit trail - Support resume from interruption - Validate outputs before writing to table ## Workflow ### Step 1: Identify Inputs - **Papers**: Single file, multiple files, or folder path → 支持 PDF/DOCX/TXT - **Table template**: XLSX 或 CSV，含表头定义结构 - **Language**: 自动检测或用户指定 - **Domain**: Psychology / Cognitive Neuroscience / Computer Science / Brain Science / General ### Step 2: Read Table Headers ```bash python scripts/read_table.py <table_path> ``` 输出：列名、数据类型约束、领域推断。 ### Step 3: Extract Paper Content ```bash python scripts/extract_paper.py <paper_path> --structured ``` **自动处理格式**：PDF→pdfplumber/PyMuPDF/OCR fallback；DOCX→python-docx；TXT→直接读取。 **输出结构化 JSON**：包含 `full_text` 和 `sections`（abstract/introduction/methods/results/discussion/conclusion）。 ### Step 4: LLM Extraction (Critical) **原则**：只提取论文明确陈述的信息，绝不臆造。 **输入**：表头 + 论文全文/章节 **输出格式**（每个字段）： ```json { "FieldName": { "value": "extracted value 或 N/A", "confidence": "HIGH/MEDIUM/LOW", "source": "paper location" } } ``` **CRITICAL RULES：** 1. JSON keys 必须与表头完全匹配（大小写敏感） 2. 缺失信息→`"N/A"`，不做推断 3. 多值用分号分隔 4. 保留原文语言 5. LOW confidence 字段需说明原因 **提取优先级**：Abstract→Methods→Results→Discussion→补充材料 ### Step 5: Validate & Write ```bash python scripts/write_table.py <table_path> '<json_data>' --validate ``` **验证内容**：JSON格式、键名匹配、无重复条目、数据类型合理。 **重复检测**：标题相似度>85%视为重复，跳过并警告。 ### Step 6: Report 报告：处理论文数、新增行数、跳过数（重复/错误）、LOW confidence 字段、输出路径。 ## Batch Processing ```bash python scripts/batch_process.py <papers_folder> <table_path> [output_folder] ``` - 独立处理每篇论文（单篇失败不影响其他） - 自动生成日志 `batch_log_YYYYMMDD_HHMMSS.json` - 支持断点续传（从日志恢复进度） 详细字段定义、提取策略 → `references/extraction-patterns.md` 质量检查清单 → `references/quality-checklist.md` ## Domain Specializations ### Psychology / Cognitive Neuroscience / Computer Science / Brain Science

path: references/extraction-patterns.md
content: # Extraction Patterns for Paper-to-Table

## Core Philosophy

**Extract what is there. Do not invent what is not.**

Every extraction must be grounded in the actual paper text. When in doubt, mark as "N/A" with a note about what was ambiguous.

---

## LLM Prompt Template

### Primary Extraction Prompt

```
You are extracting information from an academic paper to populate a literature review table.

## QUALITY RULES (STRICT)
1. EXTRACT ONLY what is explicitly stated in the paper
2. NEVER hallucinate or infer information not present
3. NEVER combine information from multiple sources into one field unless explicitly stated
4. Use "N/A" for information that cannot be found after thorough search
5. Rate confidence for each field: HIGH (explicitly stated), MEDIUM (implied but clear), LOW (inferred or ambiguous)
6. If a field contains multiple items, separate with semicolons
7. Preserve the original language of the paper

## TABLE HEADERS (Columns to fill)
{headers}

## HEADER DEFINITIONS
{header_definitions}

## PAPER CONTENT
{paper_text}

## EXTRACTION INSTRUCTIONS
For each header:
1. Search the paper content thoroughly for explicit mentions
2. Extract the exact or closely paraphrased information
3. Assign confidence score
4. If not found, use "N/A"

## OUTPUT FORMAT
Return ONLY a JSON object with this structure:
{{
  "Header1": {{
    "value": "extracted value or N/A",
    "confidence": "HIGH/MEDIUM/LOW",
    "source": "brief note on where in paper this was found"
  }},
  "Header2": {{ ... }},
  ...
}}
```

---

## Header Definitions by Domain

### Universal Fields (All Domains)

| Header | Definition | Extraction Rules | Examples |
|--------|-----------|------------------|----------|
| **Title/标题** | Full paper title | Extract exactly as published; include subtitle if present | "The Role of Dopamine in Reward Prediction" |
| **Authors/作者** | All authors in order | List all; use "et al." only if more than 10; preserve order | "Smith, J., Jones, M., et al." |
| **Year/年份** | Publication year | 4-digit year only; check header/footer/citation info | 2024 |
| **Journal/期刊** | Publication venue | Full journal name; include volume/issue if in header | "Nature Neuroscience, 45(3)" |
| **DOI** | Digital object identifier | Extract complete DOI string | "10.1038/s41593-024-01623-4" |
| **Objective/研究目标** | Main research question or hypothesis | State in authors' own words or close paraphrase; distinguish primary from secondary objectives | "To investigate whether dopamine signals encode reward prediction errors" |
| **Method/研究方法** | Overall study design | Summarize in 1-2 sentences; specify type (experimental, observational, computational, etc.) | "Randomized controlled trial with double-blind design" |
| **Sample Size/样本量** | Number of participants/samples | Extract exact numbers; note attrition if reported | "N = 120 (final N = 108 after attrition)" |
| **Key Findings/主要发现** | Main results | 2-3 bullet points or sentences; include effect sizes if reported | "1. Dopamine neurons showed phasic responses to reward cues (Cohen's d = 0.82)" |
| **Conclusion/结论** | Authors' main conclusion | Direct quote or close paraphrase; distinguish from speculation | "These results support the hypothesis that dopamine encodes reward prediction errors" |
| **Limitations/局限性** | Study limitations mentioned | List explicitly mentioned limitations; do not add your own | "Small sample size; cross-sectional design limits causal inference" |

---

### Psychology-Specific Fields

| Header | Definition | Extraction Rules | Examples |
|--------|-----------|------------------|----------|
| **Participants/被试** | Demographic and sampling details | Age range, gender distribution, recruitment method, inclusion/exclusion criteria | "University students (N=85), age 18-25, 62% female" |
| **Experimental Design/实验设计** | Design type and structure | Between-subjects, within-subjects, mixed; number of conditions | "2×3 mixed design: Group (control/experimental) × Time (pre/post/follow-up)" |
| **Measures/测量工具** | Psychometric instruments or behavioral measures | Name of scale, number of items, reliability (Cronbach's α) if reported | "Beck Depression Inventory-II (BDI-II; α = .92)" |
| **Manipulation/实验操纵** | Independent variable manipulation | How IV was manipulated; check manipulation validation | "Participants wrote about a negative experience for 15 minutes" |
| **Effect Size/效应量** | Reported effect sizes | Extract exact values with confidence intervals if available | "Cohen's d = 0.65, 95% CI [0.32, 0.98]" |
| **Preregistration/预注册** | Whether study was preregistered | Extract preregistration platform and ID if available | "OSF: https://osf.io/abc123" |

---

### Cognitive Neuroscience-Specific Fields

| Header | Definition | Extraction Rules | Examples |
|--------|-----------|------------------|----------|
| **Imaging Modality/成像模态** | Neuroimaging method | fMRI, EEG, MEG, PET, etc.; include scanner specifications | "3T Siemens Prisma MRI" |
| **Preprocessing/预处理** | Data preprocessing pipeline | Software used, main preprocessing steps | "SPM12: realignment, coregistration, normalization (MNI), smoothing (8mm FWHM)" |
| **Task Paradigm/任务范式** | Experimental task details | Task name, trial structure, timing parameters | "Delayed match-to-sample task; 2s stimulus, 3s delay, 2s probe" |
| **Brain Regions/脑区** | Regions of interest or significant clusters | Use standard anatomical names; include MNI coordinates for peak voxels | "Left dorsolateral prefrontal cortex (MNI: -42, 18, 34)" |
| **Analysis Type/分析类型** | Statistical analysis approach | GLM, MVPA, connectivity, etc.; software package | "GLM with FSL FEAT; group-level mixed-effects analysis" |
| **Behavioral-Neural Correlation/行为-神经关联** | Relationship between behavior and neural measures | Report correlation coefficients and significance | "r(28) = .54, p < .001 between accuracy and hippocampal activation" |

---

### Computer Science-Specific Fields

| Header | Definition | Extraction Rules | Examples |
|--------|-----------|------------------|----------|
| **Algorithm/算法** | Proposed or evaluated algorithm | Name and brief description; distinguish from baseline methods | "Transformer-XL with recurrence mechanism" |
| **Model Architecture/模型架构** | Neural network or model structure | Layer types, dimensions, key components | "6-layer encoder-decoder, 512-dim embeddings, 8 attention heads" |
| **Dataset/数据集** | Training/evaluation datasets | Dataset name, size, source; distinguish train/validation/test | "ImageNet-1K (1.28M train, 50K validation); COCO 2017" |
| **Training Details/训练细节** | Training procedure | Optimizer, learning rate, batch size, epochs, hardware | "AdamW, lr=1e-4, batch=256, 100 epochs, 8×V100" |
| **Metrics/评估指标** | Evaluation metrics | Primary and secondary metrics with values | "Accuracy: 94.2%; F1: 0.891; Inference: 12ms/sample" |
| **Code Availability/代码可用性** | Code or model availability | GitHub link, license, reproducibility statement | "https://github.com/user/repo; MIT License" |
| **Ablation Results/消融实验** | Ablation study findings | Key ablation comparisons and their impact | "Removing attention mechanism: -3.2% accuracy" |

---

### Brain Science-Specific Fields

| Header | Definition | Extraction Rules | Examples |
|--------|-----------|------------------|----------|
| **Species/物种** | Animal species or model organism | Full species name; strain if relevant | "Mus musculus (C57BL/6J)" |
| **Brain Region/脑区** | Target brain region | Standard anatomical nomenclature; laterality | "Hippocampal CA1 region, bilateral" |
| **Recording Method/记录方法** | Neural recording technique | Electrophysiology, calcium imaging, etc.; probe details | "Silicon probe recording (64-channel, 25μm spacing)" |
| **Stimulation/刺激** | Stimulation parameters if applicable | Type, frequency, intensity, duration | "Optogenetic stimulation: 473nm, 20Hz, 5ms pulses, 10s duration" |
| **Cell Type/细胞类型** | Targeted cell population | Genetic markers, morphological classification | "Parvalbumin-positive interneurons (PV-Cre mice)" |
| **Behavioral Task/行为任务** | Associated behavioral paradigm | Task name and key parameters | "Morris water maze: 90s trials, 4 trials/day, 5 days" |

---

## Extraction Strategies

### Strategy 1: Structured Sections (Preferred)

When the paper has clear sections, extract systematically:

1. **Abstract** → Title, Authors, Objective, Key Findings (brief)
2. **Introduction** → Objective, Hypothesis, Theoretical Background
3. **Methods** → Method, Sample Size, Participants, Measures, Task Paradigm, Imaging Modality
4. **Results** → Key Findings, Effect Sizes, Metrics, Brain Regions
5. **Discussion/Conclusion** → Conclusion, Limitations, Implications

### Strategy 2: Unstructured Content

When sections are unclear:
1. First 500 words → Title, Authors, Objective, Hypothesis
2. Middle sections → Look for methodology keywords ("participants", "stimuli", "procedure", "analysis")
3. Final 500 words → Conclusions, limitations, future directions
4. Tables/Figures captions → Sample sizes, key results, statistical values

### Strategy 3: Citation and Metadata

Always check:
- **Header/footer** → Journal, year, volume, DOI
- **First page** → Complete author list, affiliations
- **Acknowledgments/Funding** → Sometimes contains preregistration info
- **Supplementary materials reference** → Additional datasets, code

---

## Multi-Language Handling

### English Papers
- Extract as-is
- Maintain English for all fields unless user requests translation
- Keep technical terms in English even if translating

### Chinese Papers
- Extract as-is
- Maintain Chinese for all fields unless user requests translation
- Keep proper nouns (names, places) in original form

### Mixed Content
- Preserve the language of the original text for each field
- If a field contains both languages, include both with clear separation

---

## Confidence Scoring Guide

### HIGH Confidence
- Information is explicitly stated in the paper
- Direct quotes or clear paraphrases available
- Located in standard sections (Abstract, Methods, Results)

### MEDIUM Confidence
- Information is implied but clearly inferable
- Located in Discussion or supplementary materials
- Requires minor synthesis of multiple statements

### LOW Confidence
- Information is ambiguous or contradictory
- Requires significant inference
- Located in non-standard sections or footnotes
- Mark as LOW and provide reasoning

---

## Common Pitfalls to Avoid

1. **Don't confuse correlation with causation** → If the paper says "associated with", don't extract "causes"
2. **Don't report planned analysis as completed** → Distinguish preregistered plans from actual results
3. **Don't mix up sample sizes** → Distinguish between recruited, final, and per-analysis N
4. **Don't report predictions as findings** → Hypotheses go in Objective, results go in Key Findings
5. **Don't extract from abstracts alone for complex fields** → Always check Methods for details


path: references/quality-checklist.md
content: # Quality Checklist for Paper-to-Table

## Pre-Extraction Checks

- [ ] Paper text was successfully extracted (not empty or error message)
- [ ] Paper contains identifiable sections or coherent content
- [ ] Language is correctly identified
- [ ] Table headers are clearly defined

## Extraction Quality Checks

### Fidelity Verification
- [ ] Each extracted value can be traced to a specific location in the paper
- [ ] No information was invented or hallucinated
- [ ] Numerical values match exactly (no rounding unless specified)
- [ ] Author names are complete and correctly ordered
- [ ] Technical terms are preserved accurately

### Completeness Check
- [ ] All table headers have corresponding extracted values
- [ ] "N/A" is used appropriately (not as a shortcut for "didn't check")
- [ ] Multi-value fields use proper separator (semicolon)
- [ ] Subfields are not conflated (e.g., sample size vs. effect size)

### Confidence Assessment
- [ ] Each field has a confidence rating
- [ ] LOW confidence fields are flagged for human review
- [ ] No field is marked HIGH confidence without clear evidence
- [ ] MEDIUM confidence fields include reasoning

## Domain-Specific Validation

### Psychology
- [ ] Sample demographics match reported values
- [ ] Experimental design is correctly classified
- [ ] Psychometric measures include reliability if reported
- [ ] Effect sizes include confidence intervals if available

### Cognitive Neuroscience
- [ ] Brain regions use standard anatomical nomenclature
- [ ] MNI coordinates are correctly transcribed
- [ ] Imaging parameters match reported values
- [ ] Preprocessing steps are accurately described

### Computer Science
- [ ] Algorithm names are correctly identified
- [ ] Dataset sizes match reported values
- [ ] Metrics are clearly identified (primary vs. secondary)
- [ ] Code availability links are valid URLs

### Brain Science
- [ ] Species and strain are correctly identified
- [ ] Recording/stimulation parameters are accurately extracted
- [ ] Cell type markers are correctly named
- [ ] Behavioral task parameters match reported values

## Post-Extraction Checks

- [ ] JSON is well-formed and valid
- [ ] All keys match table headers exactly (case-sensitive)
- [ ] No duplicate entries (check against existing table)
- [ ] Data types are appropriate for each column
- [ ] Special characters are properly escaped

## Error Handling

### Paper Extraction Failures
- Log error type and paper identifier
- Attempt fallback extraction methods
- Flag for manual review if all methods fail

### Missing Information
- Distinguish between "not in paper" and "extraction failed"
- Mark clearly with "N/A" and reason
- Do not guess or infer

### Ambiguous Information
- Mark confidence as LOW
- Include multiple possible interpretations if relevant
- Flag for human review

## Batch Processing Quality

- [ ] Each paper processed independently
- [ ] Progress logged for each paper
- [ ] Failed papers don't block successful ones
- [ ] Summary report includes success/failure counts
- [ ] Output file is valid and readable


path: scripts/batch_process.py
content: #!/usr/bin/env python3
"""Batch process multiple papers with error handling and logging."""

import sys
import os
import json
import time
from pathlib import Path
from datetime import datetime

# Import local modules
sys.path.insert(0, str(Path(__file__).parent))
from extract_paper import extract
from read_table import read_headers
from write_table import write_row


def setup_logging(output_dir):
    """Setup logging for batch processing."""
    log_file = Path(output_dir) / f"batch_log_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
    return log_file


def process_single_paper(paper_path, table_path, headers, domain, log_file):
    """Process a single paper with full error handling."""
    result = {
        'paper': str(paper_path),
        'status': 'pending',
        'timestamp': datetime.now().isoformat(),
        'extraction': {},
        'errors': [],
        'warnings': []
    }
    
    try:
        # Step 1: Extract text
        print(f"Processing: {paper_path.name}")
        extraction = extract(str(paper_path), structured=True)
        
        if extraction.startswith("[ERROR:"):
            result['status'] = 'failed'
            result['errors'].append(f"Extraction failed: {extraction}")
            return result
        
        try:
            extraction_data = json.loads(extraction)
        except json.JSONDecodeError:
            extraction_data = {'full_text': extraction, 'sections': {}}
        
        result['extraction'] = {
            'text_length': len(extraction_data.get('full_text', '')),
            'sections_found': list(extraction_data.get('sections', {}).keys()),
            'metadata': extraction_data.get('metadata', {})
        }
        
        # Step 2: LLM extraction (this would be called by the main agent)
        # For now, we just prepare the data
        result['status'] = 'extracted'
        result['llm_input'] = {
            'headers': headers,
            'domain': domain,
            'paper_text': extraction_data.get('full_text', '')[:5000]  # First 5000 chars for preview
        }
        
        return result
        
    except Exception as e:
        result['status'] = 'failed'
        result['errors'].append(f"Unexpected error: {str(e)}")
        return result


def batch_process(papers_dir, table_path, output_dir=None):
    """Process all papers in a directory."""
    papers_path = Path(papers_dir)
    table_path = Path(table_path)
    
    # Setup output directory
    if output_dir is None:
        output_dir = papers_path / "output"
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)
    
    # Setup logging
    log_file = setup_logging(output_dir)
    
    # Read table headers
    table_info = read_headers(str(table_path))
    if isinstance(table_info, list) and table_info[0].startswith("[ERROR:"):
        print(f"Error reading table: {table_info[0]}")
        return
    
    headers = table_info['headers']
    domain = table_info['domain']
    
    print(f"Table domain: {domain}")
    print(f"Headers: {', '.join(headers)}")
    print(f"Log file: {log_file}")
    print("-" * 60)
    
    # Find all papers
    supported_extensions = {'.pdf', '.docx', '.txt'}
    papers = [f for f in papers_path.iterdir() if f.suffix.lower() in supported_extensions]
    
    if not papers:
        print(f"No papers found in {papers_dir}")
        return
    
    print(f"Found {len(papers)} papers to process")
    print("-" * 60)
    
    # Process each paper
    results = []
    success_count = 0
    failed_count = 0
    
    for i, paper in enumerate(papers, 1):
        print(f"\n[{i}/{len(papers)}] ", end="")
        
        result = process_single_paper(paper, table_path, headers, domain, log_file)
        results.append(result)
        
        if result['status'] == 'extracted':
            success_count += 1
            print("✓")
        else:
            failed_count += 1
            print("✗")
            if result['errors']:
                print(f"  Errors: {'; '.join(result['errors'])}")
        
        # Save progress log after each paper
        with open(log_file, 'w', encoding='utf-8') as f:
            json.dump({
                'batch_info': {
                    'total': len(papers),
                    'success': success_count,
                    'failed': failed_count,
                    'timestamp': datetime.now().isoformat()
                },
                'results': results
            }, f, ensure_ascii=False, indent=2)
        
        # Brief pause to avoid overwhelming the system
        time.sleep(0.1)
    
    # Final summary
    print("\n" + "=" * 60)
    print("BATCH PROCESSING COMPLETE")
    print("=" * 60)
    print(f"Total papers: {len(papers)}")
    print(f"Successful: {success_count}")
    print(f"Failed: {failed_count}")
    print(f"Log saved to: {log_file}")
    
    return results


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: batch_process.py <papers_directory> <table_path> [output_directory]", file=sys.stderr)
        sys.exit(1)
    
    papers_dir = sys.argv[1]
    table_path = sys.argv[2]
    output_dir = sys.argv[3] if len(sys.argv) > 3 else None
    
    batch_process(papers_dir, table_path, output_dir)


path: scripts/extract_paper.py
content: #!/usr/bin/env python3
"""Extract text from academic papers (PDF, DOCX, TXT) with section detection."""

import sys
import os
import re
import json
from pathlib import Path

# Section detection patterns
SECTION_PATTERNS = {
    'abstract': r'(?:abstract|摘要)\s*[\n\r]+',
    'introduction': r'(?:introduction|intro|背景|引言)\s*[\n\r]+',
    'methods': r'(?:methods|methodology|materials?\s+and\s+methods|方法|材料与方法)\s*[\n\r]+',
    'results': r'(?:results|发现|结果)\s*[\n\r]+',
    'discussion': r'(?:discussion|讨论)\s*[\n\r]+',
    'conclusion': r'(?:conclusion|conclusions|结论)\s*[\n\r]+',
    'references': r'(?:references|bibliography|参考文献)\s*[\n\r]+',
    'acknowledgments': r'(?:acknowledgments?|acknowledgements?|致谢)\s*[\n\r]+',
    'supplementary': r'(?:supplementary|supporting|补充材料)\s*[\n\r]+',
}


def detect_sections(text):
    """Detect paper sections and return structured content."""
    sections = {}
    current_pos = 0
    
    # Find all section headers and their positions
    section_positions = []
    for section_name, pattern in SECTION_PATTERNS.items():
        for match in re.finditer(pattern, text, re.IGNORECASE):
            section_positions.append((match.start(), match.end(), section_name))
    
    # Sort by position
    section_positions.sort()
    
    # Extract content between sections
    for i, (start, end, name) in enumerate(section_positions):
        if i + 1 < len(section_positions):
            next_start = section_positions[i + 1][0]
            sections[name] = text[end:next_start].strip()
        else:
            sections[name] = text[end:].strip()
    
    return sections


def extract_pdf(path):
    """Extract text from PDF using multiple methods."""
    text = ""
    errors = []
    
    # Method 1: pdfplumber
    try:
        import pdfplumber
        with pdfplumber.open(path) as pdf:
            for page in pdf.pages:
                page_text = page.extract_text()
                if page_text:
                    text += page_text + "\n"
        if text.strip():
            return text, None
    except ImportError:
        errors.append("pdfplumber not installed")
    except Exception as e:
        errors.append(f"pdfplumber failed: {e}")
    
    # Method 2: PyMuPDF
    try:
        import fitz
        doc = fitz.open(path)
        for page in doc:
            text += page.get_text() + "\n"
        doc.close()
        if text.strip():
            return text, None
    except ImportError:
        errors.append("PyMuPDF not installed")
    except Exception as e:
        errors.append(f"PyMuPDF failed: {e}")
    
    # Method 3: OCR fallback
    try:
        from pdf2image import convert_from_path
        import pytesseract
        images = convert_from_path(path)
        for image in images:
            text += pytesseract.image_to_string(image, lang='chi_sim+eng') + "\n"
        if text.strip():
            return text, None
    except ImportError:
        errors.append("OCR tools not installed (pdf2image, pytesseract)")
    except Exception as e:
        errors.append(f"OCR failed: {e}")
    
    if text.strip():
        return text, "; ".join(errors) if errors else None
    return "[ERROR: Could not extract text from PDF]", "; ".join(errors)


def extract_docx(path):
    """Extract text from DOCX."""
    try:
        from docx import Document
        doc = Document(path)
        return "\n".join([para.text for para in doc.paragraphs]), None
    except ImportError:
        return "[ERROR: python-docx not installed]", "python-docx not installed"
    except Exception as e:
        return f"[ERROR: Could not read DOCX: {e}]", str(e)


def extract_txt(path):
    """Extract text from TXT."""
    encodings = ['utf-8', 'gbk', 'latin-1']
    errors = []
    for encoding in encodings:
        try:
            with open(path, 'r', encoding=encoding) as f:
                return f.read(), None
        except UnicodeDecodeError:
            errors.append(f"Failed with {encoding}")
        except Exception as e:
            errors.append(f"Error with {encoding}: {e}")
    return "[ERROR: Could not decode TXT file]", "; ".join(errors)


def extract(paper_path, structured=False):
    """Route to appropriate extractor based on file extension."""
    path = Path(paper_path)
    ext = path.suffix.lower()
    
    if ext == '.pdf':
        text, error = extract_pdf(path)
    elif ext == '.docx':
        text, error = extract_docx(path)
    elif ext == '.txt':
        text, error = extract_txt(path)
    else:
        return f"[ERROR: Unsupported file format: {ext}]"
    
    if structured and not text.startswith("[ERROR:"):
        sections = detect_sections(text)
        result = {
            'full_text': text,
            'sections': sections,
            'metadata': {
                'file': str(path),
                'format': ext.lstrip('.'),
                'size': path.stat().st_size,
                'extraction_error': error
            }
        }
        return json.dumps(result, ensure_ascii=False, indent=2)
    
    return text


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: extract_paper.py <paper_path> [--structured]", file=sys.stderr)
        sys.exit(1)
    
    paper_path = sys.argv[1]
    structured = '--structured' in sys.argv
    
    result = extract(paper_path, structured=structured)
    # Write to stdout with UTF-8 encoding to avoid Windows console issues
    import sys
    sys.stdout.reconfigure(encoding='utf-8')
    print(result)


path: scripts/read_table.py
content: #!/usr/bin/env python3
"""Read table headers from XLSX or CSV with metadata extraction."""

import sys
import json
from pathlib import Path


def infer_domain(headers):
    """Infer academic domain from header patterns."""
    header_str = ' '.join(headers).lower()
    
    psych_markers = ['participants', '被试', 'experimental design', '实验设计', 'measures', '测量', 'manipulation', '操纵', 'effect size', '效应量', 'preregistration', '预注册']
    neuro_markers = ['imaging', '成像', 'preprocessing', '预处理', 'task paradigm', '任务范式', 'brain regions', '脑区', 'analysis type', '分析类型', 'behavioral-neural', '行为-神经']
    cs_markers = ['algorithm', '算法', 'model architecture', '模型架构', 'dataset', '数据集', 'training', '训练', 'metrics', '指标', 'code availability', '代码', 'ablation', '消融']
    brain_markers = ['species', '物种', 'recording method', '记录方法', 'stimulation', '刺激', 'cell type', '细胞类型', 'behavioral task', '行为任务']
    
    scores = {
        'psychology': sum(1 for m in psych_markers if m in header_str),
        'cognitive_neuroscience': sum(1 for m in neuro_markers if m in header_str),
        'computer_science': sum(1 for m in cs_markers if m in header_str),
        'brain_science': sum(1 for m in brain_markers if m in header_str),
    }
    
    max_score = max(scores.values())
    if max_score > 0:
        domain = max(scores, key=scores.get)
        return domain
    return 'general'


def infer_header_type(header):
    """Infer data type and constraints for a header."""
    h = header.lower()
    
    # Text fields
    if any(k in h for k in ['title', '标题', 'author', '作者', 'journal', '期刊', 'objective', '目标', 'conclusion', '结论', 'method', '方法', 'finding', '发现']):
        return {'type': 'text', 'constraints': 'free text, preserve original language'}
    
    # Numeric fields
    if any(k in h for k in ['year', '年份', 'size', '样本量', 'number', '数量']):
        return {'type': 'number', 'constraints': 'integer, 4 digits for year'}
    
    # Measurement fields
    if any(k in h for k in ['effect size', '效应量', 'correlation', '相关', 'metric', '指标', 'accuracy', '精确度']):
        return {'type': 'measurement', 'constraints': 'numeric with units/statistics'}
    
    # URL fields
    if any(k in h for k in ['doi', 'code', '代码', 'link', '链接', 'url']):
        return {'type': 'url', 'constraints': 'valid URL format'}
    
    # List fields
    if any(k in h for k in ['authors', 'author', '作者', 'region', '脑区', 'measure', '测量']):
        return {'type': 'list', 'constraints': 'semicolon-separated values'}
    
    return {'type': 'text', 'constraints': 'free text'}


def read_xlsx_headers(path):
    """Read headers from XLSX with metadata."""
    try:
        import openpyxl
        wb = openpyxl.load_workbook(path)
        ws = wb.active
        headers = []
        for cell in ws[1]:
            if cell.value:
                headers.append(str(cell.value).strip())
        
        # Check for second row with descriptions
        descriptions = {}
        if ws.max_row >= 2:
            for col_idx, header in enumerate(headers, 1):
                desc_cell = ws.cell(row=2, column=col_idx)
                if desc_cell.value and desc_cell.value != header:
                    descriptions[header] = str(desc_cell.value).strip()
        
        return headers, descriptions
    except ImportError:
        return ["[ERROR: openpyxl not installed]"], {}
    except Exception as e:
        return [f"[ERROR: Could not read XLSX: {e}]"], {}


def read_csv_headers(path):
    """Read headers from CSV with metadata."""
    import csv
    encodings = ['utf-8', 'gbk', 'latin-1']
    
    for encoding in encodings:
        try:
            with open(path, 'r', encoding=encoding, newline='') as f:
                reader = csv.reader(f)
                headers = next(reader, [])
                headers = [h.strip() for h in headers if h.strip()]
                
                # Check for second row with descriptions
                descriptions = {}
                try:
                    second_row = next(reader, [])
                    for i, header in enumerate(headers):
                        if i < len(second_row) and second_row[i] and second_row[i] != header:
                            descriptions[header] = second_row[i].strip()
                except StopIteration:
                    pass
                
                return headers, descriptions
        except UnicodeDecodeError:
            continue
        except Exception as e:
            return [f"[ERROR: Could not read CSV: {e}]"], {}
    
    return ["[ERROR: Could not decode CSV file]"], {}


def read_headers(table_path):
    """Route to appropriate reader based on file extension."""
    path = Path(table_path)
    ext = path.suffix.lower()
    
    if ext == '.xlsx':
        headers, descriptions = read_xlsx_headers(path)
    elif ext == '.csv':
        headers, descriptions = read_csv_headers(path)
    else:
        return [f"[ERROR: Unsupported table format: {ext}]"]
    
    if headers[0].startswith("[ERROR:"):
        return headers
    
    # Build metadata
    domain = infer_domain(headers)
    header_metadata = {}
    for header in headers:
        header_metadata[header] = {
            'type': infer_header_type(header)['type'],
            'constraints': infer_header_type(header)['constraints'],
            'description': descriptions.get(header, '')
        }
    
    result = {
        'headers': headers,
        'domain': domain,
        'header_metadata': header_metadata,
        'file': str(path),
        'format': ext.lstrip('.')
    }
    
    return result


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: read_table.py <table_path>", file=sys.stderr)
        sys.exit(1)
    
    result = read_headers(sys.argv[1])
    
    if isinstance(result, list):
        # Error case
        print("\n".join(result))
    else:
        print(json.dumps(result, ensure_ascii=False, indent=2))


path: scripts/write_table.py
content: #!/usr/bin/env python3
"""Append rows to table (XLSX or CSV) preserving existing content and format."""

import sys
import json
import csv
from pathlib import Path
from difflib import SequenceMatcher


def similar(a, b):
    """Calculate string similarity ratio."""
    if not a or not b:
        return 0.0
    return SequenceMatcher(None, str(a).lower(), str(b).lower()).ratio()


def find_duplicate(xlsx_path, title, threshold=0.85):
    """Check if a paper with similar title already exists."""
    try:
        import openpyxl
        wb = openpyxl.load_workbook(xlsx_path)
        ws = wb.active
        
        # Find title column
        headers = [cell.value for cell in ws[1]]
        title_col = None
        for i, h in enumerate(headers):
            if h and ('title' in str(h).lower() or '标题' in str(h)):
                title_col = i + 1
                break
        
        if not title_col:
            return False, None
        
        # Check all existing titles
        for row in range(2, ws.max_row + 1):
            existing_title = ws.cell(row=row, column=title_col).value
            if existing_title and similar(existing_title, title) > threshold:
                return True, existing_title
        
        return False, None
    except Exception:
        return False, None


def find_duplicate_csv(csv_path, title, threshold=0.85):
    """Check if a paper with similar title already exists in CSV."""
    encodings = ['utf-8', 'gbk', 'latin-1']
    
    for encoding in encodings:
        try:
            with open(csv_path, 'r', encoding=encoding, newline='') as f:
                reader = csv.reader(f)
                headers = next(reader, [])
                
                # Find title column
                title_col = None
                for i, h in enumerate(headers):
                    if h and ('title' in str(h).lower() or '标题' in str(h)):
                        title_col = i
                        break
                
                if title_col is None:
                    return False, None
                
                # Check all existing titles
                for row in reader:
                    if len(row) > title_col:
                        existing_title = row[title_col]
                        if existing_title and similar(existing_title, title) > threshold:
                            return True, existing_title
            
            return False, None
        except UnicodeDecodeError:
            continue
        except Exception:
            return False, None
    
    return False, None


def validate_data(row_data, header_metadata=None):
    """Validate extracted data before writing."""
    errors = []
    warnings = []
    
    # Check for required fields
    if not row_data:
        errors.append("Empty data")
        return False, errors, warnings
    
    # Check for hallucination indicators
    hallucination_indicators = [
        '可能', 'probably', 'maybe', 'perhaps', '似乎', 'seems',
        '根据常识', 'according to common sense', '通常', 'usually'
    ]
    
    for key, value in row_data.items():
        if isinstance(value, dict):
            val_str = str(value.get('value', ''))
        else:
            val_str = str(value)
        
        # Check for hallucination language
        for indicator in hallucination_indicators:
            if indicator in val_str.lower():
                warnings.append(f"Field '{key}' may contain inference: '{val_str[:50]}...'")
        
        # Check for suspicious patterns
        if val_str.startswith('根据') or val_str.startswith('Based on'):
            warnings.append(f"Field '{key}' starts with inference language")
    
    return len(errors) == 0, errors, warnings


def append_xlsx(table_path, row_data, validate=True):
    """Append row to XLSX preserving format."""
    try:
        import openpyxl
        
        # Validate if requested
        if validate:
            valid, errors, warnings = validate_data(row_data)
            if not valid:
                print(f"[ERROR: Validation failed: {'; '.join(errors)}]", file=sys.stderr)
                return False
            if warnings:
                print(f"[WARNING: {'; '.join(warnings)}]", file=sys.stderr)
        
        wb = openpyxl.load_workbook(table_path)
        ws = wb.active
        
        # Get headers from first row
        headers = [cell.value for cell in ws[1]]
        
        # Check for duplicates if title exists
        if isinstance(row_data, dict):
            title_value = row_data.get('Title') or row_data.get('标题') or row_data.get('title')
            if title_value:
                is_dup, dup_title = find_duplicate(table_path, title_value)
                if is_dup:
                    print(f"[WARNING: Duplicate detected - similar to '{dup_title}']", file=sys.stderr)
                    return False
        
        # Find next empty row
        next_row = ws.max_row + 1
        
        # Write data matching headers
        for col_idx, header in enumerate(headers, 1):
            if header:
                # Handle nested dict (with confidence) or direct value
                if header in row_data:
                    value = row_data[header]
                    if isinstance(value, dict):
                        ws.cell(row=next_row, column=col_idx, value=value.get('value', ''))
                    else:
                        ws.cell(row=next_row, column=col_idx, value=value)
        
        wb.save(table_path)
        return True
    except Exception as e:
        print(f"[ERROR: Failed to write XLSX: {e}]", file=sys.stderr)
        return False


def append_csv(table_path, row_data, validate=True):
    """Append row to CSV."""
    encodings = ['utf-8', 'gbk', 'latin-1']
    
    # Validate if requested
    if validate:
        valid, errors, warnings = validate_data(row_data)
        if not valid:
            print(f"[ERROR: Validation failed: {'; '.join(errors)}]", file=sys.stderr)
            return False
        if warnings:
            print(f"[WARNING: {'; '.join(warnings)}]", file=sys.stderr)
    
    for encoding in encodings:
        try:
            # Read headers
            with open(table_path, 'r', encoding=encoding, newline='') as f:
                reader = csv.reader(f)
                headers = next(reader, [])
            
            # Check for duplicates if title exists
            if isinstance(row_data, dict):
                title_value = row_data.get('Title') or row_data.get('标题') or row_data.get('title')
                if title_value:
                    is_dup, dup_title = find_duplicate_csv(table_path, title_value)
                    if is_dup:
                        print(f"[WARNING: Duplicate detected - similar to '{dup_title}']", file=sys.stderr)
                        return False
            
            # Append row
            with open(table_path, 'a', encoding=encoding, newline='') as f:
                writer = csv.writer(f)
                row = []
                for h in headers:
                    h_stripped = h.strip() if h else ''
                    if h_stripped in row_data:
                        value = row_data[h_stripped]
                        if isinstance(value, dict):
                            row.append(value.get('value', ''))
                        else:
                            row.append(value)
                    else:
                        row.append('')
                writer.writerow(row)
            return True
        except UnicodeDecodeError:
            continue
        except Exception as e:
            print(f"[ERROR: Failed to write CSV: {e}]", file=sys.stderr)
            return False
    
    return False


def write_row(table_path, row_data, validate=True):
    """Route to appropriate writer based on file extension."""
    path = Path(table_path)
    ext = path.suffix.lower()
    
    if ext == '.xlsx':
        return append_xlsx(path, row_data, validate=validate)
    elif ext == '.csv':
        return append_csv(path, row_data, validate=validate)
    else:
        print(f"[ERROR: Unsupported table format: {ext}]", file=sys.stderr)
        return False


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: write_table.py <table_path> '<json_data>' [--validate]", file=sys.stderr)
        sys.exit(1)
    
    table_path = sys.argv[1]
    row_data = json.loads(sys.argv[2])
    validate = '--validate' in sys.argv
    
    success = write_row(table_path, row_data, validate=validate)
    sys.exit(0 if success else 1)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)