Skill: vardhineediganesh877-ui/deep-research-pipeline (version 2.0.1)

--- name: deep-research version: 2.0.1 description: "Multi-stage deep research with reflection loops, multi-query retrieval, LLM chunk selection, and citation integrity. Use when: deep research, literature review, topic investigation, multi-source analysis, fact-checking, competitive analysis, technology deep-dives." --- # Deep Research Pipeline Deep Research Pipeline turns broad questions into cited, publication-quality reports through a staged research workflow: planning, multi-query retrieval, chunk selection, analysis, reflection, writing, and optional verification. It is designed for research that should not be answered from memory or a single search result. The pipeline keeps claims tied to sources, surfaces contradictions, tracks gaps, and can resume from checkpoints. ## Why Use It - **Multi-stage research, not one-shot summarization** — separate researcher, analyst, reflection, and writer stages. - **Citation integrity** — findings and final claims trace back to URLs/sources. - **Reflection loops** — the pipeline checks coverage and decides whether another cycle is needed. - **Portable LLM config** — supports `LLM_API_KEY`/`LLM_API_BASE`, OpenAI-compatible endpoints, or Z.AI GLM. - **Operational controls** — checkpoint/resume, time limits, token budgets, output formats, and mock mode. ## When to Use Deep research, comprehensive analysis, literature reviews, competitive analysis, fact-checking, technology deep-dives — anything needing multiple sources, synthesis, and verified citations. ## Quick Start ```bash cd skills/deep-research # Optional: configure any OpenAI-compatible provider export LLM_API_KEY="your-key" export LLM_API_BASE="https://api.example.com/v1" export LLM_MODEL="your-model" # Or use OpenAI-compatible env names export OPENAI_API_KEY="your-key" export OPENAI_BASE_URL="https://api.example.com/v1" # Run a report python3 scripts/research_pipeline.py \ "Compare Vercel, Netlify, and Cloudflare Pages in 2026" \ --max-cycles 2 \ --format report \ --output report.md # Test without API calls python3 scripts/research_pipeline.py "test question" --mock --output report.md ``` If no universal/OpenAI-compatible variables are set, the skill still supports Z.AI via `ZAI_API_KEY` and `ZAI_API_ENDPOINT`. ## Architecture ``` ORCHESTRATOR (you) │ ├── Plan → Decompose question into research dimensions │ ├── REFLECTION LOOP (0-8 cycles) │ ├── Researcher Agent (parallel) → multi-query search + chunk selection │ ├── Analyst Agent → dedupe + themes + contradictions │ └── Reflection → coverage check, gap analysis, continue decision │ ├── Writer Agent → polished report (report/summary/brief/json) │ └── Verify (optional) → adversarial fact-check ``` **Key principle:** Orchestrator NEVER searches directly. Clean output flows between stages only. ## Two Modes ### Mode 1: Full Pipeline CLI (Recommended) Use the enhanced `research_pipeline.py` for automated end-to-end research: ```bash # Full research with all features python3 scripts/research_pipeline.py "What is the state of quantum computing in 2026?" \ --max-cycles 3 \ --output report.md \ --format report # Mock mode (no API calls, for testing) python3 scripts/research_pipeline.py "test question" --mock --output report.md # With budget limits python3 scripts/research_pipeline.py "question" \ --max-cycles 3 --time-limit 300 --token-limit 40000 # Resume from checkpoint python3 scripts/research_pipeline.py "question" \ --resume checkpoint.json --output report.md # Explicit dimensions python3 scripts/research_pipeline.py "question" \ --dimensions architecture benchmarks limitations \ --output report.md --format summary ``` **CLI Flags:** | Flag | Default | Description | |------|---------|-------------| | `--max-cycles` | 3 | Max research cycles (1-8) | | `--mock` | false | Use mock data, no API calls | | `--output` / `-o` | stdout | Output file path | | `--format` / `-f` | report | Output format: `report`, `summary`, `brief`, `json` | | `--time-limit` | 900 | Max seconds for entire pipeline | | `--token-limit` | 60000 | Max estimated tokens | | `--checkpoint` | none | Save checkpoints to path | | `--resume` | none | Resume from checkpoint file | | `--dimensions` | auto | Explicit research dimensions | | `--no-parallel` | false | Research dimensions sequentially | **Output formats:** - `report` — Full markdown: Executive Summary → Key Findings → Detailed Analysis → Contradictions → Gaps → Sources → Methodology - `summary` — Executive summary + top 5 findings + sources - `brief` — Bullet-point format for quick scanning - `json` — Structured JSON with annotated findings and metadata ### Mode 2: Orchestrated Sub-Agents (For complex research) Use when you need fine-grained control over each stage or parallel dimension research with sub-agents. ## Workflow (Orchestrated Mode) ### Phase 1: Planning - Analyze question, create slug, make `memory/research/<slug>/` directory - Generate research plan with dimensions and questions - Save to `plan.md` ### Phase 2: Research Cycle (repeat up to 8 times) #### Step A: Spawn Researcher Agent(s) Use `sessions_spawn` with a **task brief** (NOT the full query): ```json { "dimension": "technical architecture", "specific_questions": ["How does X work?", "What are Y's components?"], "context_limit": 5000, "max_sources": 10 } ``` Researcher agent does: 1. **Multi-query generation** — `scripts/query_generator.py` produces 3-5 variants 2. **Parallel search** — `web_search` for each variant 3. **Content fetching** — `web_fetch` for top results 4. **LLM chunk selection** — `scripts/chunk_selector.py` scores each chunk (≥0.7) 5. **Context expansion** — `scripts/context_expander.py` fetches surrounding content 6. Output: JSON findings with citations **Can spawn 2-3 researcher agents in parallel for different dimensions.** #### Step B: Spawn Analyst Agent After researcher(s) complete, spawn analyst with their combined output: 1. Deduplicate overlapping findings 2. Flag contradictions (explicit + implicit) 3. Group into thematic clusters 4. Identify gaps 5. Output: Cleaned JSON + gap list #### Step C: Run Reflection After analyst completes, run `scripts/reflection.py`: 1. What's covered? (themes + confidence scores) 2. What gaps remain? (unanswered questions) 3. What contradictions emerged? 4. New directions discovered? 5. **Should continue?** (coverage ≥ 0.8 + minor gaps → stop) Save reflection to `memory/research/<slug>/reflection-cycle-N.md` #### Continue Decision - Coverage ≥ 0.8 AND gaps minor → proceed to Phase 3 - Major contradictions → spawn targeted researcher - Significant gaps → another researcher cycle - Hard stop at cycle 8 ### Phase 3: Write Report Use the **Writer Agent** (`scripts/writer.py`) for publication-quality output: ```python # From Python from writer import WriterAgent, OutputFormat, write_report # Generate report using WriterAgent agent = WriterAgent(use_llm=True) result = agent.write_report( analyst_output, # from analyst or run_analyst() question="What is RAG?", fmt=OutputFormat.REPORT, ) # Or use convenience function result = write_report(analyst_output, question, fmt="report") # Save to file from writer import save_report save_report(result, "output/report.md") ``` **Report features:** - 🟢🟡🟠🔴 Confidence indicators on every finding - `[source_url]` inline citations throughout - ⚠️ Contradiction callout boxes where sources disagree - Structured sections: Summary → Findings → Analysis → Contradictions → Gaps → Sources → Methodology - Template-based fallback when no LLM available ### Phase 4: Verify (optional sub-agent) Spawn adversarial verifier: - Anchor every claim to source - Verify URLs with `web_fetch` - Remove unsourced claims - Save to `review.md` ### Phase 5: Deliver - Fix any FATAL issues from review - Copy to `final.md` - Write `provenance.md` (date, cycles, sources, verification status) - Send summary to user ## Python API ```python import sys, os sys.path.insert(0, os.path.expanduser("~/.openclaw/workspace/skills/deep-research/scripts")) from research_pipeline import run_enhanced_pipeline result = run_enhanced_pipeline( question="What is the state of quantum computing in 2026?", max_cycles=3, dimensions=["hardware", "algorithms", "applications", "challenges"], mock_mode=False, output_format="report", time_limit=900, token_limit=60000, checkpoint_path="checkpoint.json", # auto-saves progress parallel_dimensions=True, # parallel research per dimension ) # result["report"] = markdown string # result["cycles_completed"] = int # result["final_coverage"] = float (0.0-1.0) # result["metadata"] = dict with timing, findings count, etc. ``` ## Scripts | Script | Purpose | Usage | |--------|---------|-------| | `research_pipeline.py` | **Full pipeline orchestration** | `python3 scripts/research_pipeline.py "question" --max-cycles 3` | | `query_generator.py` | Generate 3-5 search query variants | `python3 scripts/query_generator.py -q "..."` | | `chunk_selector.py` | LLM scores chunks, filters by threshold | `python3 scripts/chunk_selector.py -q "..." -c chunks.json` | | `context_expander.py` | Fetch surrounding context for incomplete chunks | `python3 scripts/context_expander.py -s selected.json -q "..."` | | `reflection.py` | Mandatory gap/contradiction check | `python3 scripts/reflection.py -q "..." -f findings.json -c 1` | | `writer.py` | Publication-quality report generation | `from writer import WriterAgent, write_report` | | `analyst.py` | Dedup + themes + contradictions (no API needed) | `from analyst import analyze_findings` | | `researcher.py` | Multi-source research orchestration | `from researcher import research, research_dimension` | | `research_sources.py` | Search adapters (web, GitHub, docs) | `from research_sources import WebSearchSource` | | `fact-checker.py` | Claim extraction + source ranking | `python3 scripts/fact-checker.py "text" --sources '["url1"]'` | All LLM-enabled scripts use the shared provider-agnostic `llm_client.py`. Provider resolution order: 1. `LLM_API_KEY` + `LLM_API_BASE` + optional `LLM_MODEL` 2. `OPENAI_API_KEY` + `OPENAI_API_BASE` / `OPENAI_BASE_URL` + optional `OPENAI_MODEL` 3. `ZAI_API_KEY` + optional `ZAI_API_ENDPOINT` / `GLM_MODEL` If no key is configured, use `--mock` for local pipeline testing or rely on scripts with rule-based fallbacks where available. ## Examples ### Example 1: Quick Competitive Analysis ```bash python3 scripts/research_pipeline.py \ "Compare Vercel vs Netlify vs Cloudflare Pages features and pricing 2026" \ --max-cycles 2 \ --dimensions features pricing performance ecosystem \ --format summary \ --output competitive-analysis.md ``` ### Example 2: Deep Technology Research ```bash python3 scripts/research_pipeline.py \ "What is the current state of AI agent frameworks?" \ --max-cycles 4 \ --time-limit 600 \ --token-limit 80000 \ --checkpoint /tmp/ai-agents-checkpoint.json \ --format report \ --output ai-agents-research.md ``` ### Example 3: Literature Review (mock mode for testing) ```bash python3 scripts/research_pipeline.py \ "What does the research say about transformer architecture efficiency?" \ --mock \ --max-cycles 3 \ --format report \ --output literature-review.md ``` ### Example 4: Bullet Brief for Quick Scanning ```bash python3 scripts/research_pipeline.py \ "What are the latest developments in Rust web frameworks?" \ --max-cycles 2 \ --format brief \ --output rust-web-brief.md ``` ### Example 5: JSON Output for Programmatic Use ```bash python3 scripts/research_pipeline.py \ "What is the market size of edge computing?" \ --max-cycles 2 \ --format json \ --output edge-computing-data.json ``` ## Integration with Night Shift To queue research plans for Night Shift execution: 1. **Create a research plan file:** ```json // memory/research/queued/<slug>.json { "question": "What is the state of quantum computing in 2026?", "max_cycles": 3, "dimensions": ["hardware", "algorithms", "applications"], "output_format": "report", "output_path": "memory/research/quantum-2026/final.md", "time_limit": 600, "created_at": "2026-04-25T06:00:00Z" } ``` 2. **Night Shift picks up queued plans** and runs them via: ```bash python3 scripts/research_pipeline.py "$QUESTION" \ --max-cycles $MAX_CYCLES \ --dimensions $DIMENSIONS \ --format $FORMAT \ --output $OUTPUT_PATH \ --time-limit $TIME_LIMIT ``` 3. **Results are saved** to `memory/research/<slug>/final.md` with provenance metadata. ## File Layout ``` memory/research/<slug>/ ├── plan.md # Research plan with dimensions ├── reflection-cycle-1.md # Reflection after each cycle ├── reflection-cycle-2.md ├── researcher-output-*.json # Raw researcher findings ├── analyst-output.json # Merged/deduped findings ├── draft.md # First draft ├── brief.md # Verified brief ├── review.md # Adversarial review (optional) ├── final.md # Final report ├── provenance.md # Metadata + source verification status └── checkpoint.json # Pipeline checkpoint (auto-saved) ``` ## Quick Mode Skip sub-agents and the full pipeline. Do 5-10 searches yourself. Still use evidence tables, verify URLs, cite sources. Shorter, inline in chat. ## Integrity Commandments 1. Never fabricate a source — no URL = don't mention it 2. Never claim existence without checking 3. Never extrapolate unread details 4. Read before summarizing 5. No fake certainty — never say "verified" unless checked 6. Never invent numbers/benchmarks/comparisons 7. Separate observations from inferences 8. **Every claim traces to a source** — citation integrity is mandatory 9. **Reflection is not optional** — run it after every cycle 10. **Stage separation** — orchestrator never searches, researchers never see full plan ## Scale Decision - Single fact → Quick Mode (3-10 tool calls, no sub-agents) - 2-3 item comparison → 2 parallel researcher sub-agents, 2-3 cycles - Broad/multi-faceted → 3-4 researcher sub-agents, 3-5 cycles - PhD-level deep dive → 4+ researchers, 5-8 cycles ## See Also - **DOCS.md** — Full API reference, architecture diagrams, troubleshooting - **test_integration.py** — 29 integration tests covering the full pipeline - **test_pipeline.py** — 26 unit tests for individual components

path: DOCS.md
content: # Deep Research v2 — Documentation

Comprehensive multi-stage research pipeline with reflection loops, multi-query retrieval, LLM chunk selection, and citation integrity.

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR (you)                           │
│                                                                 │
│  1. PLAN ─── Decompose question into research dimensions       │
│              Save plan to memory/research/<slug>/plan.md        │
│                                                                 │
│  2. RESEARCH LOOP (0-8 cycles)                                 │
│     ┌──────────────────────────────────────────────────────┐   │
│     │  ┌─────────────┐    ┌─────────────┐    ┌──────────┐  │   │
│     │  │ RESEARCHER   │───▶│  ANALYST    │───▶│ REFLECT  │  │   │
│     │  │ (parallel)   │    │  (dedup +   │    │ (gap     │  │   │
│     │  │              │    │   themes +   │    │  check + │  │   │
│     │  │ query_gen    │    │   contrad.) │    │  budget) │  │   │
│     │  │ web_search   │    │             │    │          │  │   │
│     │  │ chunk_select │    │             │    │ continue?│  │   │
│     │  │ context_exp  │    │             │    │   │      │  │   │
│     │  └─────────────┘    └─────────────┘    │───┘      │  │   │
│     │                                            │      │  │   │
│     │              ◀─────────────────────────────┘      │  │   │
│     └──────────────────────────────────────────────────────┘   │
│                                                                 │
│  3. WRITE ─── WriterAgent produces publication-quality report   │
│              Multiple formats: report, summary, brief, json     │
│              Inline citations, confidence indicators             │
│                                                                 │
│  4. VERIFY ─── Adversarial fact-check (optional)                │
│                                                                 │
│  5. DELIVER ── Final report + provenance                        │
└─────────────────────────────────────────────────────────────────┘
```

## Components

### 1. Query Generator (`query_generator.py`)

Generates 3-5 diverse search query variants for a research question using LLM + rule-based fallback.

**API:**
```python
from query_generator import generate_queries

queries = generate_queries("How does RAG work?", max_queries=5)
# Returns: [{"type": "semantic", "query": "...", "rationale": "..."}, ...]
```

**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `question` | str | required | Research question |
| `max_queries` | int | 5 | Max query variants |
| `api_key` | str | None | ZAI_API_KEY override |

**Query types:** semantic, keyword, broad, specific, comparative

### 2. Research Sources (`research_sources.py`)

Multi-source search adapters:
- `WebSearchSource` — DuckDuckGo HTML/Lite + content fetching
- `GitHubSource` — GitHub REST API (repos + code search)
- `DocSource` — Documentation URL fetching + discovery

**API:**
```python
from research_sources import WebSearchSource, GitHubSource, DocSource, score_results, deduplicate_results

web = WebSearchSource()
results = web.search("OpenClaw skills", limit=5)

scored = score_results(results, topic="OpenClaw", use_llm=True)
unique = deduplicate_results(scored)
```

**Also provides:** `score_results()` (LLM + keyword fallback), `deduplicate_results()` (URL + content hash)

### 3. Chunk Selector (`chunk_selector.py`)

LLM scores each chunk for relevance, filters by threshold. The "hallucination killer."

**API:**
```python
from chunk_selector import select_relevant_chunks

selected = select_relevant_chunks(
    question="How does RAG work?",
    content=chunks,          # list of {content, url, title}
    min_score=0.7,           # threshold (0.0-1.0)
    max_context_tokens=8000, # batch size limit
)
```

**Batching:** Chunks are batched to fit within token limits. Each chunk scored 0.0-1.0.

### 4. Context Expander (`context_expander.py`)

Decides if selected chunks need surrounding context, fetches it from source URLs.

**API:**
```python
from context_expander import expand_context

expanded = expand_context(
    selected_chunks=selected,
    question="How does RAG work?",
    max_expansions=5,  # max URLs to re-fetch
)
```

### 5. Researcher Agent (`researcher.py`)

Orchestrates multi-source searches: query generation → parallel search → dedup → scoring → chunk selection → context expansion → finding extraction.

**API:**
```python
from researcher import research, research_dimension

# Full research on a topic
result = research("OpenClaw skills system", sources=["web", "github"])

# Dimension-based research (for subagents)
result = research_dimension(
    dimension="architecture",
    questions=["How does X work?", "What are the components?"],
    sources=["web", "github"],
)
```

**`research()` returns:**
```json
{
  "status": "complete|partial|no_results",
  "topic": "string",
  "findings": [{"claim": "...", "source_url": "...", "confidence": 0.9, "category": "..."}],
  "sources": [{"title": "...", "url": "...", "relevance_score": 0.8}],
  "gaps": [{"question": "...", "reason": "...", "importance": "high|medium|low"}],
  "metadata": {"queries_generated": 5, "raw_results": 20, "elapsed_seconds": 12.3}
}
```

**`research_dimension()` returns:**
```json
{
  "dimension": "architecture",
  "questions": ["..."],
  "findings": [...],
  "sources": [...],
  "coverage": 0.8,
  "question_coverage": {"q1": {"covered": true, "findings_count": 3}},
  "metadata": {...}
}
```

### 6. Analyst (`analyst.py`)

Local, deterministic analysis. No API calls needed.

**API:**
```python
from analyst import analyze_findings, AnalysisResult

result: AnalysisResult = analyze_findings(
    findings=[{"claim": "...", "source_url": "...", "confidence": 0.9}],
    dimensions=["architecture", "performance"],
)
```

**Returns `AnalysisResult` (frozen dataclass):**
| Field | Type | Description |
|-------|------|-------------|
| `themes` | list[dict] | 3-7 thematic clusters with keyword co-occurrence |
| `contradictions` | list[dict] | Opposing claims detected via polarity heuristics |
| `gaps` | list[dict] | Dimensions with insufficient coverage |
| `confidence_map` | dict[str, float] | Reliability scores per finding (0.0-1.0) |
| `coverage_score` | float | Overall dimension coverage (0.0-1.0) |

**Features:**
- Deduplication by hash + Jaccard similarity (≥0.6 threshold)
- Theme extraction via keyword co-occurrence clusters
- Contradiction detection via polarity heuristics (up/down/affirm/negate)
- Confidence scoring: source quality × extractor confidence + corroboration bonus

### 7. Reflection (`reflection.py`)

Coverage check + gap analysis + continue decision. Runs after every research cycle.

**API:**
```python
from reflection import ResearchPlan, reflect, ReflectionResult

plan = ResearchPlan(
    question="What is RAG?",
    dimensions=["architecture", "benchmarks", "limitations"],
    dimension_questions={"architecture": ["What is RAG?"]},
    budget_seconds=900,
    budget_tokens=60000,
)

result: ReflectionResult = reflect(plan, findings, cycle=1, max_cycles=8)
```

**Returns `ReflectionResult` (frozen dataclass):**
| Field | Type | Description |
|-------|------|-------------|
| `should_continue` | bool | Whether to run another research cycle |
| `gaps` | list[str] | Dimensions needing more research |
| `coverage_score` | float | Fraction of dimensions covered (0.0-1.0) |
| `next_dimensions` | list[str] | Suggested dimensions for next cycle |
| `summary` | str | Human-readable progress summary |

**Stop conditions:**
- Cycle ≥ max_cycles (hard stop at 8)
- Coverage ≥ 0.8 with no significant gaps
- Time budget ≥ 90% used
- Token budget ≥ 90% used
- Empty findings after cycle ≥ 1

### 8. Writer Agent (`writer.py`)

Publication-quality report generation with multiple output formats.

**API:**
```python
from writer import WriterAgent, write_report, OutputFormat, save_report

# Using the agent class
agent = WriterAgent(use_llm=True)
result = agent.write_report(
    analyst_output,           # from analyst or run_analyst()
    question="What is RAG?",
    fmt=OutputFormat.REPORT,  # REPORT, SUMMARY, BRIEF, JSON
    metadata={"cycles": 3},
)

# Using the convenience function
result = write_report(analyst_output, question, fmt="report", use_llm=True)

# Save to file
save_report(result, "output/report.md")
```

**Output formats:**

| Format | Description |
|--------|-------------|
| `report` | Full markdown with all sections (default) |
| `summary` | Executive summary + top 5 findings |
| `brief` | Bullet-point format for quick scanning |
| `json` | Structured JSON with annotated findings |

**Report sections:**
1. Title + metadata
2. Executive Summary (2-3 paragraphs)
3. Key Findings (numbered, with confidence badges 🟢🟡🟠🔴)
4. Detailed Analysis (one section per theme)
5. Contradictions (callout boxes for source disagreements)
6. Knowledge Gaps (prioritized by importance)
7. Sources (numbered reference list)
8. Methodology (research process description)

**Confidence indicators:**
- 🟢 High (≥0.8) — directly stated in sources
- 🟡 Medium (≥0.6) — strongly implied
- 🟠 Low (≥0.4) — inferred
- 🔴 Uncertain (<0.4) — weak support

### 9. Research Pipeline (`research_pipeline.py`)

Full orchestration with CLI, progress reporting, checkpoints, and budget awareness.

**CLI Usage:**
```bash
# Full pipeline
python3 research_pipeline.py "What is quantum computing?" \
    --max-cycles 3 \
    --output report.md \
    --format report

# Mock mode (no API calls)
python3 research_pipeline.py "test question" --mock --output report.md

# With budget limits
python3 research_pipeline.py "question" \
    --max-cycles 3 \
    --time-limit 300 \
    --token-limit 40000

# Resume from checkpoint
python3 research_pipeline.py "question" \
    --resume checkpoint.json \
    --output report.md

# Explicit dimensions
python3 research_pipeline.py "question" \
    --dimensions architecture benchmarks limitations \
    --output report.md

# Subcommand mode (backward compatible)
python3 research_pipeline.py researcher --question "..." --dimension "..."
python3 research_pipeline.py analyst --input researcher_output.json
python3 research_pipeline.py writer --input analyst_output.json --question "..."
```

**Python API:**
```python
from research_pipeline import (
    run_enhanced_pipeline, run_looping_pipeline,
    run_researcher, run_analyst, run_writer,
)

# Enhanced pipeline with all features
result = run_enhanced_pipeline(
    question="What is quantum computing?",
    max_cycles=3,
    dimensions=["hardware", "applications", "theory"],
    mock_mode=False,
    output_format="report",
    time_limit=900,
    token_limit=60000,
    checkpoint_path="checkpoint.json",
    resume_from=None,
    parallel_dimensions=True,
)

# Backward-compatible wrapper
result = run_looping_pipeline(question="...", max_cycles=3, mock_mode=True)
```

**CLI Flags:**
| Flag | Default | Description |
|------|---------|-------------|
| `--max-cycles` | 3 | Max research cycles (1-8) |
| `--mock` | false | Use mock data, no API calls |
| `--output` / `-o` | stdout | Output file path |
| `--format` / `-f` | report | Output format (report/summary/brief/json) |
| `--time-limit` | 900 | Max seconds |
| `--token-limit` | 60000 | Max estimated tokens |
| `--checkpoint` | none | Save checkpoints to path |
| `--resume` | none | Resume from checkpoint file |
| `--dimensions` | auto | Explicit research dimensions |
| `--no-parallel` | false | Research dimensions sequentially |

### 10. Fact Checker (`fact-checker.py`)

Simple claim extraction and source ranking.

```bash
python3 fact-checker.py "Text with claims" --sources '["url1", "url2"]'
```

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_API_KEY` | (none) | Universal provider API key, highest priority |
| `LLM_API_BASE` | (none) | Universal OpenAI-compatible base URL, highest priority |
| `LLM_MODEL` | provider default | Universal model name override |
| `OPENAI_API_KEY` | (none) | OpenAI or OpenAI-compatible API key |
| `OPENAI_API_BASE` / `OPENAI_BASE_URL` | OpenAI default | OpenAI-compatible endpoint |
| `OPENAI_MODEL` | provider default | OpenAI-compatible model override |
| `ZAI_API_KEY` | (none) | Z.AI GLM API key, still supported for local OpenClaw setups |
| `ZAI_API_ENDPOINT` | `https://api.z.ai/api/coding/paas/v4` | Z.AI GLM endpoint |
| `GLM_MODEL` | `glm-5-turbo` | Z.AI model override |
| `GITHUB_TOKEN` | (none) | GitHub API token (higher rate limits) |
| [REDACTED_SECRET] | 900 | Default time budget |
| `DEEP_RESEARCH_TOKEN_BUDGET` | 60000 | Default token budget |

### Key Parameters

| Parameter | Recommended Range | Description |
|-----------|------------------|-------------|
| `max_cycles` | 2-5 | More cycles = deeper research, longer time |
| `max_queries` | 3-5 | Query variants per question |
| `min_relevance` | 0.4-0.6 | Relevance filter threshold |
| `chunk_selector_threshold` | 0.7 | Min chunk score to keep |
| `max_context_tokens` | 8000-12000 | Context window per LLM call |

## Example Workflows

### Quick Research (single fact)
```python
# No subagents, direct pipeline
result = run_enhanced_pipeline(
    question="What is the current price of Bitcoin?",
    max_cycles=1,
    mock_mode=False,
    output_format="summary",
)
```

### Multi-Dimension Deep Dive
```python
result = run_enhanced_pipeline(
    question="What is the state of quantum computing in 2026?",
    max_cycles=4,
    dimensions=["hardware", "algorithms", "applications", "challenges", "investment"],
    time_limit=600,
    token_limit=80000,
    checkpoint_path="/tmp/quantum-checkpoint.json",
    parallel_dimensions=True,
)
```

### Resume Interrupted Research
```bash
# First run (gets interrupted)
python3 research_pipeline.py "Complex topic" \
    --max-cycles 5 \
    --checkpoint /tmp/checkpoint.json \
    --output report.md

# Resume
python3 research_pipeline.py "Complex topic" \
    --resume /tmp/checkpoint.json \
    --output report.md
```

### Generate Multiple Formats
```python
from writer import WriterAgent, OutputFormat

agent = WriterAgent(use_llm=True)

# Full report
report = agent.write_report(analyst_output, question, OutputFormat.REPORT)

# Executive summary
summary = agent.write_report(analyst_output, question, OutputFormat.SUMMARY)

# Quick brief
brief = agent.write_report(analyst_output, question, OutputFormat.BRIEF)

# Structured JSON
json_out = agent.write_report(analyst_output, question, OutputFormat.JSON)
```

## Troubleshooting

### "LLM_API_KEY, OPENAI_API_KEY, or ZAI_API_KEY not set"
Set any supported provider configuration:
```bash
# Universal OpenAI-compatible provider
export LLM_API_KEY="your-key-here"
export LLM_API_BASE="https://api.example.com/v1"
export LLM_MODEL="your-model"

# Or OpenAI-compatible env names
export OPENAI_API_KEY="your-key-here"
export OPENAI_BASE_URL="https://api.example.com/v1"

# Or Z.AI GLM
export ZAI_API_KEY="your-key-here"
```
Or use `--mock` mode for testing without API access.

### Pipeline hangs
- Check network connectivity to your configured provider endpoint
- Reduce `--max-cycles` (try 1-2)
- Set explicit `--dimensions` to skip query generation
- Use `--time-limit` to enforce a timeout

### Empty results
- Try broader search queries
- Check if DuckDuckGo is accessible from your network
- Use `--dimensions` with explicit terms
- Try adding more source types

### Low quality findings
- Increase `max_queries` (5-7)
- Lower `min_relevance` threshold (0.3-0.4)
- Run more cycles (3-5)
- Add GitHub/docs sources

### Tests failing
```bash
# Run all tests
cd skills/deep-research/scripts
python3 -m pytest test_analyst.py test_pipeline.py test_integration.py -v

# Run specific test class
python3 -m pytest test_integration.py::TestWriterAgentReport -v
```

### Checkpoint file format
```json
{
  "plan": {
    "question": "...",
    "dimensions": ["dim1", "dim2"],
    "dimension_questions": {"dim1": ["q1"]},
    "budget_seconds": 900,
    "budget_tokens": 60000,
    "created_at_unix": 1714022400.0
  },
  "cycle_history": [...],
  "all_researcher_outputs": [...]
}
```

## Design Principles

1. **Pure Python 3, stdlib only** — no external dependencies
2. **Stage separation** — orchestrator never searches, researchers never see full plan
3. **Citation integrity** — every claim traces to a source URL
4. **Reflection is mandatory** — run after every cycle
5. **Graceful degradation** — LLM failures fall back to rule-based methods
6. **Budget awareness** — time and token limits prevent runaway research


path: scripts/__init__.py
content: # Deep Research v2 — Onyx-inspired research pipeline


path: scripts/analyst.py
content: """Deep Research v2 — Analyst (local, deterministic).

Implements lightweight synthesis utilities for a list of extracted findings.

Core entrypoint:
    analyze_findings(findings: list[dict], dimensions: list[str]) -> AnalysisResult

Expected finding fields (best-effort, all optional):
    - claim / content / text: str
    - source_url / url: str
    - dimension: str
    - confidence: float (0..1) from extractor (treated as claim-level certainty)
    - source_score: float (0..1) or source.score: float (0..1)
"""

from __future__ import annotations

from dataclasses import dataclass
import hashlib
import math
import re
from collections import Counter, defaultdict



@dataclass(frozen=True)
class AnalysisResult:
    themes: list[dict]
    contradictions: list[dict]
    gaps: list[dict]
    confidence_map: dict[str, float]
    coverage_score: float


_WORD_RE = re.compile(r"[a-z0-9][a-z0-9\-']{1,}")
_NEGATIONS = {"no", "not", "never", "none", "without", "cannot", "can't", "won't"}
_DIRECTION_UP = {"increase", "increas", "increases", "increased", "higher", "more", "up", "rise", "ris", "rises", "rising", "grow", "grows", "growth"}
_DIRECTION_DOWN = {"decrease", "decreas", "decreases", "decreased", "lower", "less", "down", "fall", "falls", "falling", "decline", "declines", "declined", "drop", "drops", "dropped"}

# A tiny, intentionally conservative stopword list (stdlib-only).
_STOPWORDS = {
    "a",
    "an",
    "and",
    "are",
    "as",
    "at",
    "be",
    "by",
    "for",
    "from",
    "has",
    "have",
    "in",
    "into",
    "is",
    "it",
    "its",
    "of",
    "on",
    "or",
    "that",
    "the",
    "their",
    "this",
    "to",
    "was",
    "were",
    "will",
    "with",
}


def analyze_findings(findings: list[dict], dimensions: list[str]) -> AnalysisResult:
    """Analyze a list of extracted findings across given research dimensions.

    Steps:
      1) Deduplicate by simple hash + Jaccard similarity on token sets
      2) Theme extraction via keyword co-occurrence clusters (3-7 clusters)
      3) Contradiction detection via keyword overlap + polarity heuristics
      4) Confidence scoring from source quality + corroboration count
      5) Gap analysis across provided dimensions + overall coverage score
    """
    normalized_dimensions = [d.strip() for d in dimensions if str(d).strip()]

    prepared = [_prepare_finding(f) for f in findings if _get_text(f)]
    deduped = _deduplicate(prepared)

    themes = _extract_themes(deduped)
    contradictions = _detect_contradictions(deduped)
    confidence_map = _score_confidence(deduped)
    gaps, coverage_score = _gap_analysis(deduped, normalized_dimensions)

    return AnalysisResult(
        themes=themes,
        contradictions=contradictions,
        gaps=gaps,
        confidence_map=confidence_map,
        coverage_score=coverage_score,
    )


def _get_text(finding: dict) -> str:
    for key in ("claim", "content", "text"):
        v = finding.get(key)
        if isinstance(v, str) and v.strip():
            return v.strip()
    return ""


def _get_url(finding: dict) -> str:
    for key in ("source_url", "url"):
        v = finding.get(key)
        if isinstance(v, str) and v.strip():
            return v.strip()
    return ""


def _prepare_finding(finding: dict) -> dict:
    text = _get_text(finding)
    tokens = _tokens(text)
    finding_id = _stable_id(text)

    prepared = dict(finding)
    prepared["_id"] = finding_id
    prepared["_text"] = text
    prepared["_tokens"] = tokens
    prepared["_url"] = _get_url(finding)
    prepared["_dimension"] = (finding.get("dimension") or "").strip().lower()
    prepared["_extractor_confidence"] = _clamp01(_as_float(finding.get("confidence"), default=0.5))
    prepared["_source_quality"] = _clamp01(_infer_source_quality(finding))
    prepared["_polarity"] = _polarity(text, tokens)
    return prepared


def _stable_id(text: str) -> str:
    norm = _normalize_text(text)
    return hashlib.sha1(norm.encode("utf-8")).hexdigest()[:16]


def _normalize_text(text: str) -> str:
    text = text.lower().strip()
    text = re.sub(r"\s+", " ", text)
    return text


def _tokens(text: str) -> set[str]:
    words = {_stem(m.group(0)) for m in _WORD_RE.finditer(text.lower())}
    return {w for w in words if w not in _STOPWORDS and len(w) >= 3}


def _stem(word: str) -> str:
    """Tiny stemmer to improve similarity/polarity heuristics (no third-party deps)."""
    if len(word) <= 4:
        return word
    for suffix in ("ing", "ed"):
        if word.endswith(suffix) and len(word) - len(suffix) >= 4:
            return word[: -len(suffix)]
    if word.endswith("s") and len(word) >= 5:
        return word[:-1]
    return word


def _jaccard(a: set[str], b: set[str]) -> float:
    if not a and not b:
        return 1.0
    denom = len(a | b)
    if denom == 0:
        return 0.0
    return len(a & b) / denom


def _deduplicate(prepared: list[dict]) -> list[dict]:
    """Deduplicate by exact hash first, then Jaccard similarity."""
    if not prepared:
        return []

    by_hash: dict[str, list[dict]] = defaultdict(list)
    for f in prepared:
        by_hash[f["_id"]].append(f)

    collapsed: list[dict] = []
    for group in by_hash.values():
        collapsed.append(_merge_duplicates(group))

    # Second pass: near-duplicate merge via Jaccard on tokens.
    # Skip merging when polarities differ — opposing claims should survive for
    # contradiction detection.
    merged: list[dict] = []
    for f in collapsed:
        placed = False
        for i, existing in enumerate(merged):
            if _jaccard(f["_tokens"], existing["_tokens"]) >= 0.6:
                # Don't merge opposing claims
                if _is_opposing(f["_polarity"], existing["_polarity"]):
                    continue
                merged[i] = _merge_duplicates([existing, f])
                placed = True
                break
        if not placed:
            merged.append(f)

    return merged


def _merge_duplicates(group: list[dict]) -> dict:
    """Merge duplicates, preserving best confidences and collecting source URLs."""
    if len(group) == 1:
        f = dict(group[0])
        f["_sources"] = [f["_url"]] if f["_url"] else []
        return f

    # Choose representative: highest extractor_confidence then longest text.
    rep = max(group, key=lambda x: (x.get("_extractor_confidence", 0.0), len(x.get("_text", ""))))
    merged = dict(rep)
    sources: list[str] = []
    for f in group:
        u = f.get("_url", "")
        if u and u not in sources:
            sources.append(u)
    merged["_sources"] = sources
    merged["_extractor_confidence"] = max((f.get("_extractor_confidence", 0.0) for f in group), default=merged.get("_extractor_confidence", 0.5))
    merged["_source_quality"] = max((f.get("_source_quality", 0.0) for f in group), default=merged.get("_source_quality", 0.5))
    # Preserve dimension — prefer any non-empty dimension from the group.
    dims = [f.get("_dimension", "") for f in group if f.get("_dimension")]
    merged["_dimension"] = merged.get("_dimension", "") or (dims[0] if dims else "")
    return merged


def _extract_themes(deduped: list[dict]) -> list[dict]:
    """Group findings into 3-7 themes via keyword co-occurrence."""
    if not deduped:
        return []

    keyword_counts: Counter[str] = Counter()
    for f in deduped:
        keyword_counts.update(list(f["_tokens"]))

    # Candidate keywords: reasonably frequent, capped for determinism.
    candidate_keywords = [kw for kw, c in keyword_counts.most_common(40) if c >= 2]
    if not candidate_keywords:
        # Fall back: still produce 1-3 clusters with best tokens per finding.
        candidate_keywords = [kw for kw, _ in keyword_counts.most_common(12)]

    # Co-occurrence graph among candidate keywords.
    adjacency: dict[str, Counter[str]] = {kw: Counter() for kw in candidate_keywords}
    for f in deduped:
        present = [kw for kw in candidate_keywords if kw in f["_tokens"]]
        for i in range(len(present)):
            for j in range(i + 1, len(present)):
                a, b = present[i], present[j]
                adjacency[a][b] += 1
                adjacency[b][a] += 1

    # Build clusters as connected components using a minimal edge threshold.
    seen: set[str] = set()
    clusters: list[set[str]] = []
    for kw in candidate_keywords:
        if kw in seen:
            continue
        stack = [kw]
        component: set[str] = set()
        while stack:
            cur = stack.pop()
            if cur in seen:
                continue
            seen.add(cur)
            component.add(cur)
            for nbr, w in adjacency.get(cur, {}).items():
                if w >= 2 and nbr not in seen:
                    stack.append(nbr)
        clusters.append(component)

    # Assign findings to best cluster by keyword overlap.
    theme_items: list[dict] = []
    for idx, kwset in enumerate(clusters):
        if not kwset:
            continue
        finding_ids = []
        for f in deduped:
            if f["_tokens"] & kwset:
                finding_ids.append(f["_id"])
        if not finding_ids:
            continue

        keywords_sorted = sorted(list(kwset), key=lambda k: (-keyword_counts.get(k, 0), k))
        name = _theme_name(keywords_sorted[:4])
        theme_items.append(
            {
                "name": name,
                "keywords": keywords_sorted[:12],
                "finding_ids": finding_ids,
                "size": len(finding_ids),
            }
        )

    # Ensure 3-7 themes: split/merge heuristically.
    theme_items.sort(key=lambda t: (-t["size"], t["name"]))
    if len(theme_items) > 7:
        theme_items = theme_items[:7]
    if len(theme_items) < 3 and len(deduped) >= 3:
        # Create additional singleton-ish themes from top findings tokens.
        used_ids = {fid for t in theme_items for fid in t["finding_ids"]}
        for f in deduped:
            if f["_id"] in used_ids:
                continue
            kws = sorted(list(f["_tokens"]), key=lambda k: (-keyword_counts.get(k, 0), k))[:6]
            if not kws:
                continue
            theme_items.append(
                {
                    "name": _theme_name(kws[:3]),
                    "keywords": kws,
                    "finding_ids": [f["_id"]],
                    "size": 1,
                }
            )
            used_ids.add(f["_id"])
            if len(theme_items) >= 3:
                break

    return theme_items


def _theme_name(keywords: list[str]) -> str:
    if not keywords:
        return "General"
    top = keywords[0].replace("-", " ").strip()
    if len(keywords) >= 2:
        second = keywords[1].replace("-", " ").strip()
        return f"{top} / {second}"
    return top


def _polarity(text: str, tokens: set[str]) -> str:
    """Return a coarse polarity label: 'affirm', 'negate', 'up', 'down'.

    When both directional words (up/down) appear, the *last* directional
    word in the sentence is taken as the claim's actual direction — this
    handles cases like "increasing rates *decrease* affordability".
    """
    lowered = text.lower()
    has_neg = any(re.search(rf"\b{re.escape(n)}\b", lowered) for n in _NEGATIONS)
    has_up = bool(tokens & _DIRECTION_UP)
    has_down = bool(tokens & _DIRECTION_DOWN)

    if has_up and not has_down:
        return "up"
    if has_down and not has_up:
        return "down"
    # Both up and down present — use the last directional word as the claim direction.
    if has_up and has_down:
        last_dir = "affirm"
        for m in re.finditer(r"[a-z]+", lowered):
            w = _stem(m.group(0))
            if w in _DIRECTION_UP:
                last_dir = "up"
            elif w in _DIRECTION_DOWN:
                last_dir = "down"
        if last_dir != "affirm":
            return last_dir
    if has_neg:
        return "negate"
    return "affirm"


def _detect_contradictions(deduped: list[dict]) -> list[dict]:
    """Find likely opposing claims across sources (heuristic)."""
    contradictions: list[dict] = []
    if len(deduped) < 2:
        return contradictions

    # Only compare pairs with meaningful overlap to keep it cheap and reduce false positives.
    for i in range(len(deduped)):
        a = deduped[i]
        for j in range(i + 1, len(deduped)):
            b = deduped[j]
            overlap = _jaccard(a["_tokens"], b["_tokens"])
            if overlap < 0.35:
                continue

            if _is_opposing(a["_polarity"], b["_polarity"]):
                contradictions.append(
                    {
                        "finding_id_a": a["_id"],
                        "finding_id_b": b["_id"],
                        "claim_a": a["_text"],
                        "claim_b": b["_text"],
                        "source_a": a.get("_url") or (a.get("_sources") or [""])[0],
                        "source_b": b.get("_url") or (b.get("_sources") or [""])[0],
                        "overlap": round(overlap, 3),
                        "reason": f"Opposing polarity: {a['_polarity']} vs {b['_polarity']}",
                    }
                )

    # Dedup contradictions by unordered pair.
    seen_pairs: set[tuple[str, str]] = set()
    unique: list[dict] = []
    for c in contradictions:
        pair = tuple(sorted([c["finding_id_a"], c["finding_id_b"]]))
        if pair in seen_pairs:
            continue
        seen_pairs.add(pair)
        unique.append(c)
    unique.sort(key=lambda x: (-x["overlap"], x["finding_id_a"], x["finding_id_b"]))
    return unique


def _is_opposing(p1: str, p2: str) -> bool:
    opposing_pairs = {("up", "down"), ("down", "up"), ("affirm", "negate"), ("negate", "affirm")}
    return (p1, p2) in opposing_pairs


def _score_confidence(deduped: list[dict]) -> dict[str, float]:
    """Compute reliability score combining source quality + corroboration."""
    confidence_map: dict[str, float] = {}
    if not deduped:
        return confidence_map

    # Corroboration: count of other findings with high similarity.
    corroboration_count: dict[str, int] = {f["_id"]: 0 for f in deduped}
    for i in range(len(deduped)):
        for j in range(i + 1, len(deduped)):
            sim = _jaccard(deduped[i]["_tokens"], deduped[j]["_tokens"])
            if sim >= 0.6:
                corroboration_count[deduped[i]["_id"]] += 1
                corroboration_count[deduped[j]["_id"]] += 1

    for f in deduped:
        base = 0.55 * f["_source_quality"] + 0.45 * f["_extractor_confidence"]
        # Diminishing returns: +0.07 per corroboration up to +0.21.
        corr = min(3, corroboration_count.get(f["_id"], 0))
        score = base + 0.07 * corr
        confidence_map[f["_id"]] = round(_clamp01(score), 4)

    return confidence_map


def _gap_analysis(deduped: list[dict], dimensions: list[str]) -> tuple[list[dict], float]:
    """Identify dimensions with insufficient coverage; compute coverage score."""
    if not dimensions:
        return ([], 1.0 if deduped else 0.0)

    counts: dict[str, int] = {d.lower(): 0 for d in dimensions}
    for f in deduped:
        dim = (f.get("_dimension") or "").strip().lower()
        if dim in counts:
            counts[dim] += 1

    gaps: list[dict] = []
    for dim in dimensions:
        key = dim.lower()
        c = counts.get(key, 0)
        if c == 0:
            gaps.append({"dimension": dim, "coverage": c, "severity": "high"})
        elif c == 1 and len(deduped) > 3:
            # Only flag single-coverage as medium when there are enough
            # findings that more could reasonably be expected.
            gaps.append({"dimension": dim, "coverage": c, "severity": "medium"})

    covered = sum(1 for d in dimensions if counts.get(d.lower(), 0) > 0)
    coverage_score = covered / max(1, len(dimensions))
    return (gaps, round(coverage_score, 4))


def _infer_source_quality(finding: dict) -> float:
    """Infer source quality in 0..1 from common fields (best-effort)."""
    # Explicit numeric hint.
    direct = finding.get("source_score")
    if direct is not None:
        return _as_float(direct, default=0.6)

    source = finding.get("source")
    if isinstance(source, dict):
        if "score" in source:
            return _as_float(source.get("score"), default=0.6)

    url = _get_url(finding).lower()
    if not url:
        return 0.55

    # Very coarse domain heuristics.
    if ".gov/" in url or url.endswith(".gov") or ".gov." in url:
        return 0.9
    if ".edu/" in url or url.endswith(".edu") or ".edu." in url:
        return 0.85
    if "arxiv.org" in url:
        return 0.75
    if "wikipedia.org" in url:
        return 0.65
    if "medium.com" in url or "substack.com" in url:
        return 0.55
    if "reddit.com" in url or "twitter.com" in url or "x.com" in url:
        return 0.45
    return 0.6


def _as_float(value: object, default: float) -> float:
    try:
        if value is None:
            return default
        return float(value)
    except Exception:
        return default


def _clamp01(x: float) -> float:
    if math.isnan(x) or math.isinf(x):
        return 0.0
    return max(0.0, min(1.0, x))



path: scripts/chunk_selector.py
content: """Chunk Selector — LLM scores each chunk for relevance, filters by threshold.

This is the "hallucination killer": instead of feeding raw search results to the
synthesis LLM, we first have an LLM review each chunk and keep only relevant ones.

Usage:
    python chunk_selector.py --question "..." --content <json_file_or_stdin> --min-score 0.7

Input content format (JSON):
    [
        {"content": "...", "url": "...", "title": "..."},
        ...
    ]

Output: JSON array of scored chunks above threshold.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.request
from typing import Optional

from llm_client import call_llm as _call_glm

# Max tokens per batch — avoid context overflow
MAX_BATCH_TOKENS = 8000


def _estimate_tokens(text: str) -> int:
    """Rough token estimate (1 token ≈ 4 chars)."""
    return len(text) // 4


def select_relevant_chunks(
    question: str,
    content: list[dict],
    min_score: float = 0.7,
    max_context_tokens: int = 8000,
    [REDACTED_SECRET] = None,
) -> list[dict]:
    """Score each chunk for relevance, return filtered list.

    Batches chunks to avoid context overflow. Each chunk gets scored 0.0-1.0.
    Only chunks >= min_score are returned, sorted by score descending.
    """

    if not content:
        return []

    # Batch chunks to fit in context
    batches = _batch_chunks(content, max_context_tokens)
    all_scored = []

    for batch in batches:
        scored = _score_batch(question, batch, [REDACTED_SECRET])
        all_scored.extend(scored)

    # Filter and sort
    filtered = [c for c in all_scored if c.get("score", 0) >= min_score]
    filtered.sort(key=lambda x: x.get("score", 0), reverse=True)

    return filtered


def _batch_chunks(chunks: list[dict], max_tokens: int) -> list[list[dict]]:
    """Split chunks into batches that fit within token limit."""
    batches = []
    current_batch = []
    current_tokens = 0

    for chunk in chunks:
        text = chunk.get("content", "")
        tokens = _estimate_tokens(text)
        if tokens > max_tokens:
            # Truncate oversized chunks
            chunk = {**chunk, "content": text[: max_tokens * 4]}
            tokens = max_tokens

        if current_tokens + tokens > max_tokens and current_batch:
            batches.append(current_batch)
            current_batch = []
            current_tokens = 0

        current_batch.append(chunk)
        current_tokens += tokens

    if current_batch:
        batches.append(current_batch)

    return batches


def _score_batch(question: str, batch: list[dict], [REDACTED_SECRET] = None) -> list[dict]:
    """Score a batch of chunks using LLM."""
    system = (
        "You are a research relevance scorer. Score each chunk's relevance "
        "to the research question on a scale of 0.0 to 1.0.\n\n"
        "Scoring criteria:\n"
        "- 1.0: Directly answers or is essential context for the question\n"
        "- 0.8: Highly relevant supporting information\n"
        "- 0.6: Somewhat relevant but tangential\n"
        "- 0.4: Loosely related, minor utility\n"
        "- 0.2: Barely related\n"
        "- 0.0: Completely irrelevant\n\n"
        'Output ONLY a JSON array. Each element: {"index": N, "score": 0.X, '
        '"reason": "one sentence"}. Use the same index numbers as input.'
    )

    # Build numbered list
    numbered = []
    for i, chunk in enumerate(batch):
        numbered.append(
            f"[{i}] {chunk.get('title', 'Untitled')} ({chunk.get('url', 'unknown')})\n"
            f"{chunk.get('content', '')[:2000]}"  # Truncate long chunks for prompt
        )

    prompt = (
        f"Research question: {question}\n\n"
        f"Chunks to score:\n{chr(10).join(numbered)}\n\n"
        "Score each chunk. Output JSON array only."
    )

    try:
        response = _call_glm(prompt, system, max_tokens=4096, temperature=0.1, [REDACTED_SECRET])
        response = response.strip()
        if response.startswith("```"):
            response = response.split("\n", 1)[1].rsplit("```", 1)[0].strip()

        scores = json.loads(response)
        if not isinstance(scores, list):
            scores = [scores]

        # Merge scores back into chunks
        result = []
        for score_entry in scores:
            idx = score_entry.get("index", 0)
            if 0 <= idx < len(batch):
                merged = {**batch[idx]}
                merged["score"] = float(score_entry.get("score", 0))
                merged["relevance_reason"] = score_entry.get("reason", "")
                result.append(merged)

        # Add unscored chunks with score 0
        scored_indices = {s.get("index", 0) for s in scores}
        for i, chunk in enumerate(batch):
            if i not in scored_indices:
                merged = {**chunk, "score": 0.0, "relevance_reason": "Not scored"}
                result.append(merged)

        return result

    except Exception as e:
        # On error, return all chunks with neutral score
        print(
            f"Warning: scoring failed ({e}), returning all chunks unscored",
            file=sys.stderr,
        )
        return [
            {**c, "score": 0.5, "relevance_reason": "Scoring failed"} for c in batch
        ]


def main():
    parser = argparse.ArgumentParser(description="Score and filter chunks by relevance")
    parser.add_argument("--question", "-q", required=True, help="Research question")
    parser.add_argument("--content", "-c", help="JSON file with chunks (or stdin)")
    parser.add_argument("--min-score", "-m", type=float, default=0.7)
    parser.add_argument("--max-tokens", type=int, default=MAX_BATCH_TOKENS)
    parser.add_argument("--stdin", action="store_true", help="Read content from stdin")
    args = parser.parse_args()

    # Load content
    if args.stdin or not args.content:
        content = json.load(sys.stdin)
    else:
        with open(args.content) as f:
            content = json.load(f)

    if not isinstance(content, list):
        content = [content]

    selected = select_relevant_chunks(
        args.question, content, args.min_score, args.max_tokens
    )

    print(json.dumps(selected, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


path: scripts/context_expander.py
content: """Context Expander — Decides if selected chunks need surrounding context, fetches it.

After LLM chunk selection, some chunks may be "context cliffs" — the most relevant
info is in the middle of a longer document and adjacent sections add critical context.

Usage:
    python context_expander.py --selected <json_file_or_stdin> --question "..." --max-expansions 5

Input: JSON array of selected chunks (output from chunk_selector.py)
Output: JSON array with expanded content where needed
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.request
from typing import Optional

from llm_client import call_llm as _call_glm


def _fetch_url(url: str, max_chars: int = 30000) -> Optional[str]:
    """Fetch a URL and return text content."""
    try:
        req = urllib.request.Request(
            url,
            headers={"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)"},
        )
        with urllib.request.urlopen(req, timeout=15) as resp:
            content_type = resp.headers.get("Content-Type", "")
            if "text" not in content_type and "json" not in content_type:
                return None
            data = resp.read(max_chars).decode(errors="replace")
            return data
    except Exception:
        return None


def expand_context(
    selected_chunks: list[dict],
    question: str,
    max_expansions: int = 5,
    [REDACTED_SECRET] = None,
) -> list[dict]:
    """For each chunk, decide if surrounding context would help. Fetch if so."""

    if not selected_chunks:
        return []

    # First pass: LLM decides which chunks need expansion
    expansion_decisions = _decide_expansions(selected_chunks, question, [REDACTED_SECRET])

    # Limit expansions
    to_expand = [d for d in expansion_decisions if d.get("needs_expansion", False)][
        :max_expansions
    ]

    expanded_chunks = []
    expand_urls = {d["index"] for d in to_expand}

    for i, chunk in enumerate(selected_chunks):
        if i in expand_urls:
            url = chunk.get("url", "")
            if url and url.startswith("http"):
                full_content = _fetch_url(url)
                if full_content:
                    expanded = {**chunk}
                    expanded["original_content"] = chunk.get("content", "")
                    expanded["content"] = full_content
                    expanded["expanded"] = True
                    expanded_chunks.append(expanded)
                    continue

        # No expansion needed or failed
        expanded_chunks.append({**chunk, "expanded": False})

    return expanded_chunks


def _decide_expansions(chunks: list[dict], question: str, [REDACTED_SECRET] = None) -> list[dict]:
    """Ask LLM which chunks need surrounding context."""
    system = (
        "You are a research context analyzer. For each chunk, decide if the "
        "research question would benefit from seeing more of the source document.\n\n"
        "A chunk needs expansion if:\n"
        "- It references something not explained ('as mentioned above', 'this approach')\n"
        "- It's clearly a fragment of a longer explanation\n"
        "- It mentions data/statistics without full context\n"
        "- It's a conclusion without the reasoning leading to it\n\n"
        'Output ONLY JSON array: [{"index": N, "needs_expansion": bool, '
        '"reason": "..."}]'
    )

    numbered = []
    for i, chunk in enumerate(chunks):
        numbered.append(
            f"[{i}] (score: {chunk.get('score', '?')})\n{chunk.get('content', '')[:1500]}"
        )

    prompt = (
        f"Question: {question}\n\n"
        f"Chunks:\n{chr(10).join(numbered)}\n\n"
        "Which chunks need more context from their source documents?"
    )

    try:
        response = _call_glm(prompt, system, max_tokens=2048, temperature=0.1, [REDACTED_SECRET])
        response = response.strip()
        if response.startswith("```"):
            response = response.split("\n", 1)[1].rsplit("```", 1)[0].strip()

        decisions = json.loads(response)
        if not isinstance(decisions, list):
            decisions = [decisions]
        return decisions
    except Exception:
        # On error, don't expand anything
        return [
            {"index": i, "needs_expansion": False, "reason": "LLM error"}
            for i in range(len(chunks))
        ]


def main():
    parser = argparse.ArgumentParser(
        description="Expand chunk context from source documents"
    )
    parser.add_argument("--selected", "-s", help="JSON file with selected chunks")
    parser.add_argument("--question", "-q", required=True, help="Research question")
    parser.add_argument("--max-expansions", type=int, default=5)
    parser.add_argument("--stdin", action="store_true", help="Read from stdin")
    args = parser.parse_args()

    if args.stdin or not args.selected:
        chunks = json.load(sys.stdin)
    else:
        with open(args.selected) as f:
            chunks = json.load(f)

    if not isinstance(chunks, list):
        chunks = [chunks]

    expanded = expand_context(chunks, args.question, args.max_expansions)
    print(json.dumps(expanded, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


path: scripts/fact-checker.py
content: #!/usr/bin/env python3
"""Fact Checker — verifies claims and ranks sources."""

import argparse
import json
import re


def extract_claims(text: str) -> list[dict]:
    sentences = re.split(r"(?<=[.!?])\s+", text)
    claims = []
    keywords = [
        "is",
        "are",
        "was",
        "were",
        "caused",
        "leads to",
        "proven",
        "shows",
        "found",
        "reported",
    ]
    for i, s in enumerate(sentences):
        s = s.strip()
        if len(s) > 30 and any(k in s.lower() for k in keywords):
            claims.append(
                {"claim": s, "index": i, "status": "unverified", "confidence": 0}
            )
    return claims


def rank_sources(sources: list[dict]) -> list[dict]:
    reliable_domains = [
        "edu",
        "gov",
        "nature.com",
        "science.org",
        "arxiv.org",
        "ieee.org",
        "acm.org",
    ]
    for s in sources:
        domain = s.get("url", "").split("//")[-1].split("/")[0]
        s["reliability"] = (
            "high"
            if any(d in domain for d in reliable_domains)
            else "medium"
            if "wikipedia" in domain
            else "low"
        )
    return sorted(
        sources,
        key=lambda x: {"high": 0, "medium": 1, "low": 2}[x.get("reliability", "low")],
    )


def main():
    p = argparse.ArgumentParser(description="Verify claims and rank sources")
    p.add_argument("text", help="Text containing claims to verify")
    p.add_argument("-f", "--file", help="Read text from file")
    p.add_argument("--sources", help="JSON array of source URLs")
    args = p.parse_args()

    text = open(args.file).read() if args.file else args.text
    claims = extract_claims(text)
    print(f"📝 Extracted {len(claims)} verifiable claims:\n")
    for c in claims:
        print(f"  [{c['index']}] {c['claim'][:100]}...")
        print(f"      Status: {c['status']} | Confidence: {c['confidence']}%")

    if args.sources:
        sources = [{"url": u} for u in json.loads(args.sources)]
        ranked = rank_sources(sources)
        print(f"\n📚 Source ranking ({len(ranked)} sources):")
        for s in ranked:
            print(f"  [{s['reliability'].upper()}] {s['url']}")


if __name__ == "__main__":
    main()


path: scripts/llm_client.py
content: """Shared LLM client — provider-agnostic OpenAI-compatible API caller.

Supports any OpenAI-compatible endpoint (Z.AI GLM, OpenAI, Anthropic via proxy,
Ollama, LM Studio, vLLM, etc.) via environment variables.

Priority:
  1. OPENAI_API_KEY + OPENAI_API_BASE (or OPENAI_BASE_URL) — standard OpenAI-compat
  2. ZAI_API_KEY + ZAI_API_ENDPOINT — Z.AI GLM (legacy, still supported)
  3. ANTHROPIC_API_KEY — Anthropic (requires OpenAI-compat proxy)

Environment variables:
  LLM_MODEL          Model name (default: auto-detected from provider)
  LLM_API_KEY        Universal API key (highest priority)
  LLM_API_BASE       Universal base URL (highest priority)
  OPENAI_API_KEY     OpenAI / compatible key
  OPENAI_API_BASE    OpenAI / compatible endpoint
  OPENAI_BASE_URL    Alternative to OPENAI_API_BASE
  ZAI_API_KEY        Z.AI GLM key (legacy)
  ZAI_API_ENDPOINT   Z.AI endpoint (legacy)
  GLM_MODEL          Z.AI model override (legacy)
"""

from __future__ import annotations

import json
import os
import urllib.request
from typing import Optional


def _resolve_config() -> tuple[str, str, str]:
    """Resolve API key, base URL, and model from environment.

    Returns:
        (api_key, base_url, model)

    Raises:
        RuntimeError: if no API key is configured.
    """
    # Universal overrides (highest priority)
    [REDACTED_SECRET]"LLM_API_KEY", "")
    base_url = os.environ.get("LLM_API_BASE", "")
    model = os.environ.get("LLM_MODEL", "")

    if api_key and base_url:
        return api_key, base_url.rstrip("/"), model or "gpt-4o-mini"

    # OpenAI-compatible (standard)
    openai_key = os.environ.get("OPENAI_API_KEY", "")
    openai_base = os.environ.get("OPENAI_API_BASE", "") or os.environ.get("OPENAI_BASE_URL", "")
    if openai_key:
        base = openai_base.rstrip("/") if openai_base else "https://api.openai.com/v1"
        return openai_key, base, model or "gpt-4o-mini"

    # Z.AI GLM (legacy)
    zai_key = os.environ.get("ZAI_API_KEY", "")
    zai_endpoint = os.environ.get("ZAI_API_ENDPOINT", "https://api.z.ai/api/coding/paas/v4")
    glm_model = os.environ.get("GLM_MODEL", "glm-5-turbo")
    if zai_key:
        return zai_key, zai_endpoint.rstrip("/"), model or glm_model

    raise RuntimeError(
        "No LLM API key configured. Set one of: "
        "LLM_API_KEY, OPENAI_API_KEY, or ZAI_API_KEY"
    )


def call_llm(
    prompt: str,
    system: str = "",
    max_tokens: int = 4096,
    temperature: float = 0.3,
    model: Optional[str] = None,
    [REDACTED_SECRET] = None,
    base_url: Optional[str] = None,
) -> str:
    """Call an OpenAI-compatible chat completions API.

    Args:
        prompt: User message.
        system: Optional system message.
        max_tokens: Max tokens in response.
        temperature: Sampling temperature.
        model: Model override (else auto-detected from env).
        api_key: API key override (else auto-detected from env).
        base_url: Base URL override (else auto-detected from env).

    Returns:
        Response text content.

    Raises:
        RuntimeError: on API errors or missing config.
    """
    resolved_key, resolved_base, resolved_model = _resolve_config()
    key = api_key or resolved_key
    base = (base_url or resolved_base).rstrip("/")
    mdl = model or resolved_model

    messages = []
    if system:
        messages.append({"role": "system", "content": system})
    messages.append({"role": "user", "content": prompt})

    payload = {
        "model": mdl,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": temperature,
    }

    data = json.dumps(payload).encode()
    endpoint = f"{base}/chat/completions"
    req = urllib.request.Request(
        endpoint,
        data=data,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {key}",
        },
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            result = json.loads(resp.read())
            return result["choices"][0]["message"]["content"]
    except urllib.error.HTTPError as e:
        body = e.read().decode(errors="replace")
        raise RuntimeError(f"LLM API {e.code} ({endpoint}): {body}")


# Backward compatibility alias
_call_glm = call_llm


path: scripts/query_generator.py
content: """Query Generator — Generates multiple search query variants for a research question.

Usage:
    python query_generator.py --question "How does RAG work?" --max-queries 5
    echo '{"question": "..."}' | python query_generator.py --stdin

Output: JSON array of query objects with type, query, and rationale.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.request
from typing import Optional

from llm_client import call_llm as _call_glm


def generate_queries(
    question: str,
    max_queries: int = 5,
    [REDACTED_SECRET] = None,
) -> list[dict]:
    """Generate multiple search query variants using LLM + rule-based expansion.

    Returns list of dicts: [{"type": str, "query": str, "rationale": str}]
    """

    system = (
        "You are a research query generator. Given a research question, "
        "generate diverse search query variants. Each should cover a different "
        "angle or aspect. Output ONLY valid JSON array, no markdown."
    )

    prompt = f"""Generate {max_queries} search query variants for this research question:

"{question}"

Rules:
- Mix broad and specific queries
- Include technical and layperson phrasings
- Include at least one query targeting recent developments (add year if helpful)
- Include at least one query targeting comparisons or alternatives
- No duplicate intents

Output JSON array: [{{"type": "semantic|keyword|broad|specific|comparative", "query": "...", "rationale": "..."}}]"""

    try:
        response = _call_glm(prompt, system, max_tokens=2048, temperature=0.7, [REDACTED_SECRET])
        # Strip markdown fences if present
        response = response.strip()
        if response.startswith("```"):
            response = response.split("\n", 1)[1].rsplit("```", 1)[0].strip()

        queries = json.loads(response)
        if not isinstance(queries, list):
            queries = [queries]
        return queries[:max_queries]
    except Exception:
        # Fallback: rule-based query generation
        return _rule_based_queries(question, max_queries)


def _rule_based_queries(question: str, max_queries: int) -> list[dict]:
    """Fallback: generate query variants without LLM."""
    queries = []

    # Original
    queries.append(
        {
            "type": "semantic",
            "query": question,
            "rationale": "Original question",
        }
    )

    # Add "how" variant
    if not question.lower().startswith("how"):
        queries.append(
            {
                "type": "specific",
                "query": f"how does {question.lower().rstrip('?')}",
                "rationale": "Procedural angle",
            }
        )

    # Add comparison variant
    queries.append(
        {
            "type": "comparative",
            "query": f"{question} alternatives compared 2026",
            "rationale": "Comparison angle with recency",
        }
    )

    # Add broad variant — remove modifiers
    words = question.split()
    if len(words) > 4:
        core = " ".join(words[:4])
        queries.append(
            {
                "type": "broad",
                "query": core,
                "rationale": "Broader search, fewer modifiers",
            }
        )

    # Add "best practices" variant
    queries.append(
        {
            "type": "keyword",
            "query": f"best practices {question.lower().rstrip('?')} 2026",
            "rationale": "Practical guidance angle",
        }
    )

    return queries[:max_queries]


def main():
    parser = argparse.ArgumentParser(description="Generate search query variants")
    parser.add_argument("--question", "-q", help="Research question")
    parser.add_argument("--max-queries", "-n", type=int, default=5)
    parser.add_argument(
        "--stdin", action="store_true", help="Read question from stdin JSON"
    )
    args = parser.parse_args()

    if args.stdin:
        data = json.load(sys.stdin)
        question = data.get("question", data.get("q", ""))
    elif args.question:
        question = args.question
    else:
        print("Error: provide --question or --stdin", file=sys.stderr)
        sys.exit(1)

    queries = generate_queries(question, args.max_queries)
    print(json.dumps(queries, indent=2))


if __name__ == "__main__":
    main()


path: scripts/reflection.py
content: """Reflection — Coverage/gap check between research cycles.

After every researcher → analyst cycle, the orchestrator runs reflection to decide
whether to continue researching or proceed to writing.

Usage:
    python reflection.py --question "..." --findings <json_file_or_stdin> --cycle 1 --original-plan <json_file>

Input findings: JSON array of analyst output
    [{"theme": "...", "claims": [...], "sources": [...], "confidence": 0.8}]

Output: JSON with coverage assessment, gaps, and continue decision.
"""

from __future__ import annotations

import argparse
import dataclasses
import json
import os
import sys
import time
import urllib.request
from typing import Any, Optional

from llm_client import call_llm as _call_glm

MAX_CYCLES = 8
DEFAULT_TIME_BUDGET_SECONDS = int(os.environ.get("DEEP_RESEARCH_TIME_BUDGET_SECONDS", "900"))
DEFAULT_TOKEN_BUDGET = int(os.environ.get("DEEP_RESEARCH_TOKEN_BUDGET", "60000"))


@dataclasses.dataclass(frozen=True)
class ResearchPlan:
    question: str
    dimensions: list[str]
    dimension_questions: dict[str, list[str]]
    created_at_unix: float = dataclasses.field(default_factory=lambda: time.time())
    budget_seconds: int = DEFAULT_TIME_BUDGET_SECONDS
    budget_tokens: int = DEFAULT_TOKEN_BUDGET


@dataclasses.dataclass(frozen=True)
class ReflectionResult:
    should_continue: bool
    gaps: list[str]
    coverage_score: float
    next_dimensions: list[str]
    summary: str


def _normalize_findings(findings_input) -> list[dict]:
    """Convert analyst output formats into the themed format reflection expects.

    Handles:
    - Flat list: [{"claim": "...", "dimension": "...", ...}]
    - Wrapped: {"merged_findings": [...]}
    - Already themed: [{"theme": "...", "claims": [...], "sources": [...]}]
    - Researcher format: [{"dimension": "...", "findings": [...]}]
    """
    # Unwrap if dict with merged_findings key
    if isinstance(findings_input, dict):
        if "merged_findings" in findings_input:
            findings_input = findings_input["merged_findings"]
        elif "findings" in findings_input and "dimension" in findings_input:
            # Single researcher output
            findings_input = [findings_input]
        else:
            findings_input = [findings_input]

    if not isinstance(findings_input, list):
        return []

    # Check if already in themed format
    if (
        findings_input
        and "theme" in findings_input[0]
        and "claims" in findings_input[0]
    ):
        return findings_input

    # Check if researcher format: [{dimension, findings: [...]}]
    if (
        findings_input
        and isinstance(findings_input[0], dict)
        and "findings" in findings_input[0]
    ):
        groups = {}
        for item in findings_input:
            dim = item.get("dimension", "Unknown")
            if dim not in groups:
                groups[dim] = {
                    "theme": dim,
                    "claims": [],
                    "sources": [],
                    "confidence": 0,
                }
            for f in item.get("findings", []):
                groups[dim]["claims"].append(f.get("claim", str(f)))
                src = f.get("source", f.get("source_url", ""))
                if src and src not in groups[dim]["sources"]:
                    groups[dim]["sources"].append(src)
                conf = f.get("confidence", 0)
                groups[dim]["confidence"] = max(groups[dim]["confidence"], conf)
        return list(groups.values())

    # Flat list of findings — group by dimension/theme
    groups = {}
    for f in findings_input:
        theme = f.get("dimension", f.get("theme", "General"))
        if theme not in groups:
            groups[theme] = {
                "theme": theme,
                "claims": [],
                "sources": [],
                "confidence": 0,
            }
        groups[theme]["claims"].append(f.get("claim", str(f)))
        src = f.get("source", f.get("source_url", f.get("sources", "")))
        if isinstance(src, list):
            for s in src:
                if s and s not in groups[theme]["sources"]:
                    groups[theme]["sources"].append(s)
        elif src and src not in groups[theme]["sources"]:
            groups[theme]["sources"].append(src)
        conf = f.get("confidence", 0)
        groups[theme]["confidence"] = max(groups[theme]["confidence"], conf)
    return list(groups.values())


def reflect_llm(
    question: str,
    findings: list[dict],
    cycle: int,
    original_plan: Optional[dict] = None,
    max_cycles: int = MAX_CYCLES,
    [REDACTED_SECRET] = None,
) -> dict:
    """Legacy LLM-based reflection (kept for backwards compatibility)."""

    system = (
        "You are a research reflection analyzer. After a research cycle, you assess "
        "what's been found, what's missing, and whether more research would help.\n\n"
        "Be rigorous and honest.\n\n"
        "Output ONLY valid JSON with this schema:\n"
        "{\n"
        '  "covered": [{"theme": "...", "confidence": 0.9, "source_count": N}],\n'
        '  "gaps": [{"question": "...", "why_important": "..."}],\n'
        '  "coverage_score": 0.0,\n'
        '  "should_continue": true,\n'
        '  "continue_reason": "...",\n'
        '  "next_cycle_plan": {"dimensions": ["..."], "specific_questions": ["..."]}\n'
        "}\n\n"
        f"Rules:\n- Hard stop at cycle {max_cycles}\n"
    )

    findings = _normalize_findings(findings)
    findings_text = []
    for f in findings:
        theme = f.get("theme", "Unknown")
        claims = f.get("claims", [])
        confidence = f.get("confidence", "?")
        sources = f.get("sources", [])
        findings_text.append(
            f"Theme: {theme} (confidence: {confidence}, sources: {len(sources)})\n"
            f"Claims: {json.dumps(claims[:5], ensure_ascii=False)[:500]}"
        )

    plan_text = ""
    if original_plan:
        dims = original_plan.get("dimensions", [])
        plan_text = f"\nOriginal plan dimensions: {json.dumps(dims)}"

    prompt = (
        f"Research question: {question}\n"
        f"Cycle: {cycle}/{max_cycles}{plan_text}\n\n"
        f"Findings this cycle:\n{chr(10).join(findings_text)}\n\n"
        "Run reflection. Output JSON only."
    )

    response = _call_glm(prompt, system, max_tokens=4096, temperature=0.3, [REDACTED_SECRET]).strip()
    if response.startswith("```"):
        response = response.split("\n", 1)[1].rsplit("```", 1)[0].strip()
    reflection = json.loads(response)
    if cycle >= max_cycles:
        reflection["should_continue"] = False
        reflection["continue_reason"] = f"Hard stop at max cycles ({max_cycles})"
    reflection["cycle"] = cycle
    return reflection


def _estimate_tokens(obj: Any) -> int:
    """Cheap token proxy: ~4 chars/token."""
    try:
        s = json.dumps(obj, ensure_ascii=False)
    except Exception:
        s = str(obj)
    return max(1, len(s) // 4)


def _dimension_hits(plan_dimensions: list[str], findings: list[dict]) -> dict[str, int]:
    dims_norm = {d.strip().lower(): d for d in plan_dimensions if d and d.strip()}
    hits: dict[str, int] = {d: 0 for d in dims_norm.values()}

    for f in findings:
        dim = (f.get("dimension") or f.get("theme") or "").strip()
        if not dim:
            continue
        dim_key = dim.lower()
        if dim_key in dims_norm:
            hits[dims_norm[dim_key]] += 1
            continue
        for k_norm, original in dims_norm.items():
            if k_norm and (k_norm in dim_key or dim_key in k_norm):
                hits[original] += 1
                break

    return hits


def _gap_score(hit_count: int, avg_confidence: float, target_hits: int = 2) -> float:
    """0.0 = no gap, 1.0 = major gap."""
    coverage_component = 1.0 - min(1.0, hit_count / max(1, target_hits))
    confidence_component = 1.0 - max(0.0, min(1.0, avg_confidence))
    return max(0.0, min(1.0, coverage_component * 0.75 + confidence_component * 0.25))


def _progress_summary(plan: ResearchPlan, findings: list[dict], cycle: int, max_cycles: int) -> str:
    hits = _dimension_hits(plan.dimensions, findings)
    covered = [d for d, c in hits.items() if c > 0]
    missing = [d for d, c in hits.items() if c == 0]

    top_claims: list[str] = []
    for f in findings[:10]:
        claim = (f.get("claim") or "").strip()
        if claim:
            top_claims.append(claim[:160])

    lines = [
        f"Cycle {cycle}/{max_cycles}",
        f"Covered dimensions: {', '.join(covered) if covered else 'none'}",
        f"Missing dimensions: {', '.join(missing) if missing else 'none'}",
    ]
    if top_claims:
        lines.append("Top findings:")
        lines.extend([f"- {c}" for c in top_claims[:5]])
    return "\n".join(lines).strip()


def reflect(
    plan: ResearchPlan,
    findings: list[dict],
    cycle: int,
    max_cycles: int,
) -> ReflectionResult:
    """Coverage check + gap analysis + continue decision (max 8 cycles + budgets)."""
    normalized_findings = findings if isinstance(findings, list) else []
    max_cycles = min(int(max_cycles), MAX_CYCLES)
    cycle = int(cycle)

    plan_dimensions = [d for d in (plan.dimensions or []) if isinstance(d, str) and d.strip()]
    if not plan_dimensions:
        plan_dimensions = ["overview"]

    hits = _dimension_hits(plan_dimensions, normalized_findings)

    dim_conf_values: dict[str, list[float]] = {d: [] for d in plan_dimensions}
    for f in normalized_findings:
        dim_raw = (f.get("dimension") or f.get("theme") or "").strip().lower()
        conf = f.get("confidence", None)
        if conf is None:
            continue
        try:
            conf_f = float(conf)
        except Exception:
            continue
        for d in plan_dimensions:
            d_key = d.strip().lower()
            if not d_key:
                continue
            if dim_raw == d_key or d_key in dim_raw or dim_raw in d_key:
                dim_conf_values[d].append(conf_f)

    gap_scores: dict[str, float] = {}
    for d in plan_dimensions:
        avg_conf = (
            sum(dim_conf_values[d]) / len(dim_conf_values[d]) if dim_conf_values.get(d) else 0.0
        )
        gap_scores[d] = _gap_score(hits.get(d, 0), avg_conf)

    covered_count = sum(1 for d in plan_dimensions if hits.get(d, 0) > 0)
    coverage_score = covered_count / max(1, len(plan_dimensions))

    gap_ranked = sorted(gap_scores.items(), key=lambda kv: kv[1], reverse=True)
    gaps = [d for d, score in gap_ranked if score >= 0.45]
    next_dimensions = [d for d, score in gap_ranked if score >= 0.35][:3]

    now = time.time()
    elapsed = max(0.0, now - float(plan.created_at_unix))
    tokens_est = _estimate_tokens(normalized_findings)
    time_budget_hit = elapsed >= float(plan.budget_seconds) * 0.9
    token_budget_hit = tokens_est >= int(plan.budget_tokens) * 0.9

    should_continue = True
    if cycle >= max_cycles:
        should_continue = False
    elif time_budget_hit or token_budget_hit:
        should_continue = False
    elif coverage_score >= 0.8 and not gaps:
        should_continue = False
    elif not normalized_findings and cycle >= 1:
        should_continue = False

    summary = _progress_summary(plan, normalized_findings, cycle, max_cycles)
    if time_budget_hit or token_budget_hit:
        summary += (
            "\n\nBudget warning:"
            f"\n- elapsed_seconds={round(elapsed, 1)} / {plan.budget_seconds}"
            f"\n- est_tokens={tokens_est} / {plan.budget_tokens}"
        )

    return ReflectionResult(
        should_continue=should_continue,
        gaps=gaps,
        coverage_score=round(float(coverage_score), 3),
        next_dimensions=next_dimensions,
        summary=summary,
    )


def save_reflection(reflection: dict, output_dir: str) -> str:
    """Save reflection to a markdown file."""
    os.makedirs(output_dir, exist_ok=True)
    cycle = reflection.get("cycle", 0)
    filepath = os.path.join(output_dir, f"reflection-cycle-{cycle}.md")

    md = f"# Reflection Cycle {cycle}\n\n"
    md += f"**Coverage Score:** {reflection.get('coverage_score', '?')}\n"
    md += (
        f"**Should Continue:** {'Yes' if reflection.get('should_continue') else 'No'}\n"
    )
    if reflection.get("continue_reason"):
        md += f"**Reason:** {reflection['continue_reason']}\n"

    if reflection.get("covered"):
        md += "\n## Covered\n"
        for c in reflection["covered"]:
            md += f"- {c.get('theme', '?')} (confidence: {c.get('confidence', '?')}, sources: {c.get('source_count', '?')})\n"

    if reflection.get("gaps"):
        md += "\n## Gaps\n"
        for g in reflection["gaps"]:
            md += f"- {g.get('question', '?')}: {g.get('why_important', '')}\n"

    if reflection.get("contradictions"):
        md += "\n## Contradictions\n"
        for c in reflection["contradictions"]:
            md += f"- **A:** {c.get('claim_a', '')} ({c.get('source_a', '')})\n"
            md += f"  **B:** {c.get('claim_b', '')} ({c.get('source_b', '')})\n"
            md += f"  Resolution: {c.get('resolution', '?')}\n"

    if reflection.get("new_directions"):
        md += "\n## New Directions\n"
        for d in reflection["new_directions"]:
            md += f"- {d.get('topic', '?')}: {d.get('why_relevant', '')}\n"

    next_plan = reflection.get("next_cycle_plan", {})
    if next_plan.get("dimensions"):
        md += "\n## Next Cycle Plan\n"
        md += f"Dimensions: {json.dumps(next_plan['dimensions'])}\n"
        if next_plan.get("specific_questions"):
            md += f"Questions: {json.dumps(next_plan['specific_questions'], ensure_ascii=False)}\n"

    with open(filepath, "w") as f:
        f.write(md)

    return filepath


def main():
    parser = argparse.ArgumentParser(description="Research cycle reflection")
    parser.add_argument("--question", "-q", required=True)
    parser.add_argument("--findings", "-f", help="JSON file with analyst findings")
    parser.add_argument("--cycle", "-c", type=int, required=True)
    parser.add_argument(
        "--original-plan", "-p", help="JSON file with original research plan"
    )
    parser.add_argument("--max-cycles", type=int, default=MAX_CYCLES)
    parser.add_argument(
        "--output-dir", "-o", help="Directory to save reflection markdown"
    )
    parser.add_argument("--stdin", action="store_true")
    args = parser.parse_args()

    if args.stdin or not args.findings:
        findings = json.load(sys.stdin)
    else:
        with open(args.findings) as f:
            findings = json.load(f)

    original_plan_data = None
    if args.original_plan:
        with open(args.original_plan) as f:
            original_plan_data = json.load(f)

    # Build a ResearchPlan from the question and optional plan file
    if original_plan_data:
        plan = ResearchPlan(
            question=args.question,
            dimensions=original_plan_data.get("dimensions", ["overview"]),
            dimension_questions=original_plan_data.get("dimension_questions", {}),
            budget_seconds=original_plan_data.get("budget_seconds", DEFAULT_TIME_BUDGET_SECONDS),
            budget_tokens=original_plan_data.get("budget_tokens", DEFAULT_TOKEN_BUDGET),
        )
    else:
        plan = ResearchPlan(question=args.question, dimensions=["overview"])

    result = reflect(plan, findings, args.cycle, args.max_cycles)

    reflection_out = {
        "should_continue": result.should_continue,
        "gaps": result.gaps,
        "coverage_score": result.coverage_score,
        "next_dimensions": result.next_dimensions,
        "summary": result.summary,
        "cycle": args.cycle,
    }

    if args.output_dir:
        path = save_reflection(reflection_out, args.output_dir)
        print(f"Saved reflection to {path}", file=sys.stderr)

    print(json.dumps(reflection_out, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


path: scripts/research_pipeline.py
content: """Research Pipeline CLI — Enhanced with progress, checkpoints, budget, and parallel research.

Enhanced version of research_pipeline.py with:
- Progress reporting to stdout
- Resume from checkpoint if interrupted
- Budget awareness (token count, time limit)
- Parallel dimension research using ThreadPoolExecutor
- Output to file with --output flag

Usage:
    # Full pipeline with defaults
    python3 research_pipeline.py "What is quantum computing?" --max-cycles 3 --output report.md

    # Mock mode (no API calls)
    python3 research_pipeline.py "test question" --mock --output report.md

    # Resume from checkpoint
    python3 research_pipeline.py "test question" --resume checkpoint.json --output report.md

    # With budget limits
    python3 research_pipeline.py "question" --max-cycles 3 --time-limit 300 --token-limit 40000

    # Subcommand mode (backward compatible)
    python3 research_pipeline.py researcher --question "..." --dimension "..."
    python3 research_pipeline.py analyst --input researcher_output.json
    python3 research_pipeline.py writer --input analyst_output.json --question "..."
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Optional

from llm_client import call_llm as _call_glm

# Reuse existing scripts
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from query_generator import generate_queries
from chunk_selector import select_relevant_chunks
from context_expander import expand_context
from reflection import ResearchPlan, ReflectionResult, reflect
from writer import WriterAgent, write_report, OutputFormat, save_report

def _parse_json_response(text: str) -> any:
    """Parse JSON from LLM, stripping markdown fences."""
    text = text.strip()
    if text.startswith("```"):
        text = text.split("\n", 1)[1].rsplit("```", 1)[0].strip()
    return json.loads(text)


def _estimate_tokens(obj: any) -> int:
    """Cheap token proxy: ~4 chars/token."""
    try:
        s = json.dumps(obj, ensure_ascii=False)
    except Exception:
        s = str(obj)
    return max(1, len(s) // 4)


def _progress(message: str) -> None:
    """Print progress message to stdout."""
    timestamp = time.strftime("%H:%M:%S")
    print(f"[{timestamp}] {message}", flush=True)


def _save_checkpoint(data: dict, path: str) -> None:
    """Save pipeline state to a checkpoint file."""
    os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True)
    data["checkpoint_saved_at"] = time.time()
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, ensure_ascii=False)


def _load_checkpoint(path: str) -> Optional[dict]:
    """Load pipeline state from a checkpoint file."""
    try:
        with open(path, encoding="utf-8") as f:
            return json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        return None


# ─── RESEARCHER ───────────────────────────────────────────────────────────────


def _web_fetch(url: str) -> Optional[str]:
    """Fetch a URL and return text content."""
    try:
        req = urllib.request.Request(
            url,
            headers={"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)"},
        )
        with urllib.request.urlopen(req, timeout=15) as resp:
            content_type = resp.headers.get("Content-Type", "")
            if "text" not in content_type and "json" not in content_type:
                return None
            return resp.read(30000).decode(errors="replace")
    except Exception:
        return None


def run_researcher(
    question: str,
    dimension: str,
    specific_questions: list[str],
    max_sources: int = 10,
    search_results: Optional[list[dict]] = None,
) -> dict:
    """Orchestrate: query_generator → search → fetch → chunk_selector → context_expander."""
    combined_q = f"{question} — {dimension}: {', '.join(specific_questions)}"
    queries = generate_queries(combined_q, max_queries=5)

    if search_results is None:
        return {
            "status": "queries_ready",
            "dimension": dimension,
            "queries": queries,
            "specific_questions": specific_questions,
            "message": "Run searches with these queries, then call _run_researcher_with_results()",
        }

    chunks = []
    for result in search_results[:max_sources]:
        url = result.get("url", "")
        content = result.get("content") or _web_fetch(url)
        if content:
            chunks.append(
                {
                    "url": url,
                    "title": result.get("title", ""),
                    "content": content,
                }
            )

    if not chunks:
        return {
            "status": "no_content",
            "dimension": dimension,
            "queries": queries,
            "findings": [],
            "sources": [],
        }

    selected = select_relevant_chunks(question, chunks, min_score=0.7)
    expanded = expand_context(selected, question, max_expansions=5)
    findings = _extract_findings(expanded, question, dimension, specific_questions)

    return {
        "status": "complete",
        "dimension": dimension,
        "queries": [q["query"] for q in queries],
        "findings": findings,
        "sources": [
            {"url": c["url"], "title": c.get("title", ""), "score": c.get("score", 0)}
            for c in expanded
        ],
        "metadata": {
            "chunks_scanned": len(chunks),
            "chunks_selected": len(selected),
            "chunks_expanded": sum(1 for c in expanded if c.get("expanded")),
        },
    }


def _extract_findings(
    chunks: list[dict], question: str, dimension: str, specific_questions: list[str]
) -> list[dict]:
    """Use LLM to extract structured findings from selected chunks."""
    system = (
        "You are a research analyst. Extract key findings from the provided source chunks. "
        "Each finding must be directly supported by the source text.\n\n"
        "Output ONLY a JSON array. Each finding:\n"
        '{"claim": "...", "source_url": "...", "confidence": 0.0-1.0, '
        '"answers_question": "which question this addresses"}\n\n'
        "Rules:\n"
        "- Only include claims directly supported by source text\n"
        "- Never fabricate or extrapolate\n"
        "- confidence 1.0 = directly stated, 0.7 = strongly implied, 0.5 = inferred"
    )

    chunks_text = "\n\n---\n\n".join(
        f"[{c.get('url', 'unknown')}]\n{c.get('content', '')[:3000]}" for c in chunks
    )

    prompt = (
        f"Research question: {question}\n"
        f"Dimension: {dimension}\n"
        f"Target questions: {json.dumps(specific_questions)}\n\n"
        f"Source chunks:\n{chunks_text}\n\n"
        "Extract findings."
    )

    try:
        response = _call_glm(prompt, system, max_tokens=4096)
        findings = _parse_json_response(response)
        if not isinstance(findings, list):
            findings = [findings]
        return findings
    except Exception as e:
        print(f"Warning: finding extraction failed ({e})", file=sys.stderr)
        return []


# ─── ANALYST ──────────────────────────────────────────────────────────────────


def run_analyst(researcher_outputs: list[dict]) -> dict:
    """Deduplicate findings, flag contradictions, group into themes, identify gaps."""
    all_findings = []
    for output in researcher_outputs:
        all_findings.extend(output.get("findings", []))

    if not all_findings:
        return {
            "status": "no_findings",
            "themes": [],
            "contradictions": [],
            "gaps": [],
            "merged_findings": [],
        }

    system = (
        "You are a research synthesis analyst. Analyze the collected findings.\n\n"
        "Tasks:\n"
        "1. Deduplicate overlapping findings\n"
        "2. Flag contradictions (explicit disagreements between sources)\n"
        "3. Group findings into thematic clusters\n"
        "4. Identify knowledge gaps (important questions left unanswered)\n\n"
        "Output ONLY JSON:\n"
        "{\n"
        '  "themes": [{"name": "...", "findings": [...], "confidence": 0.0-1.0, '
        '"key_claims": ["..."]}],\n'
        '  "contradictions": [{"claim_a": "...", "claim_b": "...", '
        '"source_a": "...", "source_b": "...", "resolution": "..."}],\n'
        '  "gaps": [{"question": "...", "importance": "high|medium|low", '
        '"suggested_search": "..."}],\n'
        '  "merged_findings": [{"claim": "...", "sources": ["url1", "url2"], '
        '"confidence": 0.0-1.0, "theme": "...", "dimension": "..."}]\n'
        "}"
    )

    BATCH_SIZE = 15

    if len(all_findings) <= BATCH_SIZE:
        findings_text = json.dumps(all_findings, indent=2, ensure_ascii=False)
        prompt = f"Analyze these research findings:\n\n{findings_text}"
        try:
            response = _call_glm(prompt, system, max_tokens=8000)
            result = _parse_json_response(response)
            return {
                "status": "complete",
                **result,
                "input_dimensions": [
                    o.get("dimension", "unknown") for o in researcher_outputs
                ],
                "total_findings_input": len(all_findings),
            }
        except Exception as e:
            print(f"Warning: analysis failed ({e})", file=sys.stderr)
            return _analyst_fallback(all_findings)

    batch_results = []
    for i in range(0, len(all_findings), BATCH_SIZE):
        batch = all_findings[i : i + BATCH_SIZE]
        findings_text = json.dumps(batch, indent=2, ensure_ascii=False)
        prompt = f"Analyze these research findings (batch {i // BATCH_SIZE + 1}):\n\n{findings_text}"
        try:
            response = _call_glm(prompt, system, max_tokens=8000)
            batch_result = _parse_json_response(response)
            batch_results.append(batch_result)
        except Exception as e:
            print(f"Warning: batch {i // BATCH_SIZE + 1} failed ({e})", file=sys.stderr)
            batch_results.append(
                {
                    "merged_findings": batch,
                    "themes": [],
                    "contradictions": [],
                    "gaps": [],
                }
            )

    all_merged = []
    all_themes = []
    all_contradictions = []
    all_gaps = []
    for br in batch_results:
        all_merged.extend(br.get("merged_findings", []))
        all_themes.extend(br.get("themes", []))
        all_contradictions.extend(br.get("contradictions", []))
        all_gaps.extend(br.get("gaps", []))

    return {
        "status": "complete_batched",
        "themes": all_themes,
        "contradictions": all_contradictions,
        "gaps": all_gaps,
        "merged_findings": all_merged,
        "input_dimensions": [o.get("dimension", "unknown") for o in researcher_outputs],
        "total_findings_input": len(all_findings),
        "batches_processed": len(batch_results),
    }


def _analyst_fallback(all_findings: list[dict]) -> dict:
    """Simple dedup fallback when LLM analysis fails."""
    seen = set()
    merged = []
    for f in all_findings:
        key = f.get("claim", "")[:100].lower()
        if key not in seen:
            seen.add(key)
            merged.append(
                {**f, "sources": [f.get("source_url", "")], "theme": "unclassified"}
            )
    return {
        "status": "fallback",
        "themes": [],
        "contradictions": [],
        "gaps": [],
        "merged_findings": merged,
    }


# ─── Writer (using the new WriterAgent) ───────────────────────────────────────


def run_writer(analyst_output: dict, question: str) -> dict:
    """Generate polished markdown report using WriterAgent."""
    agent = WriterAgent(use_llm=True)
    result = agent.write_report(analyst_output, question, OutputFormat.REPORT)
    return result


# ─── Plan creation ────────────────────────────────────────────────────────────


def _create_plan(
    question: str,
    dimensions: Optional[list[str]] = None,
    budget_seconds: int = 900,
    budget_tokens: int = 60000,
) -> ResearchPlan:
    """Create a ResearchPlan from a question."""
    if not dimensions:
        try:
            queries = generate_queries(question, max_queries=5)
            dimensions = [q.get("query", f"dim_{i}") for i, q in enumerate(queries)]
        except Exception:
            dimensions = ["overview", "key findings", "evidence"]

    dim_questions = {d: [question] for d in dimensions}
    return ResearchPlan(
        question=question,
        dimensions=dimensions,
        dimension_questions=dim_questions,
        budget_seconds=budget_seconds,
        budget_tokens=budget_tokens,
    )


# ─── Parallel dimension research ──────────────────────────────────────────────


def _research_dimension_parallel(
    question: str,
    dimension: str,
    questions: list[str],
    search_results: Optional[dict[str, list[dict]]],
    mock_mode: bool,
) -> dict:
    """Research a single dimension (runs in a thread)."""
    if mock_mode:
        return {
            "status": "complete",
            "dimension": dimension,
            "queries": [f"mock query for {dimension}"],
            "findings": [
                {
                    "claim": f"Mock finding for {dimension}",
                    "source_url": "https://example.com/mock",
                    "confidence": 0.75,
                    "answers_question": question,
                    "dimension": dimension,
                }
            ],
            "sources": [{"url": "https://example.com/mock", "title": dimension, "score": 0.7}],
            "metadata": {"chunks_scanned": 1, "chunks_selected": 1, "chunks_expanded": 0},
        }

    sr = (search_results or {}).get(dimension)
    return run_researcher(question, dimension, questions, search_results=sr)


# ─── Enhanced looping pipeline ────────────────────────────────────────────────


def run_enhanced_pipeline(
    question: str,
    max_cycles: int = 3,
    dimensions: Optional[list[str]] = None,
    search_results: Optional[dict[str, list[dict]]] = None,
    mock_mode: bool = False,
    output_format: str = "report",
    time_limit: Optional[int] = None,
    token_limit: Optional[int] = None,
    checkpoint_path: Optional[str] = None,
    resume_from: Optional[str] = None,
    parallel_dimensions: bool = True,
) -> dict:
    """Run the full research pipeline with progress, checkpoints, and budget awareness.

    Pipeline: plan -> [research -> analyze -> reflect] x N -> write

    Args:
        question: The research question.
        max_cycles: Maximum research cycles (1-8).
        dimensions: Optional explicit dimensions.
        search_results: Dict mapping dimension -> search results.
        mock_mode: Use mock data instead of real API calls.
        output_format: Report format (report, summary, brief, json).
        time_limit: Max seconds for the entire pipeline (default: 900).
        token_limit: Max estimated tokens (default: 60000).
        checkpoint_path: Path to save checkpoints (default: no checkpointing).
        resume_from: Path to resume from a checkpoint file.
        parallel_dimensions: Research dimensions in parallel (default: True).

    Returns:
        Dict with final report, cycle history, and metadata.
    """
    max_cycles = min(max_cycles, 8)
    budget_seconds = time_limit or 900
    budget_tokens = token_limit or 60000
    start_time = time.time()

    # ── Resume from checkpoint ──
    if resume_from:
        checkpoint = _load_checkpoint(resume_from)
        if checkpoint:
            _progress(f"Resuming from checkpoint: {resume_from}")
            cycle_history = checkpoint.get("cycle_history", [])
            all_researcher_outputs = checkpoint.get("all_researcher_outputs", [])
            plan_data = checkpoint.get("plan", {})
            plan = ResearchPlan(
                question=plan_data.get("question", question),
                dimensions=plan_data.get("dimensions", dimensions or ["overview"]),
                dimension_questions=plan_data.get("dimension_questions", {}),
                budget_seconds=plan_data.get("budget_seconds", budget_seconds),
                budget_tokens=plan_data.get("budget_tokens", budget_tokens),
                created_at_unix=plan_data.get("created_at_unix", start_time),
            )
            start_cycle = len(cycle_history) + 1
            _progress(f"Resuming from cycle {start_cycle}, {len(cycle_history)} cycles completed")
        else:
            _progress(f"Checkpoint not found: {resume_from}, starting fresh")
            plan = _create_plan(question, dimensions, budget_seconds, budget_tokens)
            cycle_history = []
            all_researcher_outputs = []
            start_cycle = 1
    else:
        plan = _create_plan(question, dimensions, budget_seconds, budget_tokens)
        cycle_history = []
        all_researcher_outputs = []
        start_cycle = 1

    _progress(f"Research question: {question}")
    _progress(f"Dimensions: {plan.dimensions}")
    _progress(f"Budget: {budget_seconds}s / {budget_tokens} tokens")
    _progress(f"Max cycles: {max_cycles}")

    if checkpoint_path:
        _progress(f"Checkpoints: {checkpoint_path}")

    for cycle in range(start_cycle, max_cycles + 1):
        elapsed = time.time() - start_time
        if elapsed >= budget_seconds * 0.95:
            _progress(f"⚠️ Time budget reached ({elapsed:.0f}s / {budget_seconds}s), stopping")
            break

        cycle_data = {"cycle": cycle, "start_time": time.time()}
        _progress(f"━━━ Cycle {cycle}/{max_cycles} ━━━")

        # Determine target dimensions
        if cycle == 1:
            target_dims = plan.dimensions
        else:
            prev_reflection = cycle_history[-1].get("reflection", {})
            target_dims = prev_reflection.get("next_dimensions", [])
            if not target_dims:
                target_dims = plan.dimensions

        _progress(f"Researching dimensions: {target_dims}")

        # ── Research phase (parallel if enabled) ──
        cycle_researcher_outputs = []

        if parallel_dimensions and len(target_dims) > 1 and not mock_mode:
            _progress(f"Researching {len(target_dims)} dimensions in parallel...")
            with ThreadPoolExecutor(max_workers=min(3, len(target_dims))) as executor:
                futures = {}
                for dim in target_dims:
                    questions = plan.dimension_questions.get(dim, [question])
                    future = executor.submit(
                        _research_dimension_parallel,
                        question, dim, questions, search_results, mock_mode,
                    )
                    futures[future] = dim

                for future in as_completed(futures, timeout=120):
                    dim = futures[future]
                    try:
                        result = future.result(timeout=120)
                        cycle_researcher_outputs.append(result)
                        status = result.get("status", "?")
                        n_findings = len(result.get("findings", []))
                        _progress(f"  ✓ {dim}: {status} ({n_findings} findings)")
                    except Exception as e:
                        _progress(f"  ✗ {dim}: FAILED ({e})")
                        cycle_researcher_outputs.append({
                            "status": "error",
                            "dimension": dim,
                            "findings": [],
                            "error": str(e),
                        })
        else:
            for dim in target_dims:
                _progress(f"  Researching: {dim}...")
                try:
                    result = _research_dimension_parallel(
                        question, dim,
                        plan.dimension_questions.get(dim, [question]),
                        search_results, mock_mode,
                    )
                    cycle_researcher_outputs.append(result)
                    status = result.get("status", "?")
                    n_findings = len(result.get("findings", []))
                    _progress(f"  ✓ {dim}: {status} ({n_findings} findings)")
                except Exception as e:
                    _progress(f"  ✗ {dim}: FAILED ({e})")
                    cycle_researcher_outputs.append({
                        "status": "error",
                        "dimension": dim,
                        "findings": [],
                        "error": str(e),
                    })

        all_researcher_outputs.extend(cycle_researcher_outputs)
        cycle_data["researcher_outputs"] = cycle_researcher_outputs

        # ── Budget check after research ──
        all_tokens = _estimate_tokens(all_researcher_outputs)
        if all_tokens >= budget_tokens * 0.95:
            _progress(f"⚠️ Token budget reached ({all_tokens} / {budget_tokens}), stopping")
            break

        # ── Analyze phase ──
        _progress("  Analyzing findings...")
        if mock_mode:
            merged = []
            for ro in cycle_researcher_outputs:
                merged.extend(ro.get("findings", []))
            analyst_out = {
                "status": "complete",
                "themes": [{"name": d, "confidence": 0.75, "key_claims": []} for d in target_dims],
                "contradictions": [],
                "gaps": [],
                "merged_findings": merged,
            }
        else:
            analyst_out = run_analyst(cycle_researcher_outputs)

        n_themes = len(analyst_out.get("themes", []))
        n_merged = len(analyst_out.get("merged_findings", []))
        _progress(f"  Analysis: {n_themes} themes, {n_merged} merged findings")
        cycle_data["analyst_output"] = analyst_out

        # ── Reflect phase ──
        findings_for_reflection = analyst_out.get("merged_findings", [])
        reflection_result = reflect(plan, findings_for_reflection, cycle, max_cycles)
        cycle_data["reflection"] = {
            "should_continue": reflection_result.should_continue,
            "gaps": reflection_result.gaps,
            "coverage_score": reflection_result.coverage_score,
            "next_dimensions": reflection_result.next_dimensions,
            "summary": reflection_result.summary,
        }

        cycle_data["end_time"] = time.time()
        cycle_data["duration_seconds"] = round(cycle_data["end_time"] - cycle_data["start_time"], 2)
        cycle_history.append(cycle_data)

        _progress(
            f"  Cycle {cycle} complete: coverage={reflection_result.coverage_score:.0%} "
            f"findings={n_merged} duration={cycle_data['duration_seconds']}s"
        )

        if reflection_result.gaps:
            _progress(f"  Gaps: {', '.join(reflection_result.gaps[:3])}")

        # ── Save checkpoint ──
        if checkpoint_path:
            checkpoint_data = {
                "plan": {
                    "question": plan.question,
                    "dimensions": plan.dimensions,
                    "dimension_questions": plan.dimension_questions,
                    "budget_seconds": plan.budget_seconds,
                    "budget_tokens": plan.budget_tokens,
                    "created_at_unix": plan.created_at_unix,
                },
                "cycle_history": cycle_history,
                "all_researcher_outputs": all_researcher_outputs,
            }
            _save_checkpoint(checkpoint_data, checkpoint_path)
            _progress(f"  Checkpoint saved: {checkpoint_path}")

        if not reflection_result.should_continue:
            reason = reflection_result.summary.split("\n")[0] if reflection_result.summary else "coverage sufficient"
            _progress(f"  Stopping: {reason}")
            break

    # ── Write phase ──
    _progress("━━━ Writing Report ━━━")

    all_merged = []
    all_themes = []
    for ch in cycle_history:
        ao = ch.get("analyst_output", {})
        all_merged.extend(ao.get("merged_findings", []))
        all_themes.extend(ao.get("themes", []))

    final_analyst = {
        "themes": all_themes,
        "contradictions": [],
        "gaps": cycle_history[-1].get("reflection", {}).get("gaps", []) if cycle_history else [],
        "merged_findings": all_merged,
    }

    try:
        fmt = OutputFormat(output_format)
    except ValueError:
        fmt = OutputFormat.REPORT

    if mock_mode:
        report = f"# Mock Research Report: {question}\n\n## Executive Summary\n\n"
        report += f"Mock report generated in {len(cycle_history)} cycle(s).\n\n"
        for ch in cycle_history:
            refl = ch.get("reflection", {})
            report += f"### Cycle {ch['cycle']} (coverage: {refl.get('coverage_score', '?')})\n\n"
            report += f"{refl.get('summary', '')}\n\n"
        if refl.get("gaps"):
            report += "## Gaps Identified\n\n"
            for g in refl["gaps"]:
                report += f"- {g}\n"
        writer_out = {
            "status": "complete",
            "report": report.strip(),
            "question": question,
            "format": output_format,
            "metadata": {},
        }
    else:
        writer_agent = WriterAgent(use_llm=True)
        writer_out = writer_agent.write_report(final_analyst, question, fmt)

    total_duration = round(time.time() - start_time, 2)
    _progress(f"Report generated: {len(writer_out.get('report', ''))} chars in {total_duration}s")

    return {
        "status": "complete",
        "question": question,
        "cycles_completed": len(cycle_history),
        "final_coverage": cycle_history[-1]["reflection"]["coverage_score"] if cycle_history else 0.0,
        "report": writer_out.get("report", ""),
        "cycle_history": cycle_history,
        "metadata": {
            "total_duration_seconds": total_duration,
            "total_findings": len(all_merged),
            "max_cycles": max_cycles,
            "dimensions": plan.dimensions,
            "output_format": output_format,
        },
    }


# ─── Backward compatibility alias ──────────────────────────────────────────


def run_looping_pipeline(
    question: str,
    max_cycles: int = 3,
    dimensions: Optional[list[str]] = None,
    search_results: Optional[dict[str, list[dict]]] = None,
    mock_mode: bool = False,
) -> dict:
    """Backward-compatible wrapper for run_enhanced_pipeline."""
    return run_enhanced_pipeline(
        question=question,
        max_cycles=max_cycles,
        dimensions=dimensions,
        search_results=search_results,
        mock_mode=mock_mode,
    )


# ─── CLI ──────────────────────────────────────────────────────────────────────


def main():
    # Detect full pipeline mode (no subcommand, just a question)
    full_pipeline_args = [a for a in sys.argv[1:] if not a.startswith("-")]
    has_flags = any(a.startswith("-") for a in sys.argv[1:])

    is_full_pipeline = (
        full_pipeline_args
        and full_pipeline_args[0] not in ("researcher", "analyst", "writer", "-h", "--help")
        and "--help" not in sys.argv
        and "-h" not in sys.argv
    )

    if is_full_pipeline:
        parser = argparse.ArgumentParser(
            description="Deep Research Pipeline (full loop with progress, checkpoints, and budget)"
        )
        parser.add_argument("question", help="Research question")
        parser.add_argument("--max-cycles", type=int, default=3, help="Max research cycles (1-8)")
        parser.add_argument("--mock", action="store_true", help="Use mock data (no API calls)")
        parser.add_argument("--output", "-o", help="Output file path for the report")
        parser.add_argument("--format", "-f", choices=["report", "summary", "brief", "json"],
                            default="report", help="Output format (default: report)")
        parser.add_argument("--time-limit", type=int, default=900, help="Max seconds (default: 900)")
        parser.add_argument("--token-limit", type=int, default=60000, help="Max estimated tokens")
        parser.add_argument("--checkpoint", help="Save checkpoints to this path")
        parser.add_argument("--resume", help="Resume from checkpoint file")
        parser.add_argument("--dimensions", nargs="+", help="Explicit research dimensions")
        parser.add_argument("--no-parallel", action="store_true", help="Research dimensions sequentially")
        args = parser.parse_args()

        result = run_enhanced_pipeline(
            question=args.question,
            max_cycles=args.max_cycles,
            dimensions=args.dimensions,
            mock_mode=args.mock,
            output_format=args.format,
            time_limit=args.time_limit,
            token_limit=args.token_limit,
            checkpoint_path=args.checkpoint,
            resume_from=args.resume,
            parallel_dimensions=not args.no_parallel,
        )

        # Save to file if requested
        if args.output:
            try:
                save_report(result, args.output)
                _progress(f"Report saved to {args.output}")
            except Exception as e:
                _progress(f"Error saving report: {e}")
                print(json.dumps(result, indent=2, ensure_ascii=False))
        else:
            if args.format == "json":
                print(json.dumps(result, indent=2, ensure_ascii=False))
            else:
                print(result.get("report", ""))
        return

    # Subcommand mode (backward compatible)
    parser = argparse.ArgumentParser(description="Research pipeline orchestrator")
    sub = parser.add_subparsers(dest="command")

    rp = sub.add_parser("researcher")
    rp.add_argument("--question", "-q", required=True)
    rp.add_argument("--dimension", "-d", required=True)
    rp.add_argument("--questions", nargs="+", default=[])
    rp.add_argument("--max-sources", type=int, default=10)
    rp.add_argument("--search-results", help="JSON file with pre-fetched results")

    ap = sub.add_parser("analyst")
    ap.add_argument("--input", "-i", required=True, help="JSON file or stdin")

    wp = sub.add_parser("writer")
    wp.add_argument("--input", "-i", required=True, help="JSON file or stdin")
    wp.add_argument("--question", "-q", required=True)

    args = parser.parse_args()

    if args.command == "researcher":
        results = None
        if args.search_results:
            with open(args.search_results) as f:
                results = json.load(f)
        output = run_researcher(
            args.question, args.dimension, args.questions, args.max_sources, results
        )

    elif args.command == "analyst":
        if args.input == "-":
            data = json.load(sys.stdin)
        else:
            with open(args.input) as f:
                data = json.load(f)
        if isinstance(data, list):
            researcher_outputs = data
        else:
            researcher_outputs = [data]
        output = run_analyst(researcher_outputs)

    elif args.command == "writer":
        if args.input == "-":
            analyst_data = json.load(sys.stdin)
        else:
            with open(args.input) as f:
                analyst_data = json.load(f)
        output = run_writer(analyst_data, args.question)

    else:
        parser.print_help()
        sys.exit(1)

    print(json.dumps(output, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


path: scripts/research_sources.py
content: """Research Source Adapters — Multi-source search for the Researcher Agent.

Provides source adapters that search different information sources:
- WebSearchSource: Free web search via DuckDuckGo Lite + content fetching
- GitHubSource: GitHub public API (repos, code, issues)
- DocSource: Direct documentation URL fetching and extraction

Each adapter has a search(query, limit) → list[dict] method returning results
with {title, url, content, source_type, relevance_score}.

All adapters use urllib.request for HTTP. GitHub API works without auth for
public repos (60 req/hr unauthenticated). Web search uses DuckDuckGo Lite.

Usage:
    from research_sources import WebSearchSource, GitHubSource, DocSource

    web = WebSearchSource()
    results = web.search("OpenClaw skills system", limit=5)

    gh = GitHubSource()
    results = gh.search("openclaw", limit=5)

    doc = DocSource()
    results = doc.search("https://docs.python.org/3/", limit=3)
"""

from __future__ import annotations

import argparse
import hashlib
import html
import json
import os
import re
import sys
import urllib.parse
import urllib.request
import urllib.error
from typing import Optional
from xml.etree import ElementTree

from llm_client import call_llm as _call_glm

# ─── Shared helpers ───────────────────────────────────────────────────────────

def _fetch_url(url: str, max_chars: int = 30000, timeout: int = 15) -> Optional[str]:
    """Fetch a URL and return text content. Returns None on failure."""
    try:
        req = urllib.request.Request(
            url,
            headers={"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)"},
        )
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            content_type = resp.headers.get("Content-Type", "")
            if "text" not in content_type and "json" not in content_type:
                return None
            return resp.read(max_chars).decode(errors="replace")
    except Exception:
        return None


def _extract_text(html_content: str) -> str:
    """Rough HTML-to-text extraction: strip tags, decode entities, normalize whitespace."""
    # Remove script and style blocks
    text = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", html_content, flags=re.DOTALL | re.IGNORECASE)
    # Remove HTML tags
    text = re.sub(r"<[^>]+>", " ", text)
    # Decode HTML entities
    text = html.unescape(text)
    # Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text


def _content_hash(content: str) -> str:
    """SHA256 hash of content for deduplication."""
    return hashlib.sha256(content.encode()).hexdigest()[:16]


def _normalize_url(url: str) -> str:
    """Normalize URL for dedup (strip trailing slash, fragment, lowercase host)."""
    parsed = urllib.parse.urlparse(url)
    host = parsed.hostname.lower() if parsed.hostname else ""
    path = parsed.path.rstrip("/") or "/"
    return f"{parsed.scheme}://{host}{path}"


# ─── Base Source ──────────────────────────────────────────────────────────────


class BaseSource:
    """Base class for research source adapters."""

    source_type: str = "base"

    def search(self, query: str, limit: int = 5) -> list[dict]:
        """Search for results matching query.

        Returns list of dicts with:
            - title: str
            - url: str
            - content: str (main body text)
            - source_type: str
            - relevance_score: float (0.0-1.0)
        """
        raise NotImplementedError


# ─── WebSearchSource ──────────────────────────────────────────────────────────


class WebSearchSource(BaseSource):
    """Web search via DuckDuckGo Lite HTML endpoint (free, no API key).

    Falls back to generating search URLs if DuckDuckGo is blocked.
    Fetches full content from top result URLs.
    """

    source_type = "web"

    # DuckDuckGo Lite endpoint
    DDG_LITE = "https://lite.duckduckgo.com/lite/"
    # DuckDuckGo HTML endpoint (more reliable for scraping)
    DDG_HTML = "https://html.duckduckgo.com/html/"

    def __init__(self, fetch_content: bool = True):
        """Args:
            fetch_content: If True, fetch full content from result URLs.
        """
        self.fetch_content = fetch_content

    def search(self, query: str, limit: int = 5) -> list[dict]:
        """Search DuckDuckGo and return results with content."""
        # Try DuckDuckGo HTML endpoint first
        results = self._search_ddg_html(query, limit)
        if not results:
            # Fallback to lite endpoint
            results = self._search_ddg_lite(query, limit)

        if self.fetch_content:
            results = self._fetch_results_content(results)

        # Add metadata
        for r in results:
            r["source_type"] = self.source_type
            r["content_hash"] = _content_hash(r.get("content", ""))

        return results

    def _search_ddg_html(self, query: str, limit: int) -> list[dict]:
        """Search via DuckDuckGo HTML endpoint."""
        try:
            form_data = urllib.parse.urlencode({"q": query}).encode()
            req = urllib.request.Request(
                self.DDG_HTML,
                data=form_data,
                headers={
                    "User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)",
                    "Content-Type": "application/x-www-form-urlencoded",
                },
            )
            with urllib.request.urlopen(req, timeout=15) as resp:
                html_content = resp.read(100000).decode(errors="replace")

            return self._parse_ddg_html(html_content, limit)
        except Exception:
            return []

    def _parse_ddg_html(self, html_content: str, limit: int) -> list[dict]:
        """Parse DuckDuckGo HTML search results."""
        results = []

        # DuckDuckGo HTML wraps results in <a class="result__a"> with href
        # and snippets in <a class="result__snippet">
        # Links are redirect URLs: https://duckduckgo.com/l/?uddg=ENCODED_URL
        for match in re.finditer(
            r'<a[^>]+class="result__a"[^>]+href="([^"]+)"[^>]*>(.*?)</a>',
            html_content,
            re.DOTALL,
        ):
            raw_href = html.unescape(match.group(1))
            title = _extract_text(match.group(2))

            # Extract real URL from DDG redirect
            real_url = self._extract_ddg_url(raw_href)
            if not real_url:
                continue

            results.append({"title": title.strip(), "url": real_url, "content": ""})

            if len(results) >= limit:
                break

        # Also try to extract snippets
        snippet_matches = re.findall(
            r'<a[^>]+class="result__snippet"[^>]*>(.*?)</a>',
            html_content,
            re.DOTALL,
        )
        for i, snippet in enumerate(snippet_matches):
            if i < len(results):
                results[i]["snippet"] = _extract_text(snippet)

        return results

    def _extract_ddg_url(self, href: str) -> Optional[str]:
        """Extract real URL from DuckDuckGo redirect URL."""
        parsed = urllib.parse.urlparse(href)
        params = urllib.parse.parse_qs(parsed.query)

        # DDG uses uddg parameter for the real URL
        if "uddg" in params:
            return params["uddg"][0]

        # If it's already a direct URL
        if href.startswith("http"):
            return href

        return None

    def _search_ddg_lite(self, query: str, limit: int) -> list[dict]:
        """Search via DuckDuckGo Lite endpoint (simpler parsing)."""
        try:
            form_data = urllib.parse.urlencode({"q": query}).encode()
            req = urllib.request.Request(
                self.DDG_LITE,
                data=form_data,
                headers={
                    "User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)",
                    "Content-Type": "application/x-www-form-urlencoded",
                },
            )
            with urllib.request.urlopen(req, timeout=15) as resp:
                html_content = resp.read(100000).decode(errors="replace")

            return self._parse_ddg_lite(html_content, limit)
        except Exception:
            return []

    def _parse_ddg_lite(self, html_content: str, limit: int) -> list[dict]:
        """Parse DuckDuckGo Lite search results (table-based layout)."""
        results = []

        # DDG Lite uses tables with class="result-link"
        for match in re.finditer(
            r'<a[^>]+class="result-link"[^>]+href="([^"]+)"[^>]*>(.*?)</a>',
            html_content,
            re.DOTALL,
        ):
            raw_href = html.unescape(match.group(1))
            title = _extract_text(match.group(2))
            real_url = self._extract_ddg_url(raw_href)
            if not real_url:
                continue
            results.append({"title": title.strip(), "url": real_url, "content": ""})
            if len(results) >= limit:
                break

        # Extract snippets from table cells
        snippet_cells = re.findall(
            r'<td[^>]+class="result-snippet"[^>]*>(.*?)</td>',
            html_content,
            re.DOTALL,
        )
        for i, snippet in enumerate(snippet_cells):
            if i < len(results):
                results[i]["snippet"] = _extract_text(snippet)

        return results

    def _fetch_results_content(self, results: list[dict]) -> list[dict]:
        """Fetch full content from result URLs."""
        for r in results:
            url = r.get("url", "")
            if not url or url.startswith("https://duckduckgo.com"):
                continue
            content = _fetch_url(url)
            if content:
                r["content"] = _extract_text(content)[:8000]
            elif "snippet" in r:
                r["content"] = r["snippet"]
        return results


# ─── GitHubSource ─────────────────────────────────────────────────────────────


class GitHubSource(BaseSource):
    """GitHub search via public REST API.

    Searches repositories and code. No auth required for public repos
    (60 requests/hour limit). Set GITHUB_TOKEN env var for higher limits.

    Usage:
        gh = GitHubSource()
        repos = gh.search("openclaw", limit=5)
        code = gh.search_code("SKILL.md openclaw", limit=5)
    """

    source_type = "github"
    API_BASE = "https://api.github.com"

    def __init__(self, [REDACTED_SECRET] = None):
        self.token = token or os.environ.get("GITHUB_TOKEN", "")

    def search(self, query: str, limit: int = 5) -> list[dict]:
        """Search GitHub repositories and code."""
        repos = self._search_repos(query, limit)
        code = self._search_code(query, max(1, limit - len(repos)))
        return repos + code

    def search_repos(self, query: str, limit: int = 5) -> list[dict]:
        """Search only repositories."""
        return self._search_repos(query, limit)

    def search_code(self, query: str, limit: int = 5) -> list[dict]:
        """Search only code."""
        return self._search_code(query, limit)

    def _api_request(self, path: str, params: dict = None) -> Optional[dict]:
        """Make a GitHub API request."""
        url = f"{self.API_BASE}{path}"
        if params:
            url += "?" + urllib.parse.urlencode(params)

        headers = {
            "User-Agent": "ResearchBot/1.0",
            "Accept": "application/vnd.github.v3+json",
        }
        if self.[REDACTED_SECRET]"Authorization"] = f"token {self.token}"

        try:
            req = urllib.request.Request(url, headers=headers)
            with urllib.request.urlopen(req, timeout=15) as resp:
                return json.loads(resp.read())
        except urllib.error.HTTPError as e:
            if e.code == 403:
                print(f"GitHub API rate limit hit. Set GITHUB_TOKEN for higher limits.", file=sys.stderr)
            return None
        except Exception:
            return None

    def _search_repos(self, query: str, limit: int) -> list[dict]:
        """Search GitHub repositories."""
        data = self._api_request("/search/repositories", {
            "q": query,
            "sort": "stars",
            "order": "desc",
            "per_page": limit,
        })

        if not data or "items" not in data:
            return []

        results = []
        for item in data["items"][:limit]:
            results.append({
                "title": item.get("full_name", ""),
                "url": item.get("html_url", ""),
                "content": item.get("description", "") or "",
                "source_type": self.source_type,
                "relevance_score": 0.5,  # Default, can be re-scored later
                "metadata": {
                    "stars": item.get("stargazers_count", 0),
                    "language": item.get("language", ""),
                    "updated": item.get("updated_at", ""),
                    "topics": item.get("topics", []),
                    "forks": item.get("forks_count", 0),
                },
                "content_hash": _content_hash(item.get("description", "")),
            })
        return results

    def _search_code(self, query: str, limit: int) -> list[dict]:
        """Search GitHub code."""
        data = self._api_request("/search/code", {
            "q": query,
            "per_page": limit,
        })

        if not data or "items" not in data:
            return []

        results = []
        for item in data["items"][:limit]:
            # Fetch the file content if it's small
            raw_url = item.get("html_url", "").replace(
                "github.com", "raw.githubusercontent.com"
            ).replace("/blob/", "/")

            content = ""
            if raw_url:
                fetched = _fetch_url(raw_url, max_chars=5000)
                if fetched:
                    content = fetched[:5000]

            results.append({
                "title": f"{item.get('name', '')} in {item.get('repository', {}).get('full_name', '')}",
                "url": item.get("html_url", ""),
                "content": content,
                "source_type": self.source_type,
                "relevance_score": 0.5,
                "metadata": {
                    "repo": item.get("repository", {}).get("full_name", ""),
                    "path": item.get("path", ""),
                },
                "content_hash": _content_hash(content),
            })
        return results


# ─── DocSource ────────────────────────────────────────────────────────────────


class DocSource(BaseSource):
    """Documentation source — fetches and extracts content from documentation URLs.

    Can be used with specific URLs or auto-discover docs for known projects.

    Usage:
        doc = DocSource()
        # Direct URL fetch
        results = doc.search("https://docs.python.org/3/library/urllib.html", limit=1)

        # Auto-discover docs for a topic
        results = doc.discover_docs("python urllib", limit=3)
    """

    source_type = "docs"

    # Common documentation URL patterns
    DOC_PATTERNS = {
        "python": ["https://docs.python.org/3/"],
        "react": ["https://react.dev/reference"],
        "node": ["https://nodejs.org/docs/latest/api/"],
        "rust": ["https://doc.rust-lang.org/"],
        "go": ["https://pkg.go.dev/"],
        "docker": ["https://docs.docker.com/"],
        "kubernetes": ["https://kubernetes.io/docs/"],
        "fastapi": ["https://fastapi.tiangolo.com/"],
        "django": ["https://docs.djangoproject.com/"],
        "postgres": ["https://www.postgresql.org/docs/current/"],
    }

    def __init__(self, fetch_content: bool = True):
        self.fetch_content = fetch_content

    def search(self, query: str, limit: int = 5) -> list[dict]:
        """Fetch documentation from a URL or search for relevant docs.

        If query looks like a URL, fetch it directly.
        Otherwise, try to discover relevant documentation pages.
        """
        # If query is a URL, fetch directly
        if query.startswith("http://") or query.startswith("https://"):
            return self._fetch_doc_page(query)

        # Try to discover docs for known projects
        results = self._discover_from_known_patterns(query, limit)

        if not results:
            # Fallback: use DuckDuckGo with site:docs restriction
            results = self._search_docs_ddg(query, limit)

        return results

    def discover_docs(self, query: str, limit: int = 5) -> list[dict]:
        """Discover documentation pages for a topic."""
        return self.search(query, limit)

    def _fetch_doc_page(self, url: str) -> list[dict]:
        """Fetch a single documentation page."""
        content = _fetch_url(url, max_chars=40000)
        if not content:
            return []

        text = _extract_text(content)[:15000]
        # Try to extract a title from the page
        title_match = re.search(r"<title>(.*?)</title>", content, re.IGNORECASE)
        title = title_match.group(1).strip() if title_match else url

        # Try to extract h1 as a better title
        h1_match = re.search(r"<h1[^>]*>(.*?)</h1>", content, re.IGNORECASE | re.DOTALL)
        if h1_match:
            h1_text = _extract_text(h1_match.group(1)).strip()
            if len(h1_text) > 3 and len(h1_text) < 200:
                title = h1_text

        return [{
            "title": title,
            "url": url,
            "content": text,
            "source_type": self.source_type,
            "relevance_score": 0.7,  # Direct doc fetch gets a boost
            "content_hash": _content_hash(text),
        }]

    def _discover_from_known_patterns(self, query: str, limit: int) -> list[dict]:
        """Try to find documentation for known projects based on query keywords."""
        results = []
        query_lower = query.lower()

        for project, base_urls in self.DOC_PATTERNS.items():
            if project in query_lower:
                for base_url in base_urls:
                    content = _fetch_url(base_url, max_chars=20000)
                    if content:
                        text = _extract_text(content)[:10000]
                        results.append({
                            "title": f"{project.capitalize()} Documentation",
                            "url": base_url,
                            "content": text,
                            "source_type": self.source_type,
                            "relevance_score": 0.6,
                            "content_hash": _content_hash(text),
                        })
                    if len(results) >= limit:
                        return results

        return results

    def _search_docs_ddg(self, query: str, limit: int) -> list[dict]:
        """Search DuckDuckGo for documentation pages (site-restricted)."""
        # Build doc-focused search queries
        doc_queries = [
            f"{query} documentation",
            f"{query} docs guide",
            f"{query} official documentation",
        ]

        results = []
        seen_urls = set()

        for dq in doc_queries[:2]:  # Limit to 2 sub-queries
            try:
                form_data = urllib.parse.urlencode({"q": dq}).encode()
                req = urllib.request.Request(
                    WebSearchSource.DDG_HTML,
                    data=form_data,
                    headers={
                        "User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)",
                        "Content-Type": "application/x-www-form-urlencoded",
                    },
                )
                with urllib.request.urlopen(req, timeout=15) as resp:
                    html_content = resp.read(50000).decode(errors="replace")

                web_source = WebSearchSource(fetch_content=False)
                ddg_results = web_source._parse_ddg_html(html_content, limit)

                for r in ddg_results:
                    url = r.get("url", "")
                    norm_url = _normalize_url(url)
                    if norm_url in seen_urls:
                        continue
                    seen_urls.add(norm_url)

                    # Filter for likely doc pages
                    doc_indicators = [
                        "/docs/", "/doc/", "/documentation", "docs.",
                        "/reference", "/api/", "/guide", "/tutorial",
                        "readthedocs.io", "devdocs.io",
                    ]
                    is_doc = any(ind in url.lower() for ind in doc_indicators)

                    # Fetch content for likely doc pages
                    if is_doc and self.fetch_content:
                        content = _fetch_url(url, max_chars=30000)
                        if content:
                            r["content"] = _extract_text(content)[:12000]
                            r["source_type"] = self.source_type
                            r["relevance_score"] = 0.6
                            r["content_hash"] = _content_hash(r["content"])
                            results.append(r)
                            if len(results) >= limit:
                                return results

            except Exception:
                continue

        return results


# ─── Relevance Scorer ─────────────────────────────────────────────────────────


def score_results(
    results: list[dict],
    topic: str,
    use_llm: bool = True,
) -> list[dict]:
    """Score results by relevance to the original research topic.

    If use_llm=True and an LLM is configured, uses the shared client for intelligent scoring.
    Otherwise falls back to keyword-based scoring.

    Returns results sorted by relevance_score descending.
    """
    if not results:
        return []

    if use_llm:
        return _score_with_llm(results, topic)
    else:
        return _score_keyword(results, topic)


def _score_with_llm(results: list[dict], topic: str) -> list[dict]:
    """Score results using GLM for semantic relevance."""
    # Batch results to avoid context overflow (max ~8 results per batch)
    BATCH_SIZE = 8
    all_scored = []

    for i in range(0, len(results), BATCH_SIZE):
        batch = results[i : i + BATCH_SIZE]
        scored = _score_batch_llm(batch, topic, offset=i)
        all_scored.extend(scored)

    all_scored.sort(key=lambda x: x.get("relevance_score", 0), reverse=True)
    return all_scored


def _score_batch_llm(batch: list[dict], topic: str, offset: int = 0) -> list[dict]:
    """Score a batch of results using GLM."""
    system = (
        "You are a research relevance scorer. Score each search result's "
        "relevance to the research topic on a scale of 0.0 to 1.0.\n\n"
        "Scoring:\n"
        "- 1.0: Directly answers or is essential for the topic\n"
        "- 0.8: Highly relevant supporting information\n"
        "- 0.6: Somewhat relevant but tangential\n"
        "- 0.4: Loosely related\n"
        "- 0.2: Barely related\n"
        "- 0.0: Irrelevant\n\n"
        "Output ONLY JSON array: [{\"index\": N, \"score\": 0.X, \"reason\": \"...\"}]"
    )

    numbered = []
    for i, r in enumerate(batch):
        content_preview = (r.get("content") or "")[:500]
        numbered.append(
            f"[{i}] {r.get('title', 'Untitled')} ({r.get('url', 'unknown')})\n"
            f"{content_preview}"
        )

    prompt = (
        f"Research topic: {topic}\n\n"
        f"Results to score:\n{chr(10).join(numbered)}\n\n"
        "Score each result."
    )

    try:
        response = _call_glm(prompt, system, max_tokens=2048, temperature=0.1)
        response = response.strip()
        if response.startswith("```"):
            response = response.split("\n", 1)[1].rsplit("```", 1)[0].strip()

        scores = json.loads(response)
        if not isinstance(scores, list):
            scores = [scores]

        score_map = {s.get("index", 0): s for s in scores}

        result = []
        for i, r in enumerate(batch):
            merged = {**r}
            if i in score_map:
                merged["relevance_score"] = float(score_map[i].get("score", 0.5))
                merged["relevance_reason"] = score_map[i].get("reason", "")
            else:
                merged["relevance_score"] = 0.5
            result.append(merged)

        return result

    except Exception:
        # Fallback to keyword scoring
        return _score_keyword(batch, topic)


def _score_keyword(results: list[dict], topic: str) -> list[dict]:
    """Simple keyword-based relevance scoring (fallback)."""
    topic_words = set(topic.lower().split())
    # Remove common stop words
    stop_words = {"a", "an", "the", "is", "are", "was", "were", "be", "been",
                  "being", "have", "has", "had", "do", "does", "did", "will",
                  "would", "could", "should", "may", "might", "can", "shall",
                  "how", "what", "when", "where", "which", "who", "why", "in",
                  "on", "at", "to", "for", "of", "with", "by", "from", "and",
                  "or", "but", "not", "this", "that", "it"}
    topic_words -= stop_words

    scored = []
    for r in results:
        text = f"{r.get('title', '')} {r.get('content', '')}".lower()
        words = set(text.split())

        if not topic_words:
            score = 0.5
        else:
            overlap = len(topic_words & words)
            score = min(1.0, overlap / max(1, len(topic_words)) * 1.5)
            # Boost if title matches
            title_words = set(r.get("title", "").lower().split())
            title_overlap = len(topic_words & title_words)
            score = min(1.0, score + title_overlap * 0.1)

        scored.append({**r, "relevance_score": round(score, 2)})

    scored.sort(key=lambda x: x.get("relevance_score", 0), reverse=True)
    return scored


# ─── Deduplication ────────────────────────────────────────────────────────────


def deduplicate_results(results: list[dict]) -> list[dict]:
    """Deduplicate results by URL and content hash.

    Keeps the result with higher relevance_score when duplicates found.
    """
    seen_urls = set()
    seen_hashes = set()
    unique = []

    for r in sorted(results, key=lambda x: x.get("relevance_score", 0), reverse=True):
        url = _normalize_url(r.get("url", ""))
        content_hash = r.get("content_hash", "")

        if url and url in seen_urls:
            continue
        if content_hash and content_hash in seen_hashes:
            continue

        if url:
            seen_urls.add(url)
        if content_hash:
            seen_hashes.add(content_hash)

        unique.append(r)

    return unique


# ─── CLI ──────────────────────────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(description="Research source adapters")
    sub = parser.add_subparsers(dest="command")

    # Web search
    wp = sub.add_parser("web")
    wp.add_argument("query")
    wp.add_argument("--limit", "-n", type=int, default=5)
    wp.add_argument("--no-fetch", action="store_true", help="Skip content fetching")

    # GitHub search
    gp = sub.add_parser("github")
    gp.add_argument("query")
    gp.add_argument("--limit", "-n", type=int, default=5)
    gp.add_argument("--repos-only", action="store_true")
    gp.add_argument("--code-only", action="store_true")

    # Docs search
    dp = sub.add_parser("docs")
    dp.add_argument("query")
    dp.add_argument("--limit", "-n", type=int, default=3)

    # Score results
    sp = sub.add_parser("score")
    sp.add_argument("topic")
    sp.add_argument("--input", "-i", help="JSON file with results (or stdin)")
    sp.add_argument("--stdin", action="store_true")

    args = parser.parse_args()

    if args.command == "web":
        source = WebSearchSource(fetch_content=not args.no_fetch)
        results = source.search(args.query, args.limit)
        print(json.dumps(results, indent=2, ensure_ascii=False))

    elif args.command == "github":
        source = GitHubSource()
        if args.code_only:
            results = source.search_code(args.query, args.limit)
        elif args.repos_only:
            results = source.search_repos(args.query, args.limit)
        else:
            results = source.search(args.query, args.limit)
        print(json.dumps(results, indent=2, ensure_ascii=False))

    elif args.command == "docs":
        source = DocSource()
        results = source.search(args.query, args.limit)
        print(json.dumps(results, indent=2, ensure_ascii=False))

    elif args.command == "score":
        if args.stdin or not args.input:
            data = json.load(sys.stdin)
        else:
            with open(args.input) as f:
                data = json.load(f)
        scored = score_results(data, args.topic)
        print(json.dumps(scored, indent=2, ensure_ascii=False))

    else:
        parser.print_help()
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/researcher.py
content: """Researcher Agent — Orchestrates multi-source searches for deep research.

Takes a topic, generates search queries, searches across multiple sources
(web, GitHub, docs) in parallel, collects and deduplicates results, scores
by relevance, and returns structured research output.

This is Phase 2 of Deep Research v2. Integrates with Phase 1 scripts:
- query_generator.py for query expansion (3-5 variants per dimension)
- chunk_selector.py for relevance filtering (threshold >= 0.7)
- context_expander.py for context enrichment

Usage:
    # Full research on a topic
    python researcher.py "OpenClaw skills system"

    # Single dimension research (importable)
    from researcher import research_dimension
    findings = research_dimension("architecture", ["How does X work?", "What are the components?"])

    # CLI with options
    python researcher.py "OpenClaw skills system" --sources web github --limit 5

Output: JSON with findings, sources, gaps, and metadata.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import sys
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed, TimeoutError as FuturesTimeout
from typing import Optional

from llm_client import call_llm as _call_glm

# Reuse existing Phase 1 scripts
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from query_generator import generate_queries
from chunk_selector import select_relevant_chunks
from context_expander import expand_context
from research_sources import (
    WebSearchSource,
    GitHubSource,
    DocSource,
    score_results,
    deduplicate_results,
)

# ─── Config ───────────────────────────────────────────────────────────────────

SEARCH_TIMEOUT = 30       # seconds per individual search call
FETCH_TIMEOUT = 30        # seconds per URL fetch
MAX_PARALLEL_SEARCHES = 3 # concurrent search workers
CHUNK_SELECTOR_THRESHOLD = 0.7  # minimum score to keep a chunk


def _has_llm_config() -> bool:
    return bool(
        os.environ.get("LLM_API_KEY")
        or os.environ.get("OPENAI_API_KEY")
        or os.environ.get("ZAI_API_KEY")
    )


def _parse_json_response(text: str):
    """Parse JSON from LLM, stripping markdown fences."""
    text = text.strip()
    if text.startswith("```"):
        text = text.split("\n", 1)[1].rsplit("```", 1)[0].strip()
    return json.loads(text)


# ─── Source registry ──────────────────────────────────────────────────────────

SOURCE_REGISTRY = {
    "web": WebSearchSource,
    "github": GitHubSource,
    "docs": DocSource,
}


# ─── Parallel search helpers ──────────────────────────────────────────────────


def _search_one(source_instance, query_text: str, limit: int) -> tuple[str, list[dict]]:
    """Execute a single search call. Returns (query, results).

    Designed to run in a thread — catches all exceptions and returns
    empty results instead of raising.
    """
    try:
        results = source_instance.search(query_text, limit=limit)
        return query_text, results
    except Exception as e:
        print(f"[timeout/error] search '{query_text[:60]}' failed: {e}", file=sys.stderr)
        return query_text, []


def _parallel_search(
    source_instance,
    queries: list[dict],
    max_results_per_source: int = 5,
    max_workers: int = MAX_PARALLEL_SEARCHES,
    timeout: int = SEARCH_TIMEOUT,
) -> list[dict]:
    """Run multiple search queries in parallel, return combined results.

    Args:
        source_instance: A source adapter (WebSearchSource, GitHubSource, etc.)
        queries: List of query dicts with 'query' key
        max_results_per_source: Max results per query
        max_workers: Max parallel search threads
        timeout: Timeout per search call (seconds)

    Returns:
        Combined list of result dicts from all queries.
    """
    all_results = []
    futures = {}

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        for q in queries:
            query_text = q["query"]
            future = executor.submit(
                _search_one, source_instance, query_text, max_results_per_source
            )
            futures[future] = query_text

        for future in as_completed(futures, timeout=timeout + 5):
            try:
                query_text, results = future.result(timeout=timeout)
                all_results.extend(results)
            except FuturesTimeout:
                query_text = futures[future]
                print(f"[timeout] search '{query_text[:60]}' exceeded {timeout}s", file=sys.stderr)
            except Exception as e:
                query_text = futures[future]
                print(f"[error] search '{query_text[:60]}': {e}", file=sys.stderr)

    return all_results


# ─── Query adaptation ─────────────────────────────────────────────────────────


def _adapt_queries_for_source(queries: list[dict], source_type: str, topic: str) -> list[dict]:
    """Adapt search queries for specific source types.

    GitHub search works better with short keyword queries.
    Web search handles longer natural language queries.
    Docs search benefits from including 'docs' or 'documentation'.
    """
    if source_type == "github":
        adapted = []
        adapted.append({"type": "keyword", "query": topic, "rationale": "Direct topic"})
        for q in queries[:3]:
            query_text = q["query"]
            simplified = re.sub(
                r"\b(how|what|when|where|which|who|why|does|is|are|the|a|an)\b",
                "", query_text, flags=re.IGNORECASE,
            )
            simplified = re.sub(r"\s+", " ", simplified).strip()
            words = simplified.split()
            if len(words) > 6:
                simplified = " ".join(words[:6])
            if simplified and simplified != topic:
                adapted.append({**q, "query": simplified})
        return adapted[:3]

    elif source_type == "docs":
        adapted = []
        for q in queries[:3]:
            adapted.append({**q, "query": f"{q['query']} documentation"})
        return adapted

    else:
        return queries


# ─── Main researcher ──────────────────────────────────────────────────────────


def research(
    topic: str,
    sources: Optional[list[str]] = None,
    max_queries: int = 5,
    max_results_per_source: int = 5,
    min_relevance: float = 0.4,
    use_chunk_selector: bool = True,
    use_context_expander: bool = True,
    [REDACTED_SECRET] = None,
) -> dict:
    """Run the full researcher pipeline.

    Args:
        topic: Research topic or question.
        sources: List of source types to search (default: ["web", "github", "docs"]).
        max_queries: Max search queries to generate (3-5 recommended).
        max_results_per_source: Max results per source per query.
        min_relevance: Minimum relevance score to include in output.
        use_chunk_selector: Run Phase 1 chunk_selector for LLM-based filtering.
        use_context_expander: Run Phase 1 context_expander for enrichment.
        [REDACTED_SECRET] ZAI_API_KEY override.

    Returns:
        Dict with:
            - status: "complete" | "partial" | "no_results"
            - topic: str
            - findings: list of structured findings
            - sources: list of source metadata
            - gaps: list of identified gaps
            - metadata: pipeline stats
    """
    if [REDACTED_SECRET]"ZAI_API_KEY"] = api_key

    start_time = time.time()
    sources = sources or ["web", "github", "docs"]

    # ── Step 1: Generate search queries (3-5 variants) ─────────────────────
    queries = []
    try:
        queries = generate_queries(topic, max_queries=max(3, min(5, max_queries)))
    except Exception as e:
        print(f"Warning: query generation failed ({e}), using topic directly", file=sys.stderr)
        queries = [{"type": "direct", "query": topic, "rationale": "Fallback"}]

    query_texts = [q["query"] for q in queries]
    print(f"Generated {len(query_texts)} queries: {query_texts[:3]}...", file=sys.stderr)

    # ── Step 2: Parallel search across all sources ─────────────────────────
    all_raw_results = []
    source_stats = {}

    for source_type in sources:
        source_class = SOURCE_REGISTRY.get(source_type)
        if not source_class:
            print(f"Warning: unknown source type '{source_type}'", file=sys.stderr)
            continue

        source = source_class()
        source_queries = _adapt_queries_for_source(queries, source_type, topic)

        # Run queries in parallel (up to 3 concurrent)
        source_results = _parallel_search(
            source,
            source_queries,
            max_results_per_source=max_results_per_source,
            max_workers=MAX_PARALLEL_SEARCHES,
            timeout=SEARCH_TIMEOUT,
        )

        source_stats[source_type] = {
            "queries_run": len(source_queries),
            "results_found": len(source_results),
        }
        all_raw_results.extend(source_results)

    print(f"Collected {len(all_raw_results)} raw results from {len(sources)} sources", file=sys.stderr)

    if not all_raw_results:
        return {
            "status": "no_results",
            "topic": topic,
            "findings": [],
            "sources": [],
            "gaps": [{"question": topic, "reason": "No results found from any source"}],
            "metadata": {
                "queries": query_texts,
                "source_stats": source_stats,
                "elapsed_seconds": round(time.time() - start_time, 1),
            },
        }

    # ── Step 3: Deduplicate by URL ─────────────────────────────────────────
    deduped = deduplicate_results(all_raw_results)
    print(f"After dedup: {len(deduped)} unique results", file=sys.stderr)

    # ── Step 4: Score by relevance ────────────────────────────────────────
    scored = score_results(deduped, topic, use_llm=_has_llm_config())

    relevant = [r for r in scored if r.get("relevance_score", 0) >= min_relevance]
    print(f"After relevance filter (>= {min_relevance}): {len(relevant)} results", file=sys.stderr)

    if not relevant:
        relevant = sorted(scored, key=lambda x: x.get("relevance_score", 0), reverse=True)[:5]
        print(f"Relaxed to top {len(relevant)} results", file=sys.stderr)

    # ── Step 5: LLM chunk selection (threshold >= 0.7) ─────────────────────
    if use_chunk_selector and len(relevant) > 3:
        try:
            selected = select_relevant_chunks(
                topic, relevant,
                min_score=CHUNK_SELECTOR_THRESHOLD,
                max_context_tokens=12000,
            )
            print(f"Chunk selector (>= {CHUNK_SELECTOR_THRESHOLD}): {len(selected)} of {len(relevant)} passed", file=sys.stderr)
        except Exception as e:
            print(f"Warning: chunk selection failed ({e}), using all relevant", file=sys.stderr)
            selected = relevant
    else:
        selected = relevant

    if not selected and relevant:
        # If all were filtered out, keep the top 3
        selected = sorted(relevant, key=lambda x: x.get("relevance_score", 0), reverse=True)[:3]
        print(f"Chunk selector filtered everything, kept top {len(selected)} by relevance", file=sys.stderr)

    # ── Step 6: Context expansion for incomplete chunks ────────────────────
    if use_context_expander and len(selected) > 0:
        try:
            expanded = expand_context(selected, topic, max_expansions=5)
            print(f"Context expansion: {sum(1 for c in expanded if c.get('expanded'))} expanded", file=sys.stderr)
        except Exception as e:
            print(f"Warning: context expansion failed ({e})", file=sys.stderr)
            expanded = selected
    else:
        expanded = selected

    # ── Step 7: Extract structured findings ───────────────────────────────
    findings = _extract_findings(expanded, topic)
    gaps = _identify_gaps(expanded, topic, findings)

    # ── Step 8: Build output ──────────────────────────────────────────────
    sources_output = []
    seen_source_urls = set()
    for r in expanded:
        url = r.get("url", "")
        if url and url not in seen_source_urls:
            seen_source_urls.add(url)
            sources_output.append({
                "title": r.get("title", ""),
                "url": url,
                "source_type": r.get("source_type", "unknown"),
                "relevance_score": r.get("relevance_score", 0),
                "relevance_reason": r.get("relevance_reason", ""),
                "expanded": r.get("expanded", False),
            })

    elapsed = round(time.time() - start_time, 1)

    return {
        "status": "complete" if findings else "partial",
        "topic": topic,
        "findings": findings,
        "sources": sources_output,
        "gaps": gaps,
        "metadata": {
            "queries_generated": len(queries),
            "queries": query_texts,
            "raw_results": len(all_raw_results),
            "after_dedup": len(deduped),
            "after_scoring": len(relevant),
            "after_chunk_selection": len(selected),
            "after_expansion": len(expanded),
            "source_stats": source_stats,
            "elapsed_seconds": elapsed,
        },
    }


# ─── Dimension-based research ─────────────────────────────────────────────────


def research_dimension(
    dimension: str,
    questions: list[str],
    sources: Optional[list[str]] = None,
    max_queries_per_question: int = 3,
    max_results_per_source: int = 5,
    min_relevance: float = 0.4,
    use_chunk_selector: bool = True,
    use_context_expander: bool = True,
    [REDACTED_SECRET] = None,
) -> dict:
    """Research a specific dimension of a topic with multiple questions.

    This is the key function for sessions_spawn subagents. Each subagent
    gets a dimension (e.g., "architecture", "market_analysis", "risks") and
    a list of questions to answer. Returns structured findings.

    Args:
        dimension: The research dimension name (e.g., "architecture", "market", "risks").
        questions: List of specific questions to research within this dimension.
        sources: Source types to search (default: ["web", "github", "docs"]).
        max_queries_per_question: Search variants per question (3-5 recommended).
        max_results_per_source: Max results per source per query.
        min_relevance: Minimum relevance score for results.
        use_chunk_selector: Run LLM-based chunk filtering.
        use_context_expander: Expand context for incomplete chunks.
        [REDACTED_SECRET] ZAI_API_KEY override.

    Returns:
        Dict with:
            - dimension: str
            - questions: list[str]
            - findings: list[dict] with claim, source_url, source_title,
                        confidence, category, supporting_question
            - sources: list[dict] with url, title, relevance_score
            - coverage: float (0.0-1.0, how well questions were answered)
            - metadata: pipeline stats
    """
    if [REDACTED_SECRET]"ZAI_API_KEY"] = api_key

    start_time = time.time()
    sources = sources or ["web", "github", "docs"]
    all_findings = []
    all_sources = []
    seen_urls = set()
    question_coverage = {}
    total_queries = 0
    total_raw = 0

    for question in questions:
        q_start = time.time()
        print(f"\n── Researching: [{dimension}] {question[:80]} ──", file=sys.stderr)

        # 1. Generate 3-5 query variants for this question
        try:
            queries = generate_queries(question, max_queries=max_queries_per_question)
        except Exception as e:
            print(f"  Warning: query gen failed ({e})", file=sys.stderr)
            queries = [{"type": "direct", "query": question, "rationale": "Fallback"}]

        total_queries += len(queries)

        # 2. Parallel search across sources
        question_results = []
        for source_type in sources:
            source_class = SOURCE_REGISTRY.get(source_type)
            if not source_class:
                continue
            source = source_class()
            source_queries = _adapt_queries_for_source(queries, source_type, question)

            results = _parallel_search(
                source,
                source_queries,
                max_results_per_source=max_results_per_source,
                max_workers=MAX_PARALLEL_SEARCHES,
                timeout=SEARCH_TIMEOUT,
            )
            question_results.extend(results)

        total_raw += len(question_results)

        if not question_results:
            question_coverage[question] = {"covered": False, "reason": "No results"}
            continue

        # 3. Deduplicate by URL
        deduped = deduplicate_results(question_results)

        # 4. Score and filter
        scored = score_results(deduped, question, use_llm=_has_llm_config())
        relevant = [r for r in scored if r.get("relevance_score", 0) >= min_relevance]
        if not relevant:
            relevant = sorted(scored, key=lambda x: x.get("relevance_score", 0), reverse=True)[:3]

        # 5. LLM chunk selection (threshold >= 0.7)
        if use_chunk_selector and len(relevant) > 2:
            try:
                selected = select_relevant_chunks(
                    question, relevant,
                    min_score=CHUNK_SELECTOR_THRESHOLD,
                    max_context_tokens=10000,
                )
            except Exception:
                selected = relevant
        else:
            selected = relevant

        if not selected and relevant:
            selected = sorted(relevant, key=lambda x: x.get("relevance_score", 0), reverse=True)[:2]

        # 6. Context expansion
        if use_context_expander and len(selected) > 0:
            try:
                expanded = expand_context(selected, question, max_expansions=3)
            except Exception:
                expanded = selected
        else:
            expanded = selected

        # 7. Extract findings for this question
        findings = _extract_findings(expanded, question)

        # Tag each finding with the dimension and question
        for f in findings:
            f["dimension"] = dimension
            f["supporting_question"] = question

        all_findings.extend(findings)

        # Collect unique sources
        for r in expanded:
            url = r.get("url", "")
            if url and url not in seen_urls:
                seen_urls.add(url)
                all_sources.append({
                    "title": r.get("title", ""),
                    "url": url,
                    "source_type": r.get("source_type", "unknown"),
                    "relevance_score": r.get("relevance_score", 0),
                    "relevance_reason": r.get("relevance_reason", ""),
                })

        question_coverage[question] = {
            "covered": len(findings) > 0,
            "findings_count": len(findings),
            "elapsed": round(time.time() - q_start, 1),
        }

        print(f"  → {len(findings)} findings in {round(time.time() - q_start, 1)}s", file=sys.stderr)

    # Calculate coverage score
    covered = sum(1 for v in question_coverage.values() if v.get("covered", False))
    coverage = round(covered / max(1, len(questions)), 2)

    elapsed = round(time.time() - start_time, 1)

    return {
        "dimension": dimension,
        "questions": questions,
        "findings": all_findings,
        "sources": all_sources,
        "coverage": coverage,
        "question_coverage": question_coverage,
        "metadata": {
            "total_queries": total_queries,
            "total_raw_results": total_raw,
            "total_findings": len(all_findings),
            "unique_sources": len(all_sources),
            "elapsed_seconds": elapsed,
        },
    }


# ─── Finding extraction ──────────────────────────────────────────────────────


def _extract_findings(chunks: list[dict], topic: str) -> list[dict]:
    """Use LLM to extract structured findings from processed chunks."""
    if not chunks:
        return []

    if not _has_llm_config():
        return _extract_findings_simple(chunks, topic)

    system = (
        "You are a research analyst. Extract key findings from the provided source chunks. "
        "Each finding must be directly supported by the source text.\n\n"
        "Output ONLY a JSON array. Each finding:\n"
        '{"claim": "...", "source_url": "...", "source_title": "...", '
        '"confidence": 0.0-1.0, "category": "technical|concept|comparison|data|opinion"}\n\n'
        "Rules:\n"
        "- Only include claims directly supported by source text\n"
        "- Never fabricate or extrapolate beyond what's stated\n"
        "- confidence 1.0 = directly stated, 0.8 = strongly implied, 0.6 = inferred\n"
        "- Group similar findings into single claims with multiple sources\n"
        "- Maximum 15 findings"
    )

    chunks_text = []
    total_chars = 0
    MAX_CHARS = 20000

    for c in chunks:
        content = c.get("content", "")[:2000]
        chunk_str = f"[{c.get('url', 'unknown')}] {c.get('title', 'Untitled')}\n{content}"
        if total_chars + len(chunk_str) > MAX_CHARS:
            break
        chunks_text.append(chunk_str)
        total_chars += len(chunk_str)

    prompt = (
        f"Research topic: {topic}\n\n"
        f"Source chunks:\n{chr(10).join(chunks_text)}\n\n"
        "Extract findings. Output JSON array only."
    )

    try:
        response = _call_glm(prompt, system, max_tokens=4096, temperature=0.2)
        findings = _parse_json_response(response)
        if not isinstance(findings, list):
            findings = [findings]
        return findings[:20]
    except Exception as e:
        print(f"Warning: finding extraction failed ({e})", file=sys.stderr)
        return _extract_findings_simple(chunks, topic)


def _extract_findings_simple(chunks: list[dict], topic: str) -> list[dict]:
    """Simple rule-based finding extraction (fallback when no API key)."""
    findings = []
    for c in chunks:
        content = c.get("content", "")
        if not content or len(content) < 50:
            continue

        sentences = [s.strip() for s in content.replace("\n", ".").split(".") if len(s.strip()) > 20]
        for sentence in sentences[:2]:
            findings.append({
                "claim": sentence,
                "source_url": c.get("url", ""),
                "source_title": c.get("title", ""),
                "confidence": 0.5,
                "category": "extracted",
            })

    return findings[:15]


# ─── Gap identification ──────────────────────────────────────────────────────


def _identify_gaps(
    chunks: list[dict], topic: str, findings: list[dict]
) -> list[dict]:
    """Identify research gaps based on what was found vs. what the topic needs."""
    if not _has_llm_config() or not findings:
        return _identify_gaps_simple(topic, chunks, findings)

    system = (
        "You are a research gap analyzer. Given a research topic and the findings "
        "so far, identify what important aspects are missing or under-explored.\n\n"
        "Output ONLY a JSON array:\n"
        '[{"question": "...", "reason": "...", "importance": "high|medium|low"}]\n\n'
        "Rules:\n"
        "- Focus on gaps that would significantly improve understanding of the topic\n"
        "- Don't list things that are adequately covered by findings\n"
        "- Maximum 5 gaps\n"
        "- If findings are comprehensive, return empty array"
    )

    findings_summary = json.dumps(
        [{"claim": f.get("claim", "")[:200]} for f in findings[:10]],
        ensure_ascii=False,
    )

    prompt = (
        f"Research topic: {topic}\n\n"
        f"Findings so far ({len(findings)}):\n{findings_summary}\n\n"
        "What important aspects are still missing?"
    )

    try:
        response = _call_glm(prompt, system, max_tokens=2048, temperature=0.3)
        gaps = _parse_json_response(response)
        if not isinstance(gaps, list):
            gaps = [gaps]
        return gaps[:5]
    except Exception:
        return _identify_gaps_simple(topic, chunks, findings)


def _identify_gaps_simple(
    topic: str, chunks: list[dict], findings: list[dict]
) -> list[dict]:
    """Simple gap identification based on coverage heuristics."""
    gaps = []
    topic_lower = topic.lower()

    source_types = set(c.get("source_type", "") for c in chunks)
    if len(source_types) < 2:
        gaps.append({
            "question": f"Diversify sources for: {topic}",
            "reason": f"Only {len(source_types)} source type(s) used ({', '.join(source_types)})",
            "importance": "medium",
        })

    if len(findings) < 3:
        gaps.append({
            "question": f"More information needed on: {topic}",
            "reason": f"Only {len(findings)} finding(s) extracted",
            "importance": "high",
        })

    aspects = {
        "comparison": any(w in topic_lower for w in ["vs", "compare", "alternative", "difference"]),
        "how_to": any(w in topic_lower for w in ["how", "guide", "tutorial", "setup", "install"]),
        "technical": any(w in topic_lower for w in ["architecture", "implementation", "code", "api"]),
    }

    if aspects["comparison"] and len(findings) < 5:
        gaps.append({
            "question": f"Detailed comparison analysis for: {topic}",
            "reason": "Comparison topic but limited findings",
            "importance": "high",
        })

    if not aspects["comparison"] and not any("overview" in f.get("claim", "").lower() for f in findings):
        gaps.append({
            "question": f"High-level overview of: {topic}",
            "reason": "No overview/broad-context finding identified",
            "importance": "medium",
        })

    return gaps


# ─── Save helpers ─────────────────────────────────────────────────────────────


def save_research(output: dict, output_dir: str) -> str:
    """Save research output to files."""
    import os
    os.makedirs(output_dir, exist_ok=True)

    json_path = os.path.join(output_dir, "researcher-output.json")
    with open(json_path, "w") as f:
        json.dump(output, f, indent=2, ensure_ascii=False)

    md_path = os.path.join(output_dir, "research-summary.md")
    md = f"# Research: {output.get('topic', output.get('dimension', 'Unknown'))}\n\n"
    md += f"**Status:** {output.get('status', output.get('coverage', 'N/A'))}\n"
    md += f"**Findings:** {len(output.get('findings', []))}\n"
    md += f"**Sources:** {len(output.get('sources', []))}\n"
    md += f"**Gaps:** {len(output.get('gaps', []))}\n"

    meta = output.get("metadata", {})
    md += f"\n## Metadata\n"
    md += f"- Queries: {meta.get('queries_generated', meta.get('total_queries', 0))}\n"
    md += f"- Raw results: {meta.get('raw_results', meta.get('total_raw_results', 0))}\n"
    md += f"- After dedup: {meta.get('after_dedup', 'N/A')}\n"
    md += f"- After scoring: {meta.get('after_scoring', 'N/A')}\n"
    md += f"- Elapsed: {meta.get('elapsed_seconds', 0)}s\n"

    if output.get("findings"):
        md += "\n## Findings\n\n"
        for i, f in enumerate(output["findings"], 1):
            md += f"### {i}. {f.get('claim', '?')[:100]}\n\n"
            md += f"- **Source:** [{f.get('source_title', '?')}]({f.get('source_url', '#')})\n"
            md += f"- **Confidence:** {f.get('confidence', '?')}\n"
            md += f"- **Category:** {f.get('category', '?')}\n"
            dim = f.get("dimension", "")
            if dim:
                md += f"- **Dimension:** {dim}\n"
            sq = f.get("supporting_question", "")
            if sq:
                md += f"- **Question:** {sq}\n"
            md += "\n"

    if output.get("sources"):
        md += "## Sources\n\n"
        for s in output["sources"]:
            md += f"- [{s.get('title', '?')}]({s.get('url', '#')}) "
            md += f"(score: {s.get('relevance_score', '?')}, type: {s.get('source_type', '?')})\n"

    if output.get("gaps"):
        md += "\n## Gaps\n\n"
        for g in output["gaps"]:
            md += f"- **{g.get('question', '?')}** ({g.get('importance', '?')}): {g.get('reason', '')}\n"

    if output.get("question_coverage"):
        md += "\n## Question Coverage\n\n"
        for q, cov in output["question_coverage"].items():
            status = "✅" if cov.get("covered") else "❌"
            md += f"- {status} {q} ({cov.get('findings_count', 0)} findings, {cov.get('elapsed', '?')}s)\n"

    with open(md_path, "w") as f:
        f.write(md)

    return json_path


# ─── CLI ──────────────────────────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(
        description="Researcher Agent — multi-source research orchestration"
    )
    parser.add_argument("topic", nargs="?", help="Research topic or question")
    parser.add_argument("--stdin", action="store_true", help="Read topic from stdin JSON")
    parser.add_argument(
        "--sources", "-s",
        nargs="+",
        choices=["web", "github", "docs"],
        default=["web", "github", "docs"],
        help="Source types to search (default: all)",
    )
    parser.add_argument("--limit", "-n", type=int, default=5, help="Max results per source")
    parser.add_argument("--max-queries", type=int, default=5)
    parser.add_argument("--min-relevance", type=float, default=0.4)
    parser.add_argument("--no-chunk-selector", action="store_true", help="Skip chunk selection")
    parser.add_argument("--no-context-expander", action="store_true", help="Skip context expansion")
    parser.add_argument("--output-dir", "-o", help="Directory to save results")
    args = parser.parse_args()

    # Get topic
    if args.stdin:
        data = json.load(sys.stdin)
        topic = data.get("topic", data.get("question", data.get("q", "")))
    elif args.topic:
        topic = args.topic
    else:
        print("Error: provide a topic or use --stdin", file=sys.stderr)
        sys.exit(1)

    if not topic:
        print("Error: empty topic", file=sys.stderr)
        sys.exit(1)

    # Run research
    output = research(
        topic=topic,
        sources=args.sources,
        max_queries=args.max_queries,
        max_results_per_source=args.limit,
        min_relevance=args.min_relevance,
        use_chunk_selector=not args.no_chunk_selector,
        use_context_expander=not args.no_context_expander,
    )

    # Save if output dir specified
    if args.output_dir:
        path = save_research(output, args.output_dir)
        print(f"Saved to {path}", file=sys.stderr)

    # Print JSON output
    print(json.dumps(output, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


path: scripts/test_analyst.py
content: import os
import sys
import pytest

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from analyst import analyze_findings


def test_deduplication_merges_near_duplicates():
    findings = [
        {
            "claim": "RAG combines retrieval with LLMs to reduce hallucination.",
            "source_url": "https://example.com/a",
            "dimension": "technical architecture",
            "confidence": 0.9,
        },
        {
            "claim": "Retrieval-augmented generation combines retrieval with LLMs to reduce hallucinations.",
            "source_url": "https://example.com/b",
            "dimension": "technical architecture",
            "confidence": 0.8,
        },
    ]

    result = analyze_findings(findings, dimensions=["technical architecture"])

    # Two highly similar claims should deduplicate into one analyzed finding id.
    assert len(result.confidence_map) == 1
    assert result.coverage_score == 1.0
    assert result.gaps == []


def test_theme_extraction_groups_by_keyword_cooccurrence():
    findings = [
        {"claim": "RAG uses a vector store and embeddings for retrieval.", "source_url": "https://example.com/rag1"},
        {"claim": "Hybrid search combines dense embeddings with sparse retrieval.", "source_url": "https://example.com/rag2"},
        {"claim": "Bond yields rise when inflation expectations increase.", "source_url": "https://example.com/macro1"},
        {"claim": "Higher inflation can push yields up and reduce bond prices.", "source_url": "https://example.com/macro2"},
    ]

    result = analyze_findings(findings, dimensions=["technical", "macro"])

    # Should produce 3-7 themes when enough findings exist.
    assert 1 <= len(result.themes) <= 7
    # At least two themes should exist given two distinct topic areas.
    assert len(result.themes) >= 2
    # Themes must reference finding ids (deduped ids).
    for theme in result.themes:
        assert "finding_ids" in theme
        assert len(theme["finding_ids"]) >= 1


def test_contradiction_detection_flags_opposing_claims():
    findings = [
        {
            "claim": "Increasing interest rates increase mortgage affordability.",
            "source_url": "https://example.com/claim-up",
        },
        {
            "claim": "Increasing interest rates decrease mortgage affordability.",
            "source_url": "https://example.com/claim-down",
        },
    ]

    result = analyze_findings(findings, dimensions=["housing"])

    assert len(result.contradictions) >= 1
    c = result.contradictions[0]
    assert "claim_a" in c and "claim_b" in c
    assert "Opposing polarity" in c.get("reason", "")



path: scripts/test_integration.py
content: """Integration tests for Deep Research v2 — full pipeline: plan → research → analyze → reflect → write.

Tests use mock mode (no real API calls). Minimum 10 test cases.

Run:
    python -m pytest test_integration.py -v
"""

import json
import os
import sys
import time
import unittest
from unittest.mock import patch, MagicMock

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from writer import WriterAgent, write_report, OutputFormat, ConfidenceLevel, _confidence_level, _confidence_badge
from analyst import analyze_findings, AnalysisResult
from reflection import ResearchPlan, reflect, _normalize_findings, _dimension_hits
from research_pipeline import run_looping_pipeline, run_researcher, run_analyst, run_writer


# ─── Test Data ────────────────────────────────────────────────────────────────

MOCK_QUESTION = "What is the current state of quantum computing?"

MOCK_RESEARCHER_OUTPUTS = [
    {
        "status": "complete",
        "dimension": "hardware",
        "queries": ["quantum computing hardware 2026"],
        "findings": [
            {
                "claim": "IBM unveiled a 1,121-qubit Condor processor in 2023",
                "source_url": "https://example.com/ibm-condor",
                "confidence": 0.95,
                "dimension": "hardware",
            },
            {
                "claim": "Google's Sycamore achieved quantum supremacy in 2019",
                "source_url": "https://example.com/google-sycamore",
                "confidence": 0.9,
                "dimension": "hardware",
            },
        ],
        "sources": [
            {"url": "https://example.com/ibm-condor", "title": "IBM Condor", "score": 0.95},
            {"url": "https://example.com/google-sycamore", "title": "Google Sycamore", "score": 0.9},
        ],
        "metadata": {"chunks_scanned": 5, "chunks_selected": 3, "chunks_expanded": 1},
    },
    {
        "status": "complete",
        "dimension": "applications",
        "queries": ["quantum computing applications"],
        "findings": [
            {
                "claim": "Quantum computing shows promise for drug discovery and cryptography",
                "source_url": "https://example.com/qc-apps",
                "confidence": 0.8,
                "dimension": "applications",
            },
            {
                "claim": "Quantum error correction remains a major challenge",
                "source_url": "https://example.com/qc-errors",
                "confidence": 0.85,
                "dimension": "applications",
            },
        ],
        "sources": [
            {"url": "https://example.com/qc-apps", "title": "QC Applications", "score": 0.8},
        ],
        "metadata": {"chunks_scanned": 4, "chunks_selected": 2, "chunks_expanded": 0},
    },
]

MOCK_ANALYST_OUTPUT = {
    "status": "complete",
    "themes": [
        {
            "name": "Quantum Hardware",
            "confidence": 0.9,
            "key_claims": ["IBM Condor 1,121 qubits", "Google Sycamore supremacy"],
            "findings": [],
        },
        {
            "name": "Applications & Challenges",
            "confidence": 0.8,
            "key_claims": ["Drug discovery", "Error correction"],
            "findings": [],
        },
    ],
    "contradictions": [
        {
            "claim_a": "Quantum computers will replace classical computers within 10 years",
            "claim_b": "Quantum computers will complement, not replace, classical computers",
            "source_a": "https://example.com/optimistic",
            "source_b": "https://example.com/pragmatic",
            "resolution": "The consensus leans toward complementarity for the foreseeable future",
        },
    ],
    "gaps": [
        {"question": "What are the latest quantum error correction breakthroughs?", "importance": "high"},
        {"question": "Cost comparison of quantum vs classical for specific workloads?", "importance": "medium"},
    ],
    "merged_findings": (
        MOCK_RESEARCHER_OUTPUTS[0]["findings"] + MOCK_RESEARCHER_OUTPUTS[1]["findings"]
    ),
}


# ─── Phase 3: Writer Tests ───────────────────────────────────────────────────


class TestWriterAgentConfidence(unittest.TestCase):
    """Test confidence level mapping and badges."""

    def test_high_confidence(self):
        self.assertEqual(_confidence_level(0.9), ConfidenceLevel.HIGH)
        self.assertEqual(_confidence_level(0.8), ConfidenceLevel.HIGH)

    def test_medium_confidence(self):
        self.assertEqual(_confidence_level(0.7), ConfidenceLevel.MEDIUM)
        self.assertEqual(_confidence_level(0.6), ConfidenceLevel.MEDIUM)

    def test_low_confidence(self):
        self.assertEqual(_confidence_level(0.5), ConfidenceLevel.LOW)
        self.assertEqual(_confidence_level(0.4), ConfidenceLevel.LOW)

    def test_uncertain_confidence(self):
        self.assertEqual(_confidence_level(0.3), ConfidenceLevel.UNCERTAIN)
        self.assertEqual(_confidence_level(0.0), ConfidenceLevel.UNCERTAIN)

    def test_invalid_confidence(self):
        self.assertEqual(_confidence_level("bad"), ConfidenceLevel.UNCERTAIN)
        self.assertEqual(_confidence_level(None), ConfidenceLevel.UNCERTAIN)

    def test_badges_are_emoji(self):
        for level in ConfidenceLevel:
            badge = _confidence_badge(level)
            self.assertIsInstance(badge, str)
            self.assertTrue(len(badge) >= 1)


class TestWriterAgentReport(unittest.TestCase):
    """Test report generation in all formats."""

    def setUp(self):
        self.agent = WriterAgent(use_llm=False)  # No API calls

    def test_full_report_has_all_sections(self):
        """Full report must contain all required sections."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.REPORT
        )
        report = result["report"]

        self.assertIn("# Research Report:", report)
        self.assertIn("## Executive Summary", report)
        self.assertIn("## Key Findings", report)
        self.assertIn("## Detailed Analysis", report)
        self.assertIn("## Contradictions", report)
        self.assertIn("## Knowledge Gaps", report)
        self.assertIn("## Sources", report)
        self.assertIn("## Methodology", report)
        self.assertEqual(result["status"], "complete")

    def test_full_report_has_citations(self):
        """Report must contain inline source citations."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.REPORT
        )
        report = result["report"]

        # Citations should appear as [source](url)
        self.assertIn("source", report)

    def test_full_report_has_confidence_indicators(self):
        """Report must contain confidence badges/indicators."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.REPORT
        )
        report = result["report"]

        # Should have emoji badges
        self.assertIn("🟢", report)

    def test_full_report_has_contradiction_callouts(self):
        """Report must call out contradictions with warnings."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.REPORT
        )
        report = result["report"]

        self.assertIn("Contradiction", report)
        self.assertIn("Source A:", report)
        self.assertIn("Source B:", report)
        self.assertIn("Resolution:", report)

    def test_summary_format(self):
        """Summary format should be concise."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.SUMMARY
        )
        report = result["report"]

        self.assertIn("Executive Summary", report)
        self.assertIn("Top Findings", report)
        self.assertEqual(result["format"], "summary")

    def test_brief_format(self):
        """Brief format should be bullet-point style."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.BRIEF
        )
        report = result["report"]

        self.assertIn("Research Brief", report)
        self.assertIn("Key Points:", report)
        self.assertEqual(result["format"], "brief")

    def test_json_format(self):
        """JSON format should return structured data."""
        result = self.agent.write_report(
            MOCK_ANALYST_OUTPUT, MOCK_QUESTION, OutputFormat.JSON
        )

        self.assertEqual(result["format"], "json")
        self.assertIn("summary", result)
        self.assertIn("findings", result)
        self.assertIn("contradictions", result)
        self.assertIn("gaps", result)
        self.assertIn("sources", result)
        # Findings should have confidence_level annotations
        for f in result["findings"]:
            self.assertIn("confidence_level", f)
            self.assertIn("confidence_badge", f)

    def test_empty_analyst_output(self):
        """Writer should handle empty input gracefully."""
        empty_output = {
            "themes": [],
            "contradictions": [],
            "gaps": [],
            "merged_findings": [],
        }
        result = self.agent.write_report(
            empty_output, "Empty question?", OutputFormat.REPORT
        )
        self.assertEqual(result["status"], "complete")
        self.assertIn("Research Report", result["report"])


# ─── Phase 4: Integration Tests ──────────────────────────────────────────────


class TestFullPipelineIntegration(unittest.TestCase):
    """End-to-end integration tests: plan → research → analyze → reflect → write."""

    @patch("research_pipeline.generate_queries")
    def test_single_cycle_mock_pipeline(self, mock_gen):
        """Full pipeline in mock mode, 1 cycle."""
        mock_gen.return_value = [
            {"type": "semantic", "query": MOCK_QUESTION, "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question=MOCK_QUESTION,
            max_cycles=1,
            mock_mode=True,
        )
        self.assertEqual(result["status"], "complete")
        self.assertEqual(result["cycles_completed"], 1)
        self.assertIn("quantum computing", result["report"].lower())

        # Check cycle history
        cycle = result["cycle_history"][0]
        self.assertIn("researcher_outputs", cycle)
        self.assertIn("analyst_output", cycle)
        self.assertIn("reflection", cycle)

    @patch("research_pipeline.generate_queries")
    def test_multi_cycle_mock_stops_early(self, mock_gen):
        """Pipeline should stop when coverage is sufficient."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "Simple question", "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question="Simple question",
            max_cycles=5,
            mock_mode=True,
        )
        self.assertLessEqual(result["cycles_completed"], 2)
        self.assertEqual(result["status"], "complete")

    @patch("research_pipeline.generate_queries")
    def test_pipeline_with_writer_integration(self, mock_gen):
        """Pipeline output should be a valid report."""
        mock_gen.return_value = [
            {"type": "semantic", "query": MOCK_QUESTION, "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question=MOCK_QUESTION,
            max_cycles=1,
            mock_mode=True,
        )

        # The mock pipeline produces a basic report
        report = result["report"]
        self.assertIsInstance(report, str)
        self.assertGreater(len(report), 50)

    def test_analyst_to_writer_flow(self):
        """Analyst output feeds correctly into writer."""
        # Step 1: Run analyst on researcher outputs
        with patch("research_pipeline._call_glm") as mock_glm:
            mock_glm.return_value = json.dumps(MOCK_ANALYST_OUTPUT)
            analyst_result = run_analyst(MOCK_RESEARCHER_OUTPUTS)

        # Step 2: Feed into writer
        writer_agent = WriterAgent(use_llm=False)
        writer_result = writer_agent.write_report(
            analyst_result, MOCK_QUESTION, OutputFormat.REPORT
        )

        self.assertEqual(writer_result["status"], "complete")
        self.assertIn("Research Report", writer_result["report"])
        self.assertEqual(writer_result["metadata"]["themes_count"], 2)

    def test_reflection_in_pipeline_context(self):
        """Reflection works correctly within the pipeline loop."""
        plan = ResearchPlan(
            question=MOCK_QUESTION,
            dimensions=["hardware", "applications", "theory"],
            dimension_questions={
                "hardware": ["What processors exist?"],
                "applications": ["What are the use cases?"],
                "theory": ["How does it work?"],
            },
        )

        # Simulate findings that cover 2 of 3 dimensions
        findings = [
            {"dimension": "hardware", "claim": "c1", "confidence": 0.9},
            {"dimension": "applications", "claim": "c2", "confidence": 0.8},
        ]

        result = reflect(plan, findings, cycle=1, max_cycles=5)

        # Should continue because theory dimension is missing
        self.assertTrue(result.should_continue)
        self.assertIn("theory", result.next_dimensions)
        self.assertLess(result.coverage_score, 1.0)

    def test_empty_findings_stop_pipeline(self):
        """Empty findings should stop the pipeline."""
        plan = ResearchPlan(
            question="test",
            dimensions=["dim1", "dim2"],
            dimension_questions={},
        )
        result = reflect(plan, [], cycle=1, max_cycles=5)
        self.assertFalse(result.should_continue)

    @patch("research_pipeline.generate_queries")
    def test_max_cycles_hard_stop(self, mock_gen):
        """Pipeline must stop at max cycles regardless of coverage."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "test", "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question="test",
            max_cycles=8,
            mock_mode=True,
        )
        self.assertLessEqual(result["cycles_completed"], 8)


class TestEdgeCases(unittest.TestCase):
    """Test edge cases: empty results, failures, etc."""

    def test_empty_researcher_output_to_analyst(self):
        """Analyst handles empty researcher outputs."""
        result = run_analyst([{"findings": []}])
        self.assertEqual(result["status"], "no_findings")

    def test_writer_with_minimal_input(self):
        """Writer handles minimal analyst output."""
        minimal = {
            "themes": [],
            "contradictions": [],
            "gaps": [],
            "merged_findings": [],
        }
        agent = WriterAgent(use_llm=False)
        result = agent.write_report(minimal, "test?", OutputFormat.REPORT)
        self.assertEqual(result["status"], "complete")
        self.assertIn("Research Report", result["report"])

    def test_writer_with_single_finding(self):
        """Writer handles single finding."""
        single = {
            "themes": [{"name": "Only Theme", "confidence": 0.7, "key_claims": []}],
            "contradictions": [],
            "gaps": [],
            "merged_findings": [
                {"claim": "Only finding", "confidence": 0.7, "source_url": "https://example.com"}
            ],
        }
        agent = WriterAgent(use_llm=False)
        result = agent.write_report(single, "single?", OutputFormat.REPORT)
        self.assertIn("Only finding", result["report"])

    def test_normalize_findings_researcher_format(self):
        """Normalize handles researcher output format."""
        raw = [
            {"dimension": "dim1", "findings": [
                {"claim": "c1", "confidence": 0.9, "source_url": "u1"},
                {"claim": "c2", "confidence": 0.8, "source_url": "u2"},
            ]},
            {"dimension": "dim2", "findings": [
                {"claim": "c3", "confidence": 0.7, "source_url": "u3"},
            ]},
        ]
        result = _normalize_findings(raw)
        self.assertEqual(len(result), 2)
        self.assertEqual(result[0]["theme"], "dim1")
        self.assertEqual(len(result[0]["claims"]), 2)
        self.assertEqual(result[1]["theme"], "dim2")

    def test_dimension_hits_case_insensitive(self):
        """Dimension matching is case-insensitive."""
        hits = _dimension_hits(
            ["Hardware", "Applications"],
            [
                {"dimension": "hardware", "confidence": 0.9},
                {"theme": "APPLICATIONS", "confidence": 0.8},
            ],
        )
        self.assertEqual(hits["Hardware"], 1)
        self.assertEqual(hits["Applications"], 1)

    def test_convenience_write_report_function(self):
        """The convenience function works correctly."""
        result = write_report(
            MOCK_ANALYST_OUTPUT,
            MOCK_QUESTION,
            fmt="brief",
            use_llm=False,
        )
        self.assertEqual(result["status"], "complete")
        self.assertEqual(result["format"], "brief")
        self.assertIn("Research Brief", result["report"])


class TestReflectionWithAnalystIntegration(unittest.TestCase):
    """Test reflection using analyst output directly."""

    def test_analyst_findings_feed_reflection(self):
        """Analyst merged_findings feed into reflection correctly."""
        plan = ResearchPlan(
            question="test",
            dimensions=["hardware", "applications"],
            dimension_questions={},
        )

        # Use analyst's merged_findings as reflection input
        findings = MOCK_ANALYST_OUTPUT["merged_findings"]
        result = reflect(plan, findings, cycle=1, max_cycles=5)

        # Should have some coverage
        self.assertGreater(result.coverage_score, 0.0)

    def test_reflection_output_structure(self):
        """Reflection result has all required fields."""
        plan = ResearchPlan(question="test", dimensions=["dim1"], dimension_questions={})
        result = reflect(plan, [], cycle=1, max_cycles=3)

        self.assertIsInstance(result.should_continue, bool)
        self.assertIsInstance(result.gaps, list)
        self.assertIsInstance(result.coverage_score, float)
        self.assertIsInstance(result.next_dimensions, list)
        self.assertIsInstance(result.summary, str)
        self.assertIn("Cycle 1/3", result.summary)


if __name__ == "__main__":
    unittest.main()


path: scripts/test_pipeline.py
content: """Tests for research_pipeline.py — mock data, no real API calls."""

import json
import os
import sys
import unittest
from unittest.mock import patch

# Add scripts dir to path
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from research_pipeline import run_researcher, run_analyst, run_writer, _extract_findings, run_looping_pipeline
from reflection import ResearchPlan, ReflectionResult, reflect, _normalize_findings, _dimension_hits, _gap_score


# ─── Test data ────────────────────────────────────────────────────────────────

MOCK_SEARCH_RESULTS = [
    {
        "url": "https://example.com/rag-guide",
        "title": "RAG Architecture Guide",
        "content": "Retrieval-Augmented Generation (RAG) combines retrieval systems with LLMs. The key components are: embedding models, vector stores, and reranking. RAG reduces hallucination by grounding responses in retrieved documents.",
    },
    {
        "url": "https://example.com/rag-benchmarks",
        "title": "RAG Benchmarks 2026",
        "content": "Recent benchmarks show RAG systems achieving 85% accuracy on factual QA tasks. Chunk size of 512 tokens with overlap of 50 tokens performs best. Hybrid search (dense + sparse) outperforms either alone.",
    },
]

MOCK_RESEARCHER_OUTPUTS = [
    {
        "status": "complete",
        "dimension": "technical architecture",
        "queries": ["RAG architecture", "how RAG works"],
        "findings": [
            {
                "claim": "RAG combines retrieval with LLMs to reduce hallucination",
                "source_url": "https://example.com/rag-guide",
                "confidence": 0.9,
                "answers_question": "How does RAG work?",
            },
            {
                "claim": "RAG achieves 85% accuracy on factual QA tasks",
                "source_url": "https://example.com/rag-benchmarks",
                "confidence": 0.85,
                "answers_question": "How effective is RAG?",
            },
        ],
        "sources": [
            {
                "url": "https://example.com/rag-guide",
                "title": "RAG Architecture Guide",
                "score": 0.9,
            },
            {
                "url": "https://example.com/rag-benchmarks",
                "title": "RAG Benchmarks 2026",
                "score": 0.85,
            },
        ],
        "metadata": {"chunks_scanned": 2, "chunks_selected": 2, "chunks_expanded": 0},
    },
    {
        "status": "complete",
        "dimension": "benchmarks",
        "queries": ["RAG benchmarks 2026"],
        "findings": [
            {
                "claim": "Chunk size of 512 tokens with 50 token overlap is optimal",
                "source_url": "https://example.com/rag-benchmarks",
                "confidence": 0.8,
                "answers_question": "Best chunk size?",
            },
        ],
        "sources": [
            {
                "url": "https://example.com/rag-benchmarks",
                "title": "RAG Benchmarks 2026",
                "score": 0.85,
            },
        ],
        "metadata": {"chunks_scanned": 1, "chunks_selected": 1, "chunks_expanded": 0},
    },
]

MOCK_ANALYST_OUTPUT = {
    "status": "complete",
    "themes": [
        {
            "name": "RAG Architecture",
            "confidence": 0.9,
            "key_claims": ["RAG reduces hallucination"],
        }
    ],
    "contradictions": [],
    "gaps": [{"question": "Cost comparison with fine-tuning?", "importance": "medium"}],
    "merged_findings": MOCK_RESEARCHER_OUTPUTS[0]["findings"]
    + MOCK_RESEARCHER_OUTPUTS[1]["findings"],
}


# ─── Tests ────────────────────────────────────────────────────────────────────


class TestRunResearcher(unittest.TestCase):
    """Test researcher pipeline."""

    @patch("research_pipeline.generate_queries")
    def test_researcher_returns_queries_when_no_results(self, mock_gen):
        """Without search results, returns queries for caller."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "RAG architecture", "rationale": "test"}
        ]
        result = run_researcher(
            "How does RAG work?",
            "technical architecture",
            ["What are RAG components?"],
            max_sources=5,
        )
        self.assertEqual(result["status"], "queries_ready")
        self.assertEqual(result["dimension"], "technical architecture")
        self.assertEqual(len(result["queries"]), 1)

    @patch("research_pipeline.generate_queries")
    @patch("research_pipeline.select_relevant_chunks")
    @patch("research_pipeline.expand_context")
    @patch("research_pipeline._extract_findings")
    def test_researcher_full_pipeline(
        self, mock_extract, mock_expand, mock_select, mock_gen
    ):
        """Full pipeline with search results."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "test", "rationale": "r"}
        ]
        mock_select.return_value = [
            {
                "url": "https://example.com",
                "title": "Test",
                "content": "RAG content",
                "score": 0.9,
            }
        ]
        mock_expand.return_value = mock_select.return_value
        mock_extract.return_value = [
            {
                "claim": "RAG works",
                "source_url": "https://example.com",
                "confidence": 0.9,
                "answers_question": "q1",
            }
        ]

        result = run_researcher(
            "How does RAG work?",
            "technical architecture",
            ["What are components?"],
            max_sources=10,
            search_results=MOCK_SEARCH_RESULTS,
        )
        self.assertEqual(result["status"], "complete")
        self.assertEqual(len(result["findings"]), 1)
        self.assertEqual(result["findings"][0]["claim"], "RAG works")
        self.assertEqual(result["metadata"]["chunks_scanned"], 2)
        self.assertEqual(result["metadata"]["chunks_selected"], 1)

    @patch("research_pipeline.generate_queries")
    def test_researcher_no_content(self, mock_gen):
        """Empty search results returns no_content."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "test", "rationale": "r"}
        ]
        result = run_researcher(
            "test",
            "dim",
            [],
            max_sources=5,
            search_results=[{"url": "", "content": None}],
        )
        self.assertEqual(result["status"], "no_content")
        self.assertEqual(result["findings"], [])


class TestRunAnalyst(unittest.TestCase):
    """Test analyst pipeline."""

    @patch("research_pipeline._call_glm")
    def test_analyst_with_mock_llm(self, mock_glm):
        mock_glm.return_value = json.dumps(
            {
                "themes": [
                    {
                        "name": "RAG",
                        "findings": [],
                        "confidence": 0.9,
                        "key_claims": ["claim1"],
                    }
                ],
                "contradictions": [],
                "gaps": [
                    {
                        "question": "gap?",
                        "importance": "medium",
                        "suggested_search": "test",
                    }
                ],
                "merged_findings": [
                    {
                        "claim": "c1",
                        "sources": ["u1"],
                        "confidence": 0.9,
                        "theme": "RAG",
                    }
                ],
            }
        )

        result = run_analyst(MOCK_RESEARCHER_OUTPUTS)
        self.assertEqual(result["status"], "complete")
        self.assertEqual(len(result["themes"]), 1)
        self.assertEqual(result["themes"][0]["name"], "RAG")
        self.assertEqual(result["total_findings_input"], 3)

    @patch("research_pipeline._call_glm")
    def test_analyst_empty_input(self, mock_glm):
        result = run_analyst([{"findings": []}])
        self.assertEqual(result["status"], "no_findings")
        mock_glm.assert_not_called()

    @patch("research_pipeline._call_glm", side_effect=RuntimeError("API fail"))
    def test_analyst_fallback_on_error(self, mock_glm):
        result = run_analyst(MOCK_RESEARCHER_OUTPUTS)
        self.assertEqual(result["status"], "fallback")
        # Should still have merged findings (simple dedup)
        self.assertGreater(len(result["merged_findings"]), 0)


class TestRunWriter(unittest.TestCase):
    """Test writer pipeline."""

    @patch("writer._call_glm")
    def test_writer_generates_report(self, mock_glm):
        mock_glm.return_value = "# RAG Research Report\n\n## Executive Summary\n\nRAG is effective.\n\n## Details\n\nRAG achieves 85% accuracy [https://example.com].\n\n## Confidence\n\nSection confidence: 0.9"

        result = run_writer(MOCK_ANALYST_OUTPUT, "How does RAG work?")
        self.assertEqual(result["status"], "complete")
        self.assertIn("RAG", result["report"])
        self.assertIn("##", result["report"])
        self.assertEqual(result["metadata"]["themes_count"], 1)
        self.assertEqual(result["metadata"]["gaps_count"], 1)

    @patch("writer._call_glm", side_effect=RuntimeError("API fail"))
    def test_writer_fallback_on_error(self, mock_glm):
        result = run_writer(MOCK_ANALYST_OUTPUT, "test?")
        self.assertEqual(result["status"], "complete")
        self.assertIn("Research Report", result["report"])


class TestExtractFindings(unittest.TestCase):
    """Test finding extraction."""

    @patch("research_pipeline._call_glm")
    def test_extract_findings(self, mock_glm):
        mock_glm.return_value = json.dumps(
            [
                {
                    "claim": "c1",
                    "source_url": "http://x",
                    "confidence": 0.9,
                    "answers_question": "q1",
                }
            ]
        )
        chunks = [{"url": "http://x", "content": "some text", "score": 0.8}]
        findings = _extract_findings(chunks, "question?", "dim", ["q1"])
        self.assertEqual(len(findings), 1)
        self.assertEqual(findings[0]["claim"], "c1")

    @patch("research_pipeline._call_glm", side_effect=RuntimeError("fail"))
    def test_extract_findings_fallback(self, mock_glm):
        findings = _extract_findings([], "q?", "d", [])
        self.assertEqual(findings, [])


class TestReflect(unittest.TestCase):
    """Test reflection loop logic — no API calls."""

    def _make_plan(self, dimensions=None):
        return ResearchPlan(
            question="What is RAG?",
            dimensions=dimensions or ["architecture", "benchmarks", "limitations"],
            dimension_questions={"architecture": ["What is RAG?"], "benchmarks": ["How effective?"], "limitations": ["What are the limits?"]},
            budget_seconds=900,
            budget_tokens=60000,
        )

    def test_full_coverage_stops(self):
        """All dimensions covered with high confidence → should_continue=False."""
        plan = self._make_plan()
        findings = [
            {"theme": "architecture", "claims": ["c1"], "confidence": 0.9, "sources": ["s1"], "dimension": "architecture"},
            {"theme": "benchmarks", "claims": ["c2"], "confidence": 0.85, "sources": ["s2"], "dimension": "benchmarks"},
            {"theme": "limitations", "claims": ["c3"], "confidence": 0.8, "sources": ["s3"], "dimension": "limitations"},
        ]
        result = reflect(plan, findings, cycle=1, max_cycles=8)
        self.assertFalse(result.should_continue)
        self.assertGreaterEqual(result.coverage_score, 0.8)
        self.assertEqual(len(result.gaps), 0)

    def test_partial_coverage_continues(self):
        """Only 1 of 3 dimensions covered → should_continue=True."""
        plan = self._make_plan()
        findings = [
            {"theme": "architecture", "claims": ["c1"], "confidence": 0.9, "sources": ["s1"], "dimension": "architecture"},
        ]
        result = reflect(plan, findings, cycle=1, max_cycles=8)
        self.assertTrue(result.should_continue)
        self.assertLess(result.coverage_score, 0.8)
        self.assertGreater(len(result.next_dimensions), 0)

    def test_max_cycles_hard_stop(self):
        """At max_cycles, should_continue=False regardless of coverage."""
        plan = self._make_plan()
        findings = []  # No findings at all
        result = reflect(plan, findings, cycle=8, max_cycles=8)
        self.assertFalse(result.should_continue)

    def test_empty_findings_stops(self):
        """Empty findings after cycle >= 1 → stop."""
        plan = self._make_plan()
        result = reflect(plan, [], cycle=1, max_cycles=8)
        self.assertFalse(result.should_continue)

    def test_progress_summary_includes_cycle_info(self):
        """Summary should mention cycle number and covered/missing dims."""
        plan = self._make_plan()
        findings = [
            {"theme": "architecture", "claims": ["c1"], "confidence": 0.9, "sources": [], "dimension": "architecture"},
        ]
        result = reflect(plan, findings, cycle=2, max_cycles=5)
        self.assertIn("Cycle 2/5", result.summary)
        self.assertIn("architecture", result.summary)

    def test_dimension_hits_fuzzy_match(self):
        """Dimension matching should be case-insensitive and substring-aware."""
        hits = _dimension_hits(["Architecture", "benchmarks"], [
            {"theme": "architecture", "confidence": 0.9},
            {"dimension": "Benchmarks and metrics", "confidence": 0.8},
        ])
        self.assertEqual(hits.get("Architecture", 0), 1)
        self.assertEqual(hits.get("benchmarks", 0), 1)

    def test_gap_score_high_for_no_hits(self):
        """No hits, no confidence → gap score near 1.0."""
        score = _gap_score(hit_count=0, avg_confidence=0.0)
        self.assertGreater(score, 0.7)

    def test_gap_score_low_for_good_coverage(self):
        """Many hits, high confidence → gap score near 0.0."""
        score = _gap_score(hit_count=3, avg_confidence=0.9)
        self.assertLess(score, 0.3)

    def test_normalize_findings_wrapped(self):
        """Unwraps {merged_findings: [...]} format."""
        raw = {"merged_findings": [{"theme": "t", "claims": [], "sources": []}]}
        result = _normalize_findings(raw)
        self.assertEqual(len(result), 1)
        self.assertEqual(result[0]["theme"], "t")

    def test_normalize_findings_flat(self):
        """Groups flat findings by dimension."""
        raw = [
            {"claim": "c1", "dimension": "arch", "confidence": 0.9},
            {"claim": "c2", "dimension": "arch", "confidence": 0.8},
        ]
        result = _normalize_findings(raw)
        self.assertEqual(len(result), 1)
        self.assertEqual(result[0]["theme"], "arch")
        self.assertEqual(len(result[0]["claims"]), 2)


class TestLoopingPipelineIntegration(unittest.TestCase):
    """Integration test: full pipeline in mock mode, 1 cycle."""

    @patch("research_pipeline.generate_queries")
    def test_single_cycle_mock_pipeline(self, mock_gen):
        """Run a 1-cycle research on a simple topic with mock data."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "machine learning", "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question="What is machine learning?",
            max_cycles=1,
            mock_mode=True,
        )
        self.assertEqual(result["status"], "complete")
        self.assertEqual(result["cycles_completed"], 1)
        self.assertGreater(result["final_coverage"], 0.0)
        self.assertIn("machine learning", result["report"].lower())
        self.assertIn("cycle 1", result["report"].lower())
        self.assertEqual(len(result["cycle_history"]), 1)

        # Check cycle history structure
        cycle = result["cycle_history"][0]
        self.assertIn("researcher_outputs", cycle)
        self.assertIn("analyst_output", cycle)
        self.assertIn("reflection", cycle)
        self.assertIn("should_continue", cycle["reflection"])
        self.assertIn("coverage_score", cycle["reflection"])
        self.assertIn("summary", cycle["reflection"])

    @patch("research_pipeline.generate_queries")
    def test_multi_cycle_mock_stops_early(self, mock_gen):
        """Pipeline should stop when coverage is sufficient, even if max_cycles > 1."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "Simple question", "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question="Simple question",
            max_cycles=5,
            mock_mode=True,
        )
        # Mock data covers all dimensions in cycle 1, so it should stop
        self.assertLessEqual(result["cycles_completed"], 2)
        self.assertEqual(result["status"], "complete")

    @patch("research_pipeline.generate_queries")
    def test_pipeline_metadata(self, mock_gen):
        """Pipeline returns correct metadata."""
        mock_gen.return_value = [
            {"type": "semantic", "query": "Test question", "rationale": "test"}
        ]
        result = run_looping_pipeline(
            question="Test question",
            max_cycles=1,
            mock_mode=True,
        )
        meta = result["metadata"]
        self.assertIn("total_duration_seconds", meta)
        self.assertIn("total_findings", meta)
        self.assertEqual(meta["max_cycles"], 1)
        self.assertIsInstance(meta["dimensions"], list)


if __name__ == "__main__":
    unittest.main()


path: scripts/writer.py
content: """Writer Agent — Publication-quality research report generation.

Takes analyst output (themes, contradictions, gaps, merged findings) and
produces polished research reports in multiple formats.

Supports:
- Full markdown report (default)
- Executive summary (concise)
- Bullet-point brief (quick scan)
- JSON structured output

Section structure:
  1. Executive Summary
  2. Key Findings (with confidence indicators)
  3. Detailed Analysis (one section per theme)
  4. Contradictions (where sources disagree)
  5. Knowledge Gaps
  6. Sources
  7. Methodology

Usage:
    # As module
    from writer import WriterAgent
    agent = WriterAgent()
    report = agent.write_report(analyst_output, question="...")

    # As CLI
    python writer.py --input analyst_output.json --question "..." --format report --output report.md
    python writer.py --input analyst_output.json --question "..." --format summary
    python writer.py --input analyst_output.json --question "..." --format brief
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
import urllib.request
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Optional

from llm_client import call_llm as _call_glm


class OutputFormat(Enum):
    REPORT = "report"           # Full markdown report
    SUMMARY = "summary"         # Executive summary only
    BRIEF = "brief"             # Bullet-point brief
    JSON = "json"               # Structured JSON output


class ConfidenceLevel(Enum):
    HIGH = "high"               # >= 0.8
    MEDIUM = "medium"           # >= 0.6
    LOW = "low"                 # >= 0.4
    UNCERTAIN = "uncertain"     # < 0.4


def _confidence_level(score: float) -> ConfidenceLevel:
    """Map numeric confidence to a level."""
    try:
        score = float(score)
    except (TypeError, ValueError):
        return ConfidenceLevel.UNCERTAIN
    if score >= 0.8:
        return ConfidenceLevel.HIGH
    if score >= 0.6:
        return ConfidenceLevel.MEDIUM
    if score >= 0.4:
        return ConfidenceLevel.LOW
    return ConfidenceLevel.UNCERTAIN


def _confidence_badge(level: ConfidenceLevel) -> str:
    """Emoji badge for confidence level."""
    badges = {
        ConfidenceLevel.HIGH: "🟢",
        ConfidenceLevel.MEDIUM: "🟡",
        ConfidenceLevel.LOW: "🟠",
        ConfidenceLevel.UNCERTAIN: "🔴",
    }
    return badges.get(level, "⚪")


def _parse_json_response(text: str) -> Any:
    """Parse JSON from LLM, stripping markdown fences."""
    text = text.strip()
    if text.startswith("```"):
        text = text.split("\n", 1)[1].rsplit("```", 1)[0].strip()
    return json.loads(text)


class WriterAgent:
    """Produces publication-quality research reports from analyst output.

    Args:
        use_llm: If True, uses GLM for section generation.
                 If False, uses template-based generation (no API calls).
        [REDACTED_SECRET] ZAI_API_KEY override.
    """

    MAX_INPUT_CHARS = 15000

    def __init__(self, use_llm: bool = True, [REDACTED_SECRET] = None):
        self.use_llm = use_llm
        self.[REDACTED_SECRET]

    def write_report(
        self,
        analyst_output: dict,
        question: str,
        fmt: OutputFormat = OutputFormat.REPORT,
        metadata: Optional[dict] = None,
    ) -> dict:
        """Generate a research report from analyst output.

        Args:
            analyst_output: Dict with themes, contradictions, gaps, merged_findings.
            question: The original research question.
            fmt: Output format (report, summary, brief, json).
            metadata: Optional metadata to include (cycles, sources count, etc.)

        Returns:
            Dict with 'status', 'report' (markdown string), 'question', 'metadata'.
        """
        themes = analyst_output.get("themes", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])
        merged = analyst_output.get("merged_findings", [])

        if fmt == OutputFormat.JSON:
            return self._write_json(analyst_output, question, metadata)
        elif fmt == OutputFormat.SUMMARY:
            report = self._write_summary(analyst_output, question, metadata)
        elif fmt == OutputFormat.BRIEF:
            report = self._write_brief(analyst_output, question, metadata)
        else:
            report = self._write_full_report(analyst_output, question, metadata)

        return {
            "status": "complete",
            "report": report.strip(),
            "question": question,
            "format": fmt.value,
            "metadata": {
                "themes_count": len(themes),
                "contradictions_count": len(contradictions),
                "gaps_count": len(gaps),
                "total_findings": len(merged),
                **(metadata or {}),
                "generated_at": datetime.now(timezone.utc).isoformat(),
            },
        }

    # ─── Full Report ─────────────────────────────────────────────────────

    def _write_full_report(
        self, analyst_output: dict, question: str, metadata: Optional[dict]
    ) -> str:
        """Write a full markdown report with all sections."""
        themes = analyst_output.get("themes", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])
        merged = analyst_output.get("merged_findings", [])

        sections = []

        # ── Title ──
        sections.append(f"# Research Report: {question}\n")
        sections.append(f"*Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}*")

        if metadata:
            cycles = metadata.get("cycles_completed", "")
            coverage = metadata.get("final_coverage", "")
            if cycles:
                sections[0] += f" | Cycles: {cycles}"
            if coverage:
                sections[0] += f" | Coverage: {coverage:.0%}"

        sections.append("")

        # ── Executive Summary ──
        sections.append("## Executive Summary\n")
        summary = self._generate_executive_summary(analyst_output, question)
        sections.append(summary)
        sections.append("")

        # ── Key Findings ──
        if merged:
            sections.append("## Key Findings\n")
            key_findings = self._format_key_findings(merged)
            sections.append(key_findings)
            sections.append("")

        # ── Detailed Analysis ──
        if themes:
            sections.append("## Detailed Analysis\n")
            for theme in themes:
                theme_section = self._generate_theme_section(theme, merged, question)
                sections.append(theme_section)
                sections.append("")
        elif merged:
            # No themes — write findings directly
            sections.append("## Findings\n")
            findings_text = self._format_findings_as_text(merged, question)
            sections.append(findings_text)
            sections.append("")

        # ── Contradictions ──
        if contradictions:
            sections.append("## Contradictions\n")
            sections.append("> ⚠️ The following areas show disagreement between sources.\n")
            for i, c in enumerate(contradictions, 1):
                sections.append(self._format_contradiction(c, i))
            sections.append("")

        # ── Knowledge Gaps ──
        if gaps:
            sections.append("## Knowledge Gaps\n")
            for g in gaps:
                importance = g.get("importance", "medium")
                icon = {"high": "🔴", "medium": "🟡", "low": "🟢"}.get(importance, "⚪")
                q = g.get("question", "?")
                suggested = g.get("suggested_search", "")
                sections.append(f"- {icon} **{q}** ({importance} importance)")
                if suggested:
                    sections.append(f"  - *Suggested search: {suggested}*")
            sections.append("")

        # ── Sources ──
        all_sources = self._collect_sources(merged, analyst_output)
        if all_sources:
            sections.append("## Sources\n")
            for i, src in enumerate(all_sources, 1):
                title = src.get("title", src.get("url", "?"))
                url = src.get("url", "#")
                score = src.get("score", src.get("relevance_score", ""))
                score_str = f" (relevance: {score})" if score else ""
                sections.append(f"{i}. [{title}]({url}){score_str}")
            sections.append("")

        # ── Methodology ──
        sections.append("## Methodology\n")
        sections.append(self._generate_methodology(analyst_output, metadata))

        return "\n".join(sections)

    # ─── Executive Summary ───────────────────────────────────────────────

    def _generate_executive_summary(
        self, analyst_output: dict, question: str
    ) -> str:
        """Generate executive summary (2-3 paragraphs)."""
        themes = analyst_output.get("themes", [])
        merged = analyst_output.get("merged_findings", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])

        if not self.use_llm:
            return self._template_executive_summary(
                question, themes, merged, contradictions, gaps
            )

        system = (
            "You are a research report writer. Write a concise executive summary "
            "(2-3 paragraphs) for this research report.\n\n"
            "Rules:\n"
            "- Start with the most important finding\n"
            "- Mention key themes covered\n"
            "- Note any contradictions or significant gaps\n"
            "- Be precise — no vague language\n"
            "- Output markdown only, no JSON"
        )

        summary_input = json.dumps(
            {
                "question": question,
                "themes": [t.get("name", "") for t in themes],
                "findings_count": len(merged),
                "top_claims": [f.get("claim", "")[:200] for f in merged[:10]],
                "contradictions_count": len(contradictions),
                "gaps": [g.get("question", "") for g in gaps[:5]],
            },
            ensure_ascii=False,
        )

        prompt = (
            f"Research summary data:\n{summary_input}\n\n"
            "Write an executive summary."
        )

        try:
            return _call_glm(prompt, system, max_tokens=800, temperature=0.4, [REDACTED_SECRET]).strip()
        except Exception as e:
            return self._template_executive_summary(
                question, themes, merged, contradictions, gaps
            )

    def _template_executive_summary(
        self, question, themes, merged, contradictions, gaps
    ) -> str:
        """Template-based executive summary (no LLM needed)."""
        lines = []
        lines.append(
            f"This report investigates: **{question}**. "
            f"Research identified {len(merged)} key findings across "
            f"{len(themes)} thematic area{'s' if len(themes) != 1 else ''}."
        )

        if themes:
            theme_names = [t.get("name", "") for t in themes[:5]]
            lines.append(f"Key areas examined: {', '.join(theme_names)}.")

        if contradictions:
            lines.append(
                f"⚠️ {len(contradictions)} "
                f"area{'s' if len(contradictions) != 1 else ''} of disagreement "
                f"between sources were identified."
            )

        if gaps:
            high_gaps = [g for g in gaps if g.get("importance") == "high"]
            if high_gaps:
                lines.append(
                    f"Significant gaps remain in: "
                    f"{', '.join(g.get('question', '?') for g in high_gaps[:3])}."
                )

        return "\n\n".join(lines)

    # ─── Key Findings ────────────────────────────────────────────────────

    def _format_key_findings(self, merged: list[dict]) -> str:
        """Format merged findings as a bulleted list with confidence indicators."""
        lines = []
        for i, f in enumerate(merged[:15], 1):
            claim = f.get("claim", str(f))
            confidence = f.get("confidence", 0.5)
            level = _confidence_level(confidence)
            badge = _confidence_badge(level)

            # Build citation
            sources = f.get("sources", [])
            if not sources:
                url = f.get("source_url", f.get("url", ""))
                if url:
                    sources = [url]

            citation = ""
            if sources:
                source_strs = []
                for s in sources[:2]:
                    if s.startswith("http"):
                        source_strs.append(f"[source]({s})")
                    else:
                        source_strs.append(s)
                citation = f" — {', '.join(source_strs)}"

            lines.append(f"{i}. {badge} {claim}{citation}")

        if len(merged) > 15:
            lines.append(f"\n*...and {len(merged) - 15} additional findings*")

        return "\n".join(lines)

    # ─── Theme Sections ──────────────────────────────────────────────────

    def _generate_theme_section(
        self, theme: dict, merged: list[dict], question: str
    ) -> str:
        """Generate a detailed section for a single theme."""
        theme_name = theme.get("name", "Unknown")
        theme_confidence = theme.get("confidence", 0)

        # Gather findings for this theme
        theme_findings = [
            f for f in merged
            if f.get("theme") == theme_name or f.get("dimension") == theme_name
        ]
        if not theme_findings:
            theme_findings = theme.get("findings", [])

        level = _confidence_level(theme_confidence)
        badge = _confidence_badge(level)
        header = f"### {theme_name} {badge}"

        if not self.use_llm or not theme_findings:
            return self._template_theme_section(
                header, theme_name, theme_findings, theme_confidence
            )

        section_data = json.dumps(
            {
                "theme": theme_name,
                "confidence": theme_confidence,
                "key_claims": theme.get("key_claims", []),
                "findings": theme_findings,
            },
            indent=2,
            ensure_ascii=False,
        )

        # Truncate if too large
        if len(section_data) > self.MAX_INPUT_CHARS:
            truncated = theme_findings[:5]
            section_data = (
                f"[TRUNCATED — showing top 5 of {len(theme_findings)} findings]\n\n"
                + json.dumps(
                    {"theme": theme_name, "findings": truncated},
                    indent=2,
                    ensure_ascii=False,
                )
            )

        system = (
            "You are a research report writer. Write a detailed markdown section "
            "based on the provided research findings.\n\n"
            "Rules:\n"
            "- Every factual claim MUST have an inline citation: [source URL]\n"
            "- Include confidence indicators for key claims\n"
            "- Use sub-headers, bullet points, and clear structure\n"
            "- Never fabricate — only use provided findings\n"
            "- Output markdown only, no JSON wrapping\n"
            "- Start with a brief overview paragraph, then detailed points"
        )

        prompt = (
            f"Research question: {question}\n"
            f"Theme: {theme_name}\n"
            f"Theme confidence: {theme_confidence}\n\n"
            f"Findings:\n{section_data}\n\n"
            f"Write a detailed markdown section for this theme."
        )

        try:
            section_md = _call_glm(
                prompt, system, max_tokens=8000, temperature=0.4, [REDACTED_SECRET]
            )
            return f"{header}\n\n{section_md.strip()}"
        except Exception as e:
            return self._template_theme_section(
                header, theme_name, theme_findings, theme_confidence
            )

    def _template_theme_section(
        self, header: str, theme_name: str, findings: list[dict], confidence: float
    ) -> str:
        """Template-based theme section (no LLM needed)."""
        lines = [header, ""]
        level = _confidence_level(confidence)
        lines.append(f"*Section confidence: {level.value} ({confidence:.0%})*\n")

        for i, f in enumerate(findings[:10], 1):
            claim = f.get("claim", str(f))
            conf = f.get("confidence", 0.5)
            clevel = _confidence_level(conf)
            cbadge = _confidence_badge(clevel)

            sources = f.get("sources", [])
            if not sources:
                url = f.get("source_url", f.get("url", ""))
                if url:
                    sources = [url]

            citation = ""
            if sources:
                source_strs = []
                for s in sources[:2]:
                    if s.startswith("http"):
                        source_strs.append(f"[source]({s})")
                    else:
                        source_strs.append(s)
                citation = f" {cbadge} — {', '.join(source_strs)}"

            lines.append(f"- {claim}{citation}")

        return "\n".join(lines)

    # ─── Contradiction Formatting ────────────────────────────────────────

    def _format_contradiction(self, c: dict, index: int) -> str:
        """Format a single contradiction as a callout box."""
        claim_a = c.get("claim_a", "?")
        claim_b = c.get("claim_b", "?")
        source_a = c.get("source_a", "")
        source_b = c.get("source_b", "?")
        resolution = c.get("resolution", "Unresolved")
        reason = c.get("reason", "")

        lines = [
            f"**Contradiction {index}**",
        ]
        if reason:
            lines.append(f"*{reason}*")
        lines.append("")
        lines.append(f"- **Source A:** {claim_a}")
        if source_a:
            lines.append(f"  ({source_a})")
        lines.append(f"- **Source B:** {claim_b}")
        if source_b:
            lines.append(f"  ({source_b})")
        lines.append(f"- **Resolution:** {resolution}")

        return "\n".join(lines)

    # ─── Source Collection ───────────────────────────────────────────────

    def _collect_sources(
        self, merged: list[dict], analyst_output: dict
    ) -> list[dict]:
        """Collect unique sources from findings."""
        seen_urls = set()
        sources = []

        # From merged findings
        for f in merged:
            urls = f.get("sources", [])
            if not urls:
                url = f.get("source_url", f.get("url", ""))
                if url:
                    urls = [url]

            for url in urls:
                if url and url not in seen_urls and url != "https://example.com/mock":
                    seen_urls.add(url)
                    sources.append({
                        "url": url,
                        "title": f.get("source_title", url),
                        "score": f.get("confidence", ""),
                    })

        # From analyst_output sources
        for s in analyst_output.get("sources", []):
            url = s.get("url", "")
            if url and url not in seen_urls:
                seen_urls.add(url)
                sources.append(s)

        return sources[:50]

    # ─── Methodology ─────────────────────────────────────────────────────

    def _generate_methodology(
        self, analyst_output: dict, metadata: Optional[dict]
    ) -> str:
        """Generate methodology section."""
        lines = []
        lines.append(
            "This report was generated using the Deep Research v2 pipeline, "
            "which follows a multi-stage research architecture:"
        )
        lines.append("")
        lines.append("1. **Planning** — Research question decomposed into dimensions")
        lines.append("2. **Research** — Multi-query search across web, GitHub, and documentation sources")
        lines.append("3. **Analysis** — Deduplication, theme extraction, contradiction detection")
        lines.append("4. **Reflection** — Coverage assessment and gap analysis")
        lines.append("5. **Writing** — Report generation with inline citations and confidence scoring")

        if metadata:
            lines.append("")
            lines.append("**Research parameters:**")
            if "cycles_completed" in metadata:
                lines.append(f"- Research cycles: {metadata['cycles_completed']}")
            if "total_findings" in metadata:
                lines.append(f"- Total findings: {metadata['total_findings']}")
            if "dimensions" in metadata:
                lines.append(f"- Dimensions: {', '.join(metadata['dimensions'])}")

        lines.append("")
        lines.append(
            "All findings are directly sourced and cited. "
            "Confidence scores reflect source quality and corroboration across multiple sources."
        )

        return "\n".join(lines)

    # ─── Summary Format ──────────────────────────────────────────────────

    def _write_summary(
        self, analyst_output: dict, question: str, metadata: Optional[dict]
    ) -> str:
        """Write executive summary format (concise)."""
        themes = analyst_output.get("themes", [])
        merged = analyst_output.get("merged_findings", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])

        lines = [f"# Executive Summary: {question}\n"]

        # Summary paragraph
        summary = self._generate_executive_summary(analyst_output, question)
        lines.append(summary)
        lines.append("")

        # Top 5 findings
        if merged:
            lines.append("## Top Findings\n")
            for i, f in enumerate(merged[:5], 1):
                claim = f.get("claim", str(f))
                conf = f.get("confidence", 0.5)
                level = _confidence_level(conf)
                badge = _confidence_badge(level)
                lines.append(f"{i}. {badge} {claim}")
            lines.append("")

        # Contradictions count
        if contradictions:
            lines.append(f"⚠️ **{len(contradictions)} contradiction(s)** identified between sources.")

        # High-priority gaps
        high_gaps = [g for g in gaps if g.get("importance") == "high"]
        if high_gaps:
            lines.append("")
            lines.append("## Key Gaps\n")
            for g in high_gaps:
                lines.append(f"- {g.get('question', '?')}")

        return "\n".join(lines)

    # ─── Brief Format ────────────────────────────────────────────────────

    def _write_brief(
        self, analyst_output: dict, question: str, metadata: Optional[dict]
    ) -> str:
        """Write bullet-point brief format."""
        themes = analyst_output.get("themes", [])
        merged = analyst_output.get("merged_findings", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])

        lines = [f"📋 **Research Brief:** {question}\n"]

        # One-liner
        lines.append(f"**Findings:** {len(merged)} | **Themes:** {len(themes)} | "
                     f"**Contradictions:** {len(contradictions)} | **Gaps:** {len(gaps)}\n")

        # Bullet findings
        if merged:
            lines.append("**Key Points:**\n")
            for f in merged[:10]:
                claim = f.get("claim", str(f))
                conf = f.get("confidence", 0.5)
                level = _confidence_level(conf)
                badge = _confidence_badge(level)
                lines.append(f"- {badge} {claim}")

        # Contradictions
        if contradictions:
            lines.append("\n**⚠️ Conflicts:**\n")
            for c in contradictions[:3]:
                lines.append(
                    f"- {c.get('claim_a', '?')[:80]} ↔ {c.get('claim_b', '?')[:80]}"
                )

        # Gaps
        if gaps:
            lines.append("\n**Gaps:**\n")
            for g in gaps[:5]:
                importance = g.get("importance", "medium")
                icon = {"high": "🔴", "medium": "🟡", "low": "🟢"}.get(importance, "⚪")
                lines.append(f"- {icon} {g.get('question', '?')}")

        return "\n".join(lines)

    # ─── JSON Output ─────────────────────────────────────────────────────

    def _write_json(
        self, analyst_output: dict, question: str, metadata: Optional[dict]
    ) -> dict:
        """Write structured JSON output."""
        merged = analyst_output.get("merged_findings", [])
        themes = analyst_output.get("themes", [])
        contradictions = analyst_output.get("contradictions", [])
        gaps = analyst_output.get("gaps", [])

        # Annotate findings with confidence levels
        annotated_findings = []
        for f in merged:
            af = dict(f)
            conf = f.get("confidence", 0.5)
            af["confidence_level"] = _confidence_level(conf).value
            af["confidence_badge"] = _confidence_badge(_confidence_level(conf))
            annotated_findings.append(af)

        return {
            "status": "complete",
            "question": question,
            "format": "json",
            "summary": {
                "total_findings": len(merged),
                "themes": len(themes),
                "contradictions": len(contradictions),
                "gaps": len(gaps),
            },
            "themes": themes,
            "findings": annotated_findings,
            "contradictions": contradictions,
            "gaps": gaps,
            "sources": self._collect_sources(merged, analyst_output),
            "metadata": {
                "generated_at": datetime.now(timezone.utc).isoformat(),
                **(metadata or {}),
            },
        }

    # ─── Helper: format findings as plain text (no LLM) ──────────────────

    def _format_findings_as_text(
        self, findings: list[dict], question: str
    ) -> str:
        """Format findings as structured text without LLM."""
        lines = []
        for i, f in enumerate(findings[:15], 1):
            claim = f.get("claim", str(f))
            conf = f.get("confidence", 0.5)
            level = _confidence_level(conf)
            badge = _confidence_badge(level)

            sources = f.get("sources", [])
            if not sources:
                url = f.get("source_url", f.get("url", ""))
                if url:
                    sources = [url]

            citation = ""
            if sources:
                source_strs = []
                for s in sources[:2]:
                    if s.startswith("http"):
                        source_strs.append(f"[source]({s})")
                    else:
                        source_strs.append(s)
                citation = f" — {', '.join(source_strs)}"

            lines.append(f"{i}. {badge} {claim}{citation}")

        if len(findings) > 15:
            lines.append(f"\n*...and {len(findings) - 15} additional findings*")

        return "\n".join(lines)


# ─── Convenience functions ────────────────────────────────────────────────────


def write_report(
    analyst_output: dict,
    question: str,
    fmt: str = "report",
    use_llm: bool = True,
    metadata: Optional[dict] = None,
) -> dict:
    """Convenience function: generate a research report.

    Args:
        analyst_output: Dict from analyst (themes, contradictions, gaps, merged_findings).
        question: The original research question.
        fmt: Output format — "report", "summary", "brief", or "json".
        use_llm: Use GLM for section generation (default True).
        metadata: Optional metadata dict.

    Returns:
        Dict with 'status', 'report', 'question', 'metadata'.
    """
    try:
        output_fmt = OutputFormat(fmt)
    except ValueError:
        output_fmt = OutputFormat.REPORT

    agent = WriterAgent(use_llm=use_llm)
    return agent.write_report(analyst_output, question, output_fmt, metadata)


def save_report(result: dict, output_path: str) -> str:
    """Save report to a file.

    Args:
        result: Output from write_report().
        output_path: File path to save to.

    Returns:
        The output path.
    """
    os.makedirs(os.path.dirname(os.path.abspath(output_path)), exist_ok=True)

    if result.get("format") == "json":
        content = json.dumps(result, indent=2, ensure_ascii=False)
    else:
        content = result.get("report", "")

    with open(output_path, "w", encoding="utf-8") as f:
        f.write(content)

    return output_path


# ─── CLI ──────────────────────────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(
        description="Writer Agent — generate research reports"
    )
    parser.add_argument("--input", "-i", required=True, help="Analyst output JSON file (or - for stdin)")
    parser.add_argument("--question", "-q", required=True, help="Research question")
    parser.add_argument(
        "--format", "-f",
        choices=["report", "summary", "brief", "json"],
        default="report",
        help="Output format (default: report)",
    )
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")
    parser.add_argument("--no-llm", action="store_true", help="Use template mode (no API calls)")
    parser.add_argument("--metadata", help="JSON file with additional metadata")
    args = parser.parse_args()

    # Load analyst output
    if args.input == "-":
        analyst_output = json.load(sys.stdin)
    else:
        with open(args.input, encoding="utf-8") as f:
            analyst_output = json.load(f)

    # Load optional metadata
    metadata = None
    if args.metadata:
        with open(args.metadata, encoding="utf-8") as f:
            metadata = json.load(f)

    # Generate report
    result = write_report(
        analyst_output=analyst_output,
        question=args.question,
        fmt=args.format,
        use_llm=not args.no_llm,
        metadata=metadata,
    )

    # Output
    if args.format == "json":
        output_text = json.dumps(result, indent=2, ensure_ascii=False)
    else:
        output_text = result.get("report", "")

    if args.output:
        save_report(result, args.output)
        print(f"Report saved to {args.output}", file=sys.stderr)
    else:
        print(output_text)


if __name__ == "__main__":
    main()


path: templates/analyst_input.json
content: {
  "researcher_outputs": [],
  "question": ""
}


path: templates/researcher_brief.json
content: {
  "dimension": "technical architecture",
  "specific_questions": [],
  "context_limit": 5000,
  "max_sources": 10,
  "question": ""
}


path: templates/writer_input.json
content: {
  "analyst_output": {},
  "question": ""
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)