Skill: nacmonad/graph-rag-builder (version 1.0.0)

--- name: GraphRAGBuilderSkill description: > Builds a fully runnable MCP (Model Context Protocol) knowledge server from any website or documentation URL. Crawls the site, extracts concepts using Claude, organizes them into a knowledge graph, generates vector embeddings, and produces a ready-to-install MCP server with 8 semantic search and graph traversal tools. Use this skill whenever the user wants to: build an MCP server from a website or docs, create a semantic search index over documentation, make a knowledge graph from a library or framework's docs, turn a tutorial site into a Claude-searchable knowledge base, index learning materials for AI retrieval, or build a "smart docs" assistant for any topic. Trigger even if the user just mentions "scraping docs", "indexing a site", "building a knowledge graph", or "making docs searchable in Claude". --- # GraphRAG Builder Skill Turns any documentation website into a runnable MCP knowledge server in 5 pipeline steps, each run on the user's local machine using scripts in the `scripts/` folder. ## Quick Reference | Step | Script | What it does | |------|--------------------------|---------------------------------------------| | M1 | `crawl.py` | BFS crawl → raw HTML + metadata per page | | M2 | `extract_concepts.py` | HTML → chunks → LLM concept extraction | | M3 | `build_graph.py` | Concepts + links → networkx knowledge graph | | M4 | `build_embeddings.py` | Chunks + concepts → numpy vector index | | M5 | `generate_mcp_server.py` | Graph + embeddings → standalone `server.py` | All scripts require Python 3.10+ and auto-install their own dependencies on first run. --- ## Step 0: Clarify Requirements Before running anything, ask the user: - **URL**: Which site to crawl (required — starting page) - **Depth**: How many link-hops to follow (default 3; suggest 2 for large sites) - **Model**: Which Claude model for concept extraction — `haiku` (fast/cheap) or `sonnet` (higher quality). Default: `haiku` Set the output slug from the URL: `https://strudel.cc` → `strudel-cc-mcp`. --- ## Step 1: Crawl (M1) Provide this command for the user to run locally: ```bash python scripts/crawl.py \ --url <URL> \ --max-depth <DEPTH> \ --output ./output ``` **What to expect:** - Creates `output/<slug>/raw_content/*.json` (one per page) - Creates `output/<slug>/crawl.json` (state tracking) - Prints a summary: pages crawled, JS fallbacks used, failures **Common issues:** - JS-heavy single-page apps → many Playwright fallbacks (normal, just slower) - Rate limiting → add `--rate-limit 1.5` to slow down - First run needs: `pip install playwright && playwright install chromium` --- ## Step 2: Extract Concepts (M2) The user must set `ANTHROPIC_API_KEY` first. Provide this command: ```bash ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \ --input ./output/<slug>-mcp \ --model haiku ``` **Dry-run first (no API cost):** ```bash python scripts/extract_concepts.py --input ./output/<slug>-mcp --dry-run ``` This validates chunking quality before spending API budget. Show them chunk counts and section names from dry-run output. **What to expect:** - Processes ~2–5 pages/minute on haiku - Creates `output/<slug>-mcp/extracted/*.json` (one per page) - Each file contains chunks with: concepts, tags, code examples, prerequisites, relationships - Skips already-extracted pages (safe to re-run after interruption) **Common issues:** - Pages showing `no_chunks` → likely JS-rendered content not captured; acceptable for a minority of pages - API rate limiting → script retries automatically with exponential backoff - `--max-pages 10` flag to test on a small sample first **Re-running after a partial run:** ```bash python scripts/extract_concepts.py --input ./output/<slug>-mcp --model haiku # (automatically skips already-extracted pages) ``` **Force re-extraction of everything:** ```bash python scripts/extract_concepts.py --input ./output/<slug>-mcp --force ``` --- ## Step 3: Build Graph (M3) ```bash python scripts/build_graph.py --input ./output/<slug>-mcp ``` **What to expect:** - Reads all non-dry-run `extracted/*.json` files - Deduplicates concept names (case-insensitive, strips trailing `()`) - Creates `output/<slug>-mcp/graph.json` - Prints node/edge counts by type **Healthy output looks like:** ``` Pages: 46 Chunks: 357 Concepts: 200+ Total edges: 1000+ MENTIONS 600+ REQUIRES 100+ HAS_CHUNK 357 LINKS_TO 80+ RELATED 40+ ``` If concepts = 0 and "Skipped N dry-run files" appears, M2 hasn't been run with a real API key yet. --- ## Step 4: Build Embeddings (M4) ```bash python scripts/build_embeddings.py --input ./output/<slug>-mcp ``` **First run downloads `all-MiniLM-L6-v2` (~80MB, cached after that).** Add `--smoke-test` to query both collections immediately after building: ```bash python scripts/build_embeddings.py --input ./output/<slug>-mcp --smoke-test ``` **What to expect:** - Creates `output/<slug>-mcp/embeddings/` with 5 numpy files (no database needed) - Two indexes: `chunks` (semantic search) and `concepts` (concept lookup) --- ## Step 5: Generate MCP Server (M5) ```bash python scripts/generate_mcp_server.py --input ./output/<slug>-mcp ``` **Outputs:** - `output/<slug>-mcp/server.py` — the runnable MCP server - `output/<slug>-mcp/mcp_config.json` — Claude Desktop config snippet **Install into Claude Desktop:** 1. Open `~/Library/Application Support/Claude/claude_desktop_config.json` 2. Merge the contents of `mcp_config.json` into the `"mcpServers"` key 3. Restart Claude Desktop 4. The server name (e.g., `strudel-cc`) appears in Claude's available tools **Test the server standalone:** ```bash python output/<slug>-mcp/server.py # Should print "Loading ... knowledge graph... Ready: N nodes, M edges" ``` --- ## The 8 MCP Tools Once installed, Claude can use these tools against the knowledge base: | Tool | Description | |------|-------------| | `search(query, n=5)` | Semantic search over all content chunks | | `get_concept(name)` | Concept details + chunks where it appears | | `get_related(concept, n=5)` | Related concepts via graph edges | | `get_learning_path(start, goal)` | Shortest concept path between topics | | `get_prerequisites(concept)` | What must be understood first | | `get_examples(concept)` | Code examples for a concept | | `list_concepts(tag?, limit=20)` | Browse all indexed concepts | | `get_page(url)` | All chunks for a specific doc page | --- ## Complete Pipeline Command Sequence For a fresh install, provide the user with all commands in order: ```bash # 0. Install system deps (once) pip install requests beautifulsoup4 lxml playwright anthropic \ networkx numpy sentence-transformers mcp playwright install chromium # 1. Crawl python scripts/crawl.py --url <URL> --max-depth 3 --output ./output # 2. Extract concepts (dry-run first) python scripts/extract_concepts.py --input ./output/<slug>-mcp --dry-run # Then real run: ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \ --input ./output/<slug>-mcp --model haiku # 3. Build graph python scripts/build_graph.py --input ./output/<slug>-mcp # 4. Build embeddings python scripts/build_embeddings.py --input ./output/<slug>-mcp --smoke-test # 5. Generate server python scripts/generate_mcp_server.py --input ./output/<slug>-mcp # 6. Test server python output/<slug>-mcp/server.py ``` --- ## Output Directory Layout ``` output/<slug>-mcp/ ├── crawl.json State tracking (incremental re-runs) ├── raw_content/ One JSON per crawled page (HTML + links) ├── extracted/ One JSON per page (chunks + LLM concepts) ├── graph.json networkx knowledge graph ├── embeddings/ numpy indexes (chunks.npy, concepts.npy + JSON) ├── server.py The runnable MCP server ← share this └── mcp_config.json Claude Desktop config snippet ← install this ``` The entire `output/<slug>-mcp/` folder is the deliverable. The user can move it anywhere as long as `server.py`, `graph.json`, and `embeddings/` stay together. --- ## Troubleshooting **"No module named X"** → The script auto-installs deps, but if it fails: ```bash pip install <package> --break-system-packages ``` **Crawl gets 0 pages** → Check robots.txt and try `--force` to bypass the crawl cache. **extract_concepts produces tiny concepts count** → The page content may be JS-only. Check `fetched_with` field in `raw_content/*.json` — pages fetched via `requests` with very little text should have been picked up by Playwright. Re-crawl with `--force`. **Server fails to start** → Run `python output/<slug>-mcp/server.py` directly and check stderr for import errors. Most common cause: `mcp` package not installed. **Claude Desktop doesn't show the server** → Verify the path in `mcp_config.json` is absolute and the file exists. Restart Claude Desktop after any config change. --- ## Deferred Features See `TODO.md` for planned improvements including: - YouTube transcript fetching - Neo4j export for large graphs - OpenAI/Voyage embedding API support - Scheduled re-crawls - Graph visualization

path: LICENSE.md
content: MIT License

Copyright (c) 2026 GraphRAG Builder Skill contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


path: .env.example.txt
content: ANTHROPIC_API_KEY=sk-ant-...

path: requirements.txt
content: # GraphRAG Builder — Python dependencies
# Install with: pip install -r requirements.txt --break-system-packages

# --- M1: Crawler ---
requests>=2.31.0
beautifulsoup4>=4.12.0
lxml>=4.9.0
playwright>=1.40.0          # Playwright install chromium also required (run once)

# --- M2: Content Extraction ---
html2text>=2020.1.16
youtube-transcript-api>=0.6.2
pdfminer.six>=20221105

# --- M3: Graph ---
networkx>=3.2.0

# --- M4: Embeddings ---
# Note: chromadb is NOT used — pure numpy index, no database dependency.
# numpy is already a transitive dep of sentence-transformers.
numpy>=1.24.0
sentence-transformers>=2.3.0

# --- M5: MCP Server ---
mcp>=1.0.0
anthropic>=0.40.0           # For concept extraction (M2 Phase 3)


path: IDEA.md
content: Hi

Help me build a Claude skill that 

1) Takes an input website
2) Scrapes learning materials (docs, courses, youtube video transcriptions)
3) Organizes into graph using 
    i) hyperlinks of site map
    ii) related content tags (can be LLM generated)
    ?) use Neo4J for this ?
4) Spits out MCP server for said topic


First Example:
https://strudel.cc/workshop/getting-started/


path: TODO.md
content: # TODO — Deferred Features

Features explicitly deferred from the initial build. Revisit after M5 (working MCP server) is complete.

---

## High Priority

### YouTube Transcript Fetching (M1/M2 revision)

`crawl.py` already detects and stores YouTube video IDs in the `youtube_ids` field of each raw page JSON. What's missing is actually fetching and indexing those transcripts in `extract_concepts.py`.

**What to add to M2 (`extract_concepts.py`):**
- After loading a raw page JSON, check for non-empty `youtube_ids`
- For each ID, call `YouTubeTranscriptApi.get_transcript(video_id)` (already in `requirements.txt` as `youtube-transcript-api`)
- Concatenate transcript segments into text, then chunk at ~500-word boundaries (reuse `_force_split`)
- Each transcript chunk becomes an additional entry in the page's `chunks` list, with `section = "YouTube Transcript"` and `url` set to `https://youtube.com/watch?v=<id>`
- Cache raw transcripts to `raw_content/<url_hash>_yt_<video_id>.json` so re-runs skip the API call

**Edge cases to handle:**
- `TranscriptsDisabled` — log and skip, don't crash
- `NoTranscriptFound` — try `find_generated_transcript(['en'])` as fallback
- Auto-generated vs. manually uploaded transcripts — prefer manual, note which was used in output JSON
- Rate limiting from YouTube — add `--yt-delay` flag (default 1.0s) between transcript fetches

**Why deferred:** Strudel's site has no YouTube embeds (uses live REPL). Implement when testing against a site that links to tutorial videos (e.g., a course site or library with video walkthroughs).

Relevant phase: Phase 2/3 (M2 `extract_concepts.py`)

---

### Neo4j Export
For knowledge graphs exceeding ~10,000 nodes, networkx + JSON becomes slow for complex multi-hop queries. Add an optional `--neo4j` flag to `build.py` that generates a `.cypher` file of `CREATE` statements for bulk import, or writes directly to a running Neo4j instance via the `neo4j` Python driver.

Relevant phase: Phase 4 (Graph Construction)

---

### Embedding API Support
Currently the skill uses local `sentence-transformers` (`all-MiniLM-L6-v2`). Some users may prefer API-based embeddings for better quality or to avoid the ~80MB model download. Add a `--embedding-model` flag with options:

- `local` (default) — `sentence-transformers`, no API key required
- `openai` — `text-embedding-3-small` via OpenAI API
- `voyage` — Anthropic's recommended `voyage-3` via Voyage AI API

Abstract the embedding step behind an interface so swapping providers requires no other changes.

Relevant phase: Phase 5 (Embedding Index), Phase 0 (dependency management)

---

## Medium Priority

### Incremental Graph Merging
Currently, on incremental re-run, only changed pages are re-processed — but their concepts and edges are completely replaced in the graph. This can leave orphaned nodes (concepts that no longer appear anywhere) or miss merges where the same concept name was extracted from multiple pages.

Add a graph merge pass after Phase 4 that:
1. Deduplicates concept nodes by normalized name (case-insensitive, lemmatized)
2. Removes nodes with no incoming or outgoing edges
3. Logs a diff of added/removed nodes between runs

---

### Scheduled Re-Crawls
Add a `--schedule` option (e.g., `--schedule daily`) that writes a cron entry or launchd plist to re-run `build.py` automatically for a given topic. The incremental mode means re-runs are cheap if the site hasn't changed.

---

### Graph Visualization
Generate a static HTML visualization of the knowledge graph (e.g., using `pyvis` or `d3.js` via a template) alongside the MCP server output. Useful for inspecting what the crawler found and debugging poor-quality concept extraction.

Output file: `output/<slug>/graph_viz.html`

---

## Lower Priority

### Additional Content Sources
- **GitHub repositories**: Crawl `README.md`, `/docs/`, and inline docstrings
- **Confluence / Notion**: Add authenticated crawl support via their APIs
- **Discord / Slack**: Index pinned messages and thread summaries (with user permission)
- **RSS/Atom feeds**: Watch for new content and trigger incremental updates

### Multi-Language Support
`youtube-transcript-api` supports auto-translated transcripts. Add a `--language` flag that fetches transcripts in the specified language and passes it as context to the LLM extractor.

### MCP Server Authentication
The generated `server.py` currently has no auth. For sharing a server with a team (rather than running it locally), add optional API key authentication via an environment variable.

### Chunk Quality Scoring
After LLM extraction, score each chunk by concept density (concepts extracted / word count). Surface low-density chunks to the user as candidates for manual review or exclusion. High-density chunks get priority in search ranking.

### Export to Other Formats
- **Obsidian vault**: Each concept becomes a note, with wiki-links for relationships
- **Anki flashcards**: Concept + definition → front/back flashcard export
- **JSON-LD / schema.org**: Structured data export for integration with other tools


path: README.md
content: # GraphRAG Builder Skill

Turn any documentation site into a runnable **MCP (Model Context Protocol) knowledge server** that Claude can query with semantic search and graph traversal.

Packaged as a [Claude Skill](https://www.anthropic.com/news/skills) — invoke it with prompts like *"build an MCP server from https://strudel.cc/workshop/"* and it walks you through a 5-stage pipeline that crawls the site, extracts concepts with an LLM, builds a knowledge graph, generates embeddings, and emits a standalone `server.py` you can drop into Claude Desktop.

First target site: [strudel.cc](https://strudel.cc/workshop/getting-started/) — a live-coding music library.

---

## Pipeline

| Stage | Script                          | Output                                    |
|-------|---------------------------------|-------------------------------------------|
| M1    | `scripts/crawl.py`              | `raw_content/*.json` — HTML + links/metadata per page |
| M2    | `scripts/extract_concepts.py`   | `extracted/*.json` — chunks + LLM-extracted concepts, tags, code, prerequisites |
| M3    | `scripts/build_graph.py`        | `graph.json` — networkx knowledge graph (concepts + relationships) |
| M4    | `scripts/build_embeddings.py`   | `embeddings/` — numpy vector indexes (chunks + concepts) |
| M5    | `scripts/generate_mcp_server.py`| `server.py` + `mcp_config.json` — runnable MCP server |

The final `output/<slug>-mcp/` folder is the deliverable — move `server.py`, `graph.json`, and `embeddings/` anywhere together.

---

## Quick Start

```bash
# 0. Install deps (once)
pip install -r requirements.txt
playwright install chromium

# 1. Crawl
python scripts/crawl.py --url https://strudel.cc/workshop/getting-started/ \
  --max-depth 3 --output ./output

# 2. Extract concepts (dry-run first, no API cost)
python scripts/extract_concepts.py --input ./output/strudel-cc-mcp --dry-run
ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \
  --input ./output/strudel-cc-mcp --model haiku

# 3. Build graph
python scripts/build_graph.py --input ./output/strudel-cc-mcp

# 4. Build embeddings
python scripts/build_embeddings.py --input ./output/strudel-cc-mcp --smoke-test

# 5. Generate MCP server
python scripts/generate_mcp_server.py --input ./output/strudel-cc-mcp
```

Install into Claude Desktop by merging `output/strudel-cc-mcp/mcp_config.json` into `~/Library/Application Support/Claude/claude_desktop_config.json` under `"mcpServers"` and restarting Claude.

See [`SKILL.md`](./SKILL.md) for the full step-by-step skill instructions Claude follows, including troubleshooting, incremental re-runs, and dry-run validation.

---

## The 8 MCP Tools

Once installed, Claude can call these tools against the generated knowledge base:

| Tool | Description |
|------|-------------|
| `search(query, n=5)` | Semantic search over all content chunks |
| `get_concept(name)` | Concept details + chunks where it appears |
| `get_related(concept, n=5)` | Related concepts via graph edges |
| `get_learning_path(start, goal)` | Shortest concept path between topics |
| `get_prerequisites(concept)` | What must be understood first |
| `get_examples(concept)` | Code examples for a concept |
| `list_concepts(tag?, limit=20)` | Browse all indexed concepts |
| `get_page(url)` | All chunks for a specific doc page |

---

## Architecture Notes

- **Crawler**: `requests` first, automatic Playwright/Chromium fallback for JS-rendered pages. Incremental via SHA256 content hashes in SQLite.
- **Concept extraction**: Claude Haiku by default (fast/cheap), Sonnet opt-in for complex technical content. Chunks at ~500 words with section-aware splitting.
- **Graph**: `networkx` + JSON. Edge types: `MENTIONS`, `REQUIRES`, `HAS_CHUNK`, `LINKS_TO`, `RELATED`. Neo4j export planned for >10k nodes.
- **Embeddings**: Local `sentence-transformers` (`all-MiniLM-L6-v2`, ~80MB) — no API key, no database. Pure numpy index.
- **MCP server**: Self-contained generated Python file, reads the graph + embeddings from disk at startup.

---

## Project Layout

```
.
├── SKILL.md                  Claude Skill definition (the orchestration prompt)
├── scripts/
│   ├── crawl.py              M1: BFS crawler with JS fallback
│   ├── extract_concepts.py   M2: Chunking + LLM concept extraction
│   ├── build_graph.py        M3: networkx knowledge graph builder
│   ├── build_embeddings.py   M4: sentence-transformers index
│   └── generate_mcp_server.py M5: Self-contained MCP server emitter
├── evals/evals.json          Eval queries for checking server quality
├── references/               Reference materials (sample docs, etc.)
├── examples/                 Example outputs
├── output/                   Generated MCP packages (gitignored)
├── requirements.txt
├── IDEA.md                   Original project pitch
├── PLAN.md                   Architecture plan
└── TODO.md                   Deferred features & roadmap
```

---

## Status & Roadmap

Implemented end-to-end for text-based documentation sites. Verified on `strudel.cc`.

Deferred (see [`TODO.md`](./TODO.md)):

- **YouTube transcript ingestion** — `crawl.py` already collects video IDs in `youtube_ids`; M2 needs the fetch + chunk step using `youtube-transcript-api`.
- **Neo4j export** — for graphs exceeding ~10k nodes.
- **API-based embeddings** — optional OpenAI / Voyage backends.
- **Incremental graph merging** with orphan cleanup.
- **Graph visualization** — static HTML via `pyvis`.
- **Additional sources** — GitHub repos, Confluence/Notion, RSS feeds.

---

## Requirements

- Python 3.10+
- `ANTHROPIC_API_KEY` (only needed for M2 concept extraction)
- ~200MB disk for the sentence-transformers model cache (first run)

---

## License

MIT — see below. Use, modify, and redistribute freely.


path: PLAN.md
content: # GraphRAG Builder Skill — Project Plan

## Core Idea

Build a **Claude Cowork Skill** that takes a website (documentation, tutorial site, or course) and:

1. Crawls and extracts all learning content (pages, YouTube transcripts, PDFs)
2. Organizes that content into a **knowledge graph** — linked by hyperlink structure and LLM-generated concept relationships
3. Embeds the content for semantic search
4. Outputs a fully runnable **MCP (Model Context Protocol) server** that lets Claude answer expert questions about the topic

The first target: `https://strudel.cc/workshop/getting-started/` — a live-coding music library.

---

## CLI Interface

The skill is driven by a single orchestrator script. All options have sensible defaults.

```bash
python scripts/build.py \
  --url https://strudel.cc/workshop/getting-started/ \
  --max-depth 3 \
  --model haiku \
  --output ./output
```

| Flag | Default | Description |
|------|---------|-------------|
| `--url` | *(required)* | Seed URL to start crawling from |
| `--max-depth` | `3` | How many link-hops from the seed URL |
| `--model` | `haiku` | LLM model for concept extraction. Options: `haiku`, `sonnet` |
| `--output` | `./output` | Root output directory |
| `--incremental` | `true` | Skip URLs whose content hash hasn't changed |
| `--force` | `false` | Ignore checksums; re-crawl and re-process everything |

The `--model` flag controls concept extraction (Phase 3). `haiku` is fast and cheap for most sites; use `sonnet` for better relationship quality on complex technical content.

---

## Architecture Decisions

| # | Decision | Rationale |
|---|----------|-----------|
| 1 | **requests-html first, Playwright fallback** | If the returned page is empty or contains "JavaScript is required", automatically re-fetch with Playwright/Chromium |
| 2 | **Default model: haiku; `--model sonnet` option** | Cost-effective default; sonnet available for precision work |
| 3 | **sentence-transformers (local)** | No API key required; `all-MiniLM-L6-v2` is fast and good quality. See TODO.md for future embedding API support |
| 4 | **Neo4j deferred** | Sufficient for <10,000 nodes with networkx + JSON. See TODO.md |
| 5 | **Incremental updates via SQLite + SHA256** | Per-URL content hashing enables efficient re-runs without full re-crawl |
| 6 | **`/output` folder, gitignored** | Full MCP package (server.py + graph.json + embeddings) lives here |

---

## Feasibility Assessment

**Yes, this is feasible.** All core pieces have well-maintained Python libraries:

| Task | Tool |
|------|------|
| Web crawling (static) | `requests-html` |
| Web crawling (JS-heavy) | `playwright` (auto-fallback) |
| YouTube transcripts | `youtube-transcript-api` |
| LLM concept extraction | Claude API (haiku/sonnet) |
| Incremental tracking | `sqlite3` (stdlib) |
| Graph construction | `networkx` → JSON |
| Semantic search index | `chromadb` + `sentence-transformers` |
| MCP server | Python `mcp` SDK (FastMCP) |

---

## Repository Structure

```
GraphRAGBuilderSkill/
├── SKILL.md                       ← Cowork skill instructions
├── PLAN.md                        ← This file
├── TODO.md                        ← Deferred features
├── .gitignore                     ← Includes /output/
├── scripts/
│   ├── build.py                   ← CLI orchestrator (main entry point)
│   ├── crawl.py                   ← Web crawler + JS fallback
│   ├── extract_youtube.py         ← YouTube transcript fetcher
│   ├── extract_concepts.py        ← LLM concept/tag extractor
│   ├── build_graph.py             ← Graph assembler
│   ├── build_embeddings.py        ← Chunk + embed content
│   └── generate_mcp_server.py     ← Output MCP server generator
├── references/
│   ├── graph_schema.md            ← Node/edge type documentation
│   └── mcp_server_template.py     ← Template used by generate_mcp_server.py
├── evals/
│   └── evals.json                 ← Test cases
└── output/                        ← Gitignored; generated MCP packages live here
    └── strudel-mcp/
        ├── server.py
        ├── graph.json
        ├── crawl.db
        ├── embeddings/
        ├── requirements.txt
        └── claude_config.json
```

`.gitignore` must include:
```
/output/
__pycache__/
*.pyc
.playwright/
```

---

## Phase-by-Phase Plan

### Phase 0 — Environment Setup

Install Python dependencies at runtime via the orchestrator:

```
requests-html
playwright (+ chromium via playwright install chromium)
youtube-transcript-api
pdfminer.six
networkx
chromadb
sentence-transformers
mcp
anthropic
```

The orchestrator checks for these at startup and installs any missing ones.

---

### Phase 1 — Content Discovery & Crawling

**Input**: `--url`, `--max-depth`

**Process**:
1. Attempt to fetch seed URL with `requests-html`
2. **JS detection**: If response body is empty or contains "JavaScript is required" / "enable JavaScript" → re-fetch with Playwright
3. Extract all internal links (same domain only) and enqueue them
4. Detect YouTube video embeds/links → enqueue separately
5. Detect PDF links → enqueue separately
6. BFS crawl to depth N with 0.5s polite delay between requests
7. Respect `robots.txt` (skip disallowed URLs with a warning)
8. Deduplicate URLs

**Incremental tracking** (see Phase 1.5): Before fetching, check SQLite for a stored hash. After fetching, compare. Skip processing if hash matches.

**Output**: `output/<slug>/raw_content/<url_hash>.json`
```json
{
  "url": "https://strudel.cc/workshop/getting-started/",
  "type": "html",
  "title": "Getting Started",
  "raw_html": "...",
  "outgoing_links": ["..."],
  "youtube_ids": ["abc123"],
  "fetched_with": "requests-html"
}
```

---

### Phase 1.5 — Incremental State Tracking (SQLite)

All state lives in `output/<slug>/crawl.db`. Schema:

```sql
CREATE TABLE pages (
  url           TEXT PRIMARY KEY,
  content_hash  TEXT,          -- SHA256 of raw content
  last_crawled  TIMESTAMP,
  title         TEXT,
  type          TEXT,          -- 'html' | 'youtube' | 'pdf'
  status        TEXT           -- 'crawled' | 'extracted' | 'embedded' | 'graphed' | 'complete'
);

CREATE TABLE concepts (
  id            TEXT PRIMARY KEY,
  name          TEXT,
  definition    TEXT,
  tags          TEXT,          -- JSON array
  last_updated  TIMESTAMP
);
```

**Re-run logic**:
- `--incremental` (default): Fetch each URL, compute SHA256 of the response body. If hash matches `crawl.db`, mark as skipped. If hash differs or URL is new, set status back to `'crawled'` and re-run downstream phases for that URL only.
- `--force`: Ignore all stored hashes; re-process everything from scratch.

**Status progression**: `crawled` → `extracted` → `embedded` → `graphed` → `complete`

Each phase only processes pages at or below the required status, making partial re-runs safe.

---

### Phase 2 — Content Extraction & Cleaning

**Process**:
1. For HTML pages: strip nav/footer boilerplate, extract main content as clean Markdown using `html2text`
2. For YouTube IDs: fetch transcript via `youtube-transcript-api`, merge segments into timestamped text blocks
3. For PDFs: extract text with `pdfminer.six`
4. **Chunk** long content into 300–500 word segments with 50-word overlap, preserving section headings in each chunk's metadata

**Output**: `output/<slug>/extracted/<url_hash>.json`
```json
{
  "url": "...",
  "title": "...",
  "type": "html",
  "chunks": [
    {
      "id": "abc123_chunk_001",
      "text": "...",
      "section": "Installation",
      "word_count": 342
    }
  ]
}
```

Pages update to status `extracted` in SQLite.

---

### Phase 3 — LLM Concept Extraction

**This is the core intelligence step.** For each chunk, call Claude (`--model` flag, default `haiku`) with a structured prompt to extract:

- **Concepts**: Named topics or skills taught in this chunk
- **Tags**: Broader category labels
- **Code examples**: Any code blocks with a 1-sentence description
- **Prerequisites**: Concepts that should be understood first
- **Relationships**: How concepts in this chunk relate to others

**Prompt pattern** (sent per chunk):
```
You are extracting structured knowledge from a documentation chunk.

Chunk title: {section}
Source URL: {url}
Content: {text}

Extract in JSON:
{
  "concepts": [{"name": "...", "definition": "..."}],
  "tags": ["..."],
  "code_examples": [{"code": "...", "description": "..."}],
  "prerequisites": ["..."],
  "relationships": [{"from": "...", "to": "...", "type": "requires|related_to|explains"}]
}
```

**Cost guidance**: For a 50-page site with ~3 chunks/page ≈ 150 API calls. With `haiku`, this is typically under $0.10. With `sonnet`, ~$1–2.

**Output**: `output/<slug>/concepts/<url_hash>.json` (structured extraction per chunk)

Pages update to status `extracted` (concepts done) in SQLite.

---

### Phase 4 — Graph Construction

**Graph Schema**:

*Node types:*

| Type | Fields | Example |
|------|--------|---------|
| `Page` | url, title, type, summary | `strudel.cc/workshop/intro` |
| `Concept` | name, definition, tags | `"mini notation"` |
| `Example` | code, description, language | `note('c3 e3')` |
| `Video` | youtube_id, title | `abc123` |

*Edge types:*

| Type | From → To | Meaning |
|------|-----------|---------|
| `links_to` | Page → Page | Hyperlink exists |
| `teaches` | Page → Concept | Page explains this concept |
| `requires` | Concept → Concept | Prerequisite relationship |
| `related_to` | Concept → Concept | Thematically related |
| `exemplifies` | Example → Concept | Code demonstrates concept |
| `appears_in` | Example → Page | Code found on this page |

**Implementation**: Built with `networkx`, serialized to `output/<slug>/graph.json`.

The graph JSON uses node-link format:
```json
{
  "directed": true,
  "nodes": [
    {"id": "concept:mini-notation", "type": "Concept", "name": "mini notation", "tags": ["syntax", "beginner"]}
  ],
  "links": [
    {"source": "page:workshop/intro", "target": "concept:mini-notation", "type": "teaches"}
  ]
}
```

Pages update to status `graphed` in SQLite.

---

### Phase 5 — Semantic Embedding Index

**Process**:
1. Collect all cleaned chunks from Phase 2
2. Generate embeddings using `sentence-transformers` with model `all-MiniLM-L6-v2`
3. Store in a persistent `chromadb` collection at `output/<slug>/embeddings/`
4. Each embedding is stored with metadata: `{url, title, section, chunk_id}`

On incremental re-run: only re-embed chunks from pages whose hash changed.

Pages update to status `embedded` in SQLite.

---

### Phase 6 — MCP Server Generation

The skill generates a self-contained Python MCP server. The server loads `graph.json` and the ChromaDB index at startup and exposes these tools to Claude:

| Tool | Signature | Description |
|------|-----------|-------------|
| `search` | `(query: str, top_k: int = 5)` | Semantic search across all content |
| `get_page` | `(url: str)` | Full content + metadata for a page |
| `list_concepts` | `()` | All concepts with brief definitions |
| `get_concept` | `(name: str)` | Concept detail + related pages + examples |
| `get_related` | `(concept: str, relation_type: str = None)` | Graph neighbors |
| `get_learning_path` | `(start: str, end: str)` | Shortest concept path (networkx BFS) |
| `get_examples` | `(concept: str)` | All code examples for a concept |
| `get_prerequisites` | `(concept: str)` | Ordered prerequisite chain |

**Output package** (`output/<slug>/`):
```
output/strudel-mcp/
├── server.py           ← MCP server (run with: python server.py)
├── graph.json          ← Serialized knowledge graph
├── embeddings/         ← ChromaDB vector store directory
├── crawl.db            ← Crawl state + incremental tracking
├── requirements.txt    ← pip install -r requirements.txt
└── claude_config.json  ← Paste this into claude_desktop_config.json
```

**`claude_config.json`** contains the ready-to-paste snippet:
```json
{
  "mcpServers": {
    "strudel-docs": {
      "command": "python",
      "args": ["/absolute/path/to/output/strudel-mcp/server.py"]
    }
  }
}
```

The entire `output/<slug>/` directory is the deployable package. All paths inside `server.py` are relative so the package is portable.

All pages update to status `complete` in SQLite.

---

### Phase 7 — Skill Packaging & Testing

1. Write `SKILL.md` with triggering instructions and step-by-step workflow
2. Create eval cases in `evals/evals.json`:
   - **Eval 1**: Strudel (`strudel.cc`) — JS-heavy, YouTube videos, creative coding docs
   - **Eval 2**: A static docs site (e.g., FastAPI docs) — simpler crawl, API reference
   - **Eval 3**: A single-page tutorial — tests shallow crawl and minimal graph
3. Run evals using the skill-creator workflow; iterate on SKILL.md
4. Package as `.skill` file

---

## Recommended Build Order (Milestones)

Each milestone is independently testable:

**M1 — Crawler**: `crawl.py` works on Strudel. Produces raw JSON files. SQLite tracking working. JS fallback triggered correctly.

**M2 — Extraction + Concepts**: `extract_concepts.py` produces structured JSON from Strudel pages. Chunks look right. LLM output is parseable.

**M3 — Graph**: `build_graph.py` produces a valid `graph.json` with reasonable nodes and edges. Can inspect it manually.

**M4 — Search**: `build_embeddings.py` works. Can run semantic queries against the ChromaDB from the command line and get relevant chunks back.

**M5 — MCP Server**: `generate_mcp_server.py` produces a `server.py` that starts cleanly. All 8 tools respond correctly. Claude Desktop can connect and answer Strudel questions.

**M6 — Skill + Evals**: Wrap in `SKILL.md`. Run evals. Iterate. Package.


path: scripts/crawl.py
content: #!/usr/bin/env python3
"""
crawl.py — Web Crawler for GraphRAG Builder

Fetches pages using requests + BeautifulSoup (fast, lightweight).
Automatically falls back to Playwright when a page requires JavaScript.
Tracks crawl state in SQLite for incremental re-runs.
Outputs one JSON file per page into output/<slug>/raw_content/.

Usage:
    python crawl.py --url https://strudel.cc/workshop/getting-started/
    python crawl.py --url https://fastapi.tiangolo.com/tutorial/ --max-depth 2
    python crawl.py --url https://docs.example.com --force
"""

import argparse
import hashlib
import json
import os
import re
import sqlite3
import sys
import time
import urllib.robotparser
from collections import deque
from datetime import datetime
from pathlib import Path
from urllib.parse import urljoin, urlparse, urlunparse

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------

def _ensure_deps():
    """Install missing dependencies at runtime."""
    deps = {
        "requests": "requests",
        "bs4": "beautifulsoup4",
        "lxml": "lxml",
    }
    missing = []
    for module, package in deps.items():
        try:
            __import__(module)
        except ImportError:
            missing.append(package)

    if missing:
        print(f"Installing missing packages: {', '.join(missing)}")
        os.system(f"pip install {' '.join(missing)} --break-system-packages -q")
        print()

_ensure_deps()

import requests
from bs4 import BeautifulSoup


# ---------------------------------------------------------------------------
# URL utilities
# ---------------------------------------------------------------------------

def url_to_slug(url: str) -> str:
    """Turn a URL's domain into a filesystem-safe slug. strudel.cc → strudel-cc"""
    domain = urlparse(url).netloc.lstrip("www.")
    return re.sub(r"[^a-zA-Z0-9]", "-", domain).strip("-")


def url_to_hash(url: str) -> str:
    """Short SHA256 of URL for use as a stable filename."""
    return hashlib.sha256(url.encode()).hexdigest()[:16]


def content_hash(text: str) -> str:
    """SHA256 of page content for change detection."""
    return hashlib.sha256(text.encode()).hexdigest()


def normalize_url(url: str) -> str:
    """
    Canonicalize a URL for deduplication:
    - Remove fragment (#section)
    - Strip trailing slash from non-root paths
    - Lowercase scheme + host
    """
    p = urlparse(url)
    path = p.path.rstrip("/") or "/"
    return urlunparse((p.scheme.lower(), p.netloc.lower(), path, p.params, p.query, ""))


def same_domain(url: str, base_domain: str) -> bool:
    """True if url belongs to base_domain (with or without www.)."""
    host = urlparse(url).netloc.lower()
    base = base_domain.lower()
    return host == base or host == f"www.{base}" or f"www.{host}" == base


def is_crawlable_url(url: str) -> bool:
    """Skip binary files, mailto, javascript:, etc."""
    p = urlparse(url)
    if p.scheme not in ("http", "https"):
        return False
    skip_exts = {
        # Documents handled via extract_pdf_links(), not as HTML pages
        ".pdf",
        # Binary / media
        ".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico",
        ".css", ".js", ".woff", ".woff2", ".ttf", ".eot",
        ".zip", ".tar", ".gz", ".mp3", ".mp4", ".webm", ".avi",
    }
    ext = os.path.splitext(p.path)[1].lower()
    return ext not in skip_exts


# ---------------------------------------------------------------------------
# Content extraction helpers
# ---------------------------------------------------------------------------

_JS_INDICATORS = [
    "javascript is required",
    "javascript must be enabled",
    "enable javascript",
    "requires javascript",
    "you need to enable javascript",
    "please enable javascript",
    "this page requires javascript",
]

def is_js_required(html: str) -> bool:
    """
    Heuristic: does this page require JavaScript to render its content?
    Two independent triggers — either one is sufficient:

    1. Keyword match: page explicitly says JS is required.
    2. Near-empty body: < 150 chars of visible text after stripping tags.
       React/Vue/SvelteKit apps typically render a single empty root div
       server-side, producing ~0-20 chars of visible text. Real documentation
       pages have at minimum a few sentences (150+ chars). Legitimate short
       pages (e.g. a one-line 404) may occasionally trigger this path, but
       Playwright will still render them correctly so false positives are safe.
    """
    soup = BeautifulSoup(html, "lxml")
    body = soup.find("body")
    text = body.get_text(separator=" ", strip=True) if body else ""
    if len(text) < 150:
        return True
    lower = html.lower()
    return any(ind in lower for ind in _JS_INDICATORS)


def extract_title(html: str) -> str:
    """Extract the best available title from HTML."""
    soup = BeautifulSoup(html, "lxml")
    if soup.title and soup.title.string:
        return soup.title.string.strip()
    h1 = soup.find("h1")
    if h1:
        return h1.get_text(strip=True)
    return ""


def extract_links(html: str, base_url: str) -> list[str]:
    """Extract all absolute <a href> links from HTML."""
    soup = BeautifulSoup(html, "lxml")
    links = []
    for tag in soup.find_all("a", href=True):
        href = tag["href"].strip()
        if not href or href.startswith("#"):
            continue
        full = urljoin(base_url, href)
        if is_crawlable_url(full):
            links.append(full)
    return links


def extract_youtube_ids(html: str) -> list[str]:
    """Extract YouTube video IDs from iframes, links, and embed tags."""
    patterns = [
        r"youtube\.com/embed/([A-Za-z0-9_-]{11})",
        r"youtube\.com/watch\?v=([A-Za-z0-9_-]{11})",
        r"youtu\.be/([A-Za-z0-9_-]{11})",
        r"youtube\.com/v/([A-Za-z0-9_-]{11})",
        r"youtube-nocookie\.com/embed/([A-Za-z0-9_-]{11})",
    ]
    ids = set()
    for pat in patterns:
        ids.update(re.findall(pat, html))
    return list(ids)


def extract_pdf_links(html: str, base_url: str) -> list[str]:
    """Extract links to PDF files."""
    soup = BeautifulSoup(html, "lxml")
    pdfs = []
    for tag in soup.find_all("a", href=True):
        href = tag["href"]
        if href.lower().endswith(".pdf"):
            pdfs.append(urljoin(base_url, href))
    return pdfs


# ---------------------------------------------------------------------------
# Fetch strategies
# ---------------------------------------------------------------------------

_REQUEST_HEADERS = {
    "User-Agent": (
        "GraphRAGBuilder/1.0 (open-source educational crawler; "
        "https://github.com/graphrag-builder)"
    ),
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


def fetch_static(url: str, session: requests.Session, timeout: int = 15) -> tuple[str | None, str]:
    """
    Fetch a URL with requests. Returns (html_content, status_message).
    status_message is "ok" on success, or an error description on failure.
    """
    try:
        resp = session.get(url, timeout=timeout, allow_redirects=True)
        resp.raise_for_status()
        content_type = resp.headers.get("content-type", "")
        if "text/html" not in content_type and "application/xhtml" not in content_type:
            return None, f"non-HTML content-type: {content_type}"
        return resp.text, "ok"
    except requests.exceptions.Timeout:
        return None, "timeout"
    except requests.exceptions.HTTPError as e:
        return None, f"HTTP {e.response.status_code}"
    except Exception as e:
        return None, str(e)


def fetch_playwright(url: str, wait_ms: int = 2000, timeout_ms: int = 30000) -> tuple[str | None, str]:
    """
    Fetch a URL using a headless Chromium browser via Playwright.
    Waits for network to settle (networkidle) before capturing HTML.
    Returns (html_content, status_message).
    """
    try:
        from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout
    except ImportError:
        print("     Installing playwright...")
        os.system("pip install playwright --break-system-packages -q")
        os.system("playwright install chromium --with-deps 2>&1 | tail -5")
        from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout

    try:
        with sync_playwright() as pw:
            browser = pw.chromium.launch(headless=True)
            context = browser.new_context(
                user_agent=_REQUEST_HEADERS["User-Agent"],
                java_script_enabled=True,
            )
            page = context.new_page()
            page.goto(url, timeout=timeout_ms, wait_until="networkidle")
            # Extra wait for dynamic rendering
            page.wait_for_timeout(wait_ms)
            html = page.content()
            browser.close()
            return html, "ok"
    except PWTimeout:
        return None, "playwright timeout"
    except Exception as e:
        return None, f"playwright error: {e}"


# ---------------------------------------------------------------------------
# SQLite state machine
# ---------------------------------------------------------------------------

class CrawlDB:
    """
    Tracks crawl state using an in-memory SQLite database, with JSON
    persistence to disk.

    Why in-memory? Some filesystems (Docker bind-mounts, certain Linux
    overlayfs configurations) reject SQLite's default journal file creation
    with a disk I/O error. Running in-memory avoids all locking/journaling
    issues. State is saved to <db_path>.json on every commit so it survives
    restarts.

    Status progression: crawled → extracted → embedded → graphed → complete
    """

    STATUSES = ("crawled", "extracted", "embedded", "graphed", "complete")

    def __init__(self, db_path: Path):
        # db_path is used as the base; we always write a .json sidecar
        # (never an actual .db file — see class docstring)
        self.json_path = db_path.with_suffix(".json")
        # Always in-memory; never write a .db file
        self.conn = sqlite3.connect(":memory:")
        self.conn.row_factory = sqlite3.Row
        self._init_schema()
        self._load_from_json()

    def _init_schema(self):
        self.conn.executescript("""
            CREATE TABLE IF NOT EXISTS pages (
                url           TEXT    PRIMARY KEY,
                content_hash  TEXT    NOT NULL DEFAULT '',
                last_crawled  TEXT    NOT NULL DEFAULT '',
                title         TEXT    DEFAULT '',
                type          TEXT    DEFAULT 'html',
                status        TEXT    DEFAULT 'crawled',
                fetched_with  TEXT    DEFAULT 'requests',
                depth         INTEGER DEFAULT 0
            );

            CREATE TABLE IF NOT EXISTS crawl_meta (
                key   TEXT PRIMARY KEY,
                value TEXT
            );
        """)
        self.conn.commit()

    # ---- JSON persistence ----

    def _load_from_json(self):
        """Restore state from the JSON sidecar file if it exists."""
        if not self.json_path.exists():
            return
        try:
            with open(self.json_path) as f:
                data = json.load(f)
            for row in data.get("pages", []):
                self.conn.execute("""
                    INSERT OR REPLACE INTO pages
                        (url, content_hash, last_crawled, title, type,
                         status, fetched_with, depth)
                    VALUES (?, ?, ?, ?, ?, ?, ?, ?)
                """, (row["url"], row["content_hash"], row["last_crawled"],
                      row["title"], row["type"], row["status"],
                      row["fetched_with"], row["depth"]))
            for row in data.get("meta", []):
                self.conn.execute(
                    "INSERT OR REPLACE INTO crawl_meta (key, value) VALUES (?, ?)",
                    (row["key"], row["value"])
                )
            self.conn.commit()
        except Exception as e:
            print(f"  ⚠  Could not load crawl state: {e}")

    def _save_to_json(self):
        """Persist current in-memory state to the JSON sidecar file."""
        pages = [dict(r) for r in self.conn.execute("SELECT * FROM pages").fetchall()]
        meta = [dict(r) for r in self.conn.execute("SELECT * FROM crawl_meta").fetchall()]
        state = {"pages": pages, "meta": meta}
        with open(self.json_path, "w") as f:
            json.dump(state, f, indent=2)

    # ---- Public API ----

    def get_page(self, url: str) -> sqlite3.Row | None:
        return self.conn.execute(
            "SELECT * FROM pages WHERE url = ?", (url,)
        ).fetchone()

    def upsert_page(self, *, url, content_hash, title, status,
                    fetched_with, depth, page_type="html"):
        self.conn.execute("""
            INSERT INTO pages
                (url, content_hash, last_crawled, title, type, status, fetched_with, depth)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT(url) DO UPDATE SET
                content_hash = excluded.content_hash,
                last_crawled = excluded.last_crawled,
                title        = excluded.title,
                type         = excluded.type,
                status       = excluded.status,
                fetched_with = excluded.fetched_with
        """, (url, content_hash, datetime.now().isoformat(),
              title, page_type, status, fetched_with, depth))
        self.conn.commit()
        self._save_to_json()

    def set_meta(self, key: str, value):
        self.conn.execute("""
            INSERT INTO crawl_meta (key, value) VALUES (?, ?)
            ON CONFLICT(key) DO UPDATE SET value = excluded.value
        """, (key, str(value)))
        self.conn.commit()
        self._save_to_json()

    def update_page_status(self, url: str, status: str):
        """Used by later phases (extract, embed, graph) to advance status."""
        self.conn.execute(
            "UPDATE pages SET status = ? WHERE url = ?", (status, url)
        )
        self.conn.commit()
        self._save_to_json()

    def pages_with_status(self, status: str) -> list[sqlite3.Row]:
        return self.conn.execute(
            "SELECT * FROM pages WHERE status = ?", (status,)
        ).fetchall()

    def summary(self) -> dict:
        rows = self.conn.execute(
            "SELECT status, COUNT(*) as n FROM pages GROUP BY status"
        ).fetchall()
        return {r["status"]: r["n"] for r in rows}

    def close(self):
        self._save_to_json()
        self.conn.close()


# ---------------------------------------------------------------------------
# Main crawl function
# ---------------------------------------------------------------------------

def crawl(
    seed_url: str,
    max_depth: int = 3,
    output_dir: str = "./output",
    incremental: bool = True,
    rate_limit: float = 0.5,
) -> Path:
    """
    BFS crawl from seed_url up to max_depth hops.
    Returns the output package path (e.g. ./output/strudel-cc-mcp/).
    """
    parsed = urlparse(seed_url)
    base_domain = parsed.netloc.lstrip("www.")
    slug = url_to_slug(seed_url)

    # --- Output directories ---
    output_path = Path(output_dir) / f"{slug}-mcp"
    raw_dir = output_path / "raw_content"
    raw_dir.mkdir(parents=True, exist_ok=True)

    # --- SQLite ---
    db = CrawlDB(output_path / "crawl.db")
    db.set_meta("seed_url", seed_url)
    db.set_meta("max_depth", max_depth)
    db.set_meta("crawl_started", datetime.now().isoformat())

    # --- robots.txt ---
    rp = urllib.robotparser.RobotFileParser()
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
    try:
        rp.set_url(robots_url)
        rp.read()
        print(f"  ✓  robots.txt loaded from {robots_url}")
    except Exception:
        rp = None
        print(f"  ⚠  robots.txt unavailable, proceeding without restrictions")

    # --- HTTP session ---
    session = requests.Session()
    session.headers.update(_REQUEST_HEADERS)

    # --- BFS state ---
    queue: deque[tuple[str, int]] = deque([(normalize_url(seed_url), 0)])
    visited: set[str] = set()
    stats = {
        "crawled": 0,
        "skipped_unchanged": 0,
        "skipped_robots": 0,
        "skipped_offsite": 0,
        "failed": 0,
        "js_fallbacks": 0,
    }

    print(f"\n{'━'*62}")
    print(f"  GraphRAG Builder — Crawler  (M1)")
    print(f"{'━'*62}")
    print(f"  Seed:    {seed_url}")
    print(f"  Domain:  {base_domain}")
    print(f"  Depth:   {max_depth}")
    print(f"  Output:  {output_path}")
    print(f"  Mode:    {'incremental (skip unchanged)' if incremental else 'full re-crawl'}")
    print(f"{'━'*62}\n")

    while queue:
        url, depth = queue.popleft()

        if url in visited:
            continue
        visited.add(url)

        if depth > max_depth:
            continue

        # Off-site check
        if not same_domain(url, base_domain):
            stats["skipped_offsite"] += 1
            continue

        # robots.txt
        if rp and not rp.can_fetch("*", url):
            print(f"  [robots]  {url}")
            stats["skipped_robots"] += 1
            continue

        print(f"  [{depth}/{max_depth}]  {url}")

        # --- Fetch ---
        html, status_msg = fetch_static(url, session)
        fetched_with = "requests"

        if html is None:
            print(f"           ✗  {status_msg}")
            stats["failed"] += 1
            continue

        # JS detection → Playwright fallback
        if is_js_required(html):
            print(f"           ↻  JS detected → Playwright")
            html, status_msg = fetch_playwright(url)
            fetched_with = "playwright"
            stats["js_fallbacks"] += 1
            if html is None:
                print(f"           ✗  Playwright failed: {status_msg}")
                stats["failed"] += 1
                continue

        # --- Incremental hash check ---
        c_hash = content_hash(html)
        if incremental:
            existing = db.get_page(url)
            if existing and existing["content_hash"] == c_hash and existing["status"] == "complete":
                print(f"           ✓  unchanged — skipping")
                stats["skipped_unchanged"] += 1
                # Still enqueue links from the stored page so we don't miss neighbours
                raw_file = raw_dir / f"{url_to_hash(url)}.json"
                if raw_file.exists():
                    with open(raw_file) as f:
                        data = json.load(f)
                    for link in data.get("outgoing_links", []):
                        norm = normalize_url(link)
                        if norm not in visited and depth < max_depth:
                            queue.append((norm, depth + 1))
                continue

        # --- Extract metadata ---
        title = extract_title(html)
        youtube_ids = extract_youtube_ids(html)
        pdf_links = extract_pdf_links(html, url)
        raw_links = extract_links(html, url)
        outgoing = list({normalize_url(l) for l in raw_links if same_domain(l, base_domain)})

        # --- Save raw JSON ---
        raw_file = raw_dir / f"{url_to_hash(url)}.json"
        page_data = {
            "url": url,
            "type": "html",
            "title": title,
            "raw_html": html,
            "outgoing_links": outgoing,
            "youtube_ids": youtube_ids,
            "pdf_links": pdf_links,
            "depth": depth,
            "fetched_with": fetched_with,
            "crawled_at": datetime.now().isoformat(),
        }
        with open(raw_file, "w", encoding="utf-8") as f:
            json.dump(page_data, f, ensure_ascii=False, indent=2)

        # --- Update SQLite ---
        db.upsert_page(
            url=url,
            content_hash=c_hash,
            title=title,
            status="crawled",
            fetched_with=fetched_with,
            depth=depth,
        )
        stats["crawled"] += 1

        print(
            f"           ✓  \"{title or '(no title)'}\"  "
            f"links={len(outgoing)}  yt={len(youtube_ids)}  pdf={len(pdf_links)}"
            f"  [{fetched_with}]"
        )

        # Enqueue internal links
        if depth < max_depth:
            for link in outgoing:
                norm = normalize_url(link)
                if norm not in visited:
                    queue.append((norm, depth + 1))

        time.sleep(rate_limit)

    # --- Wrap up ---
    db.set_meta("crawl_finished", datetime.now().isoformat())
    db.set_meta("pages_crawled_total", stats["crawled"])
    summary = db.summary()
    db.close()

    print(f"\n{'━'*62}")
    print(f"  Crawl complete!")
    print(f"  Pages crawled:       {stats['crawled']}")
    print(f"  Skipped (unchanged): {stats['skipped_unchanged']}")
    print(f"  Skipped (robots):    {stats['skipped_robots']}")
    print(f"  JS fallbacks:        {stats['js_fallbacks']}")
    print(f"  Failed:              {stats['failed']}")
    print(f"  DB status summary:   {summary}")
    print(f"  Output:              {output_path}/raw_content/")
    print(f"{'━'*62}\n")

    return output_path


# ---------------------------------------------------------------------------
# CLI entry point
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="GraphRAG Builder — Web Crawler (M1)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  python crawl.py --url https://strudel.cc/workshop/getting-started/
  python crawl.py --url https://fastapi.tiangolo.com/tutorial/ --max-depth 2
  python crawl.py --url https://docs.example.com --max-depth 4 --force
        """,
    )
    parser.add_argument("--url", required=True, help="Seed URL to start crawling from")
    parser.add_argument("--max-depth", type=int, default=3, metavar="N",
                        help="Max link-hops from seed URL (default: 3)")
    parser.add_argument("--output", default="./output", metavar="DIR",
                        help="Root output directory (default: ./output)")
    parser.add_argument("--force", action="store_true",
                        help="Ignore stored content hashes; re-crawl everything")
    parser.add_argument("--rate-limit", type=float, default=0.5, metavar="SEC",
                        help="Seconds between requests (default: 0.5)")

    args = parser.parse_args()

    crawl(
        seed_url=args.url,
        max_depth=args.max_depth,
        output_dir=args.output,
        incremental=not args.force,
        rate_limit=args.rate_limit,
    )


if __name__ == "__main__":
    main()


path: scripts/generate_mcp_server.py
content: #!/usr/bin/env python3
"""
generate_mcp_server.py — MCP Server Generator (M5)

Reads graph.json + embeddings/ from the output directory and writes a
fully self-contained MCP server (server.py) to that same directory.

The generated server exposes 8 tools:
  search            Semantic search over chunk content
  get_concept       Look up a concept and its related chunks
  get_related       Find related concepts via graph edges
  get_learning_path Shortest path between two concepts in the knowledge graph
  get_prerequisites List prerequisites for a concept
  get_examples      Find code examples for a concept
  list_concepts     List all indexed concepts (optionally filtered by tag)
  get_page          Return all chunks for a given page URL

Usage:
    python scripts/generate_mcp_server.py --input ./output/strudel-cc-mcp
    python scripts/generate_mcp_server.py --input ./output/strudel-cc-mcp \\
        --model all-MiniLM-L6-v2 --force
"""

import argparse
import json
import sys
from pathlib import Path

# ---------------------------------------------------------------------------
# Introspect the output directory to derive config values
# ---------------------------------------------------------------------------

def _infer_config(input_dir: Path, model: str) -> dict:
    """Return a config dict derived from graph.json + the --model flag."""
    graph_path = input_dir / "graph.json"
    if not graph_path.exists():
        print(f"WARNING: graph.json not found — run build_graph.py first", file=sys.stderr)
        stats = {"pages": 0, "chunks": 0, "concepts": 0}
    else:
        meta = json.loads(graph_path.read_text()).get("meta", {})
        stats = meta.get("stats", {})

    # Derive a human-readable topic from the folder name
    # e.g. "strudel-cc-mcp" → "strudel-cc"
    slug = input_dir.name
    if slug.endswith("-mcp"):
        slug = slug[:-4]
    topic = slug.replace("-", " ").replace("_", " ").title()

    return {
        "topic":           topic,
        "slug":            slug,
        "embedding_model": model,
        "stats":           stats,
    }


# ---------------------------------------------------------------------------
# Server template
# ---------------------------------------------------------------------------

_SERVER_TEMPLATE = '''\
#!/usr/bin/env python3
"""
MCP Knowledge Server: {topic}

Auto-generated by GraphRAGBuilderSkill.

This server exposes the following tools:
  search            Semantic search over all indexed content
  get_concept       Look up a concept + its related chunks
  get_related       Find related concepts via graph edges
  get_learning_path Shortest concept path between two topics
  get_prerequisites List prerequisite concepts
  get_examples      Find code examples for a concept
  list_concepts     List all indexed concepts (optionally filtered)
  get_page          Return all chunks for a given page URL

Add to Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json):

  "mcpServers": {{
    "{slug}": {{
      "command": "python",
      "args": ["/absolute/path/to/{slug}-mcp/server.py"]
    }}
  }}

Run standalone:
  python server.py
"""

import asyncio
import json
import os
import re
import sys
from pathlib import Path

# ---------------------------------------------------------------------------
# Configuration — paths are relative to this file\'s location
# ---------------------------------------------------------------------------

TOPIC = {topic!r}
EMBEDDING_MODEL = {embedding_model!r}
BASE_DIR = Path(__file__).parent.resolve()
GRAPH_JSON = BASE_DIR / "graph.json"
EMBEDDINGS_DIR = BASE_DIR / "embeddings"

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------

def _install(pkg):
    print(f"Installing {{pkg}}...", file=sys.stderr)
    os.system(f"pip install {{pkg}} --break-system-packages -q")

def _import_ok(mod):
    try:
        __import__(mod)
        return True
    except ImportError:
        return False

for mod, pkg in [("mcp", "mcp"), ("networkx", "networkx"),
                  ("numpy", "numpy"),
                  ("sentence_transformers", "sentence-transformers")]:
    if not _import_ok(mod):
        _install(pkg)

from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp import types
import networkx as nx
import numpy as np
from sentence_transformers import SentenceTransformer

# ---------------------------------------------------------------------------
# Load resources at startup
# ---------------------------------------------------------------------------

def _load_graph() -> nx.MultiDiGraph:
    if not GRAPH_JSON.exists():
        print(f"WARNING: {{GRAPH_JSON}} not found — graph tools will be limited",
              file=sys.stderr)
        return nx.MultiDiGraph()
    data = json.loads(GRAPH_JSON.read_text())
    return nx.node_link_graph(data, edges="edges")


def _load_embeddings():
    """Load numpy embedding index. Returns (model, chunk_matrix, chunk_index,
    concept_matrix, concept_index) or (None, ...) if not available."""
    meta_path = EMBEDDINGS_DIR / "meta.json"
    if not EMBEDDINGS_DIR.exists() or not meta_path.exists():
        print(f"WARNING: {{EMBEDDINGS_DIR}} not found — run build_embeddings.py first",
              file=sys.stderr)
        return None, None, None, None, None
    try:
        model         = SentenceTransformer(EMBEDDING_MODEL)
        chunk_matrix  = np.load(EMBEDDINGS_DIR / "chunks.npy")
        chunk_index   = json.loads((EMBEDDINGS_DIR / "chunks.json").read_text())
        concept_matrix = np.load(EMBEDDINGS_DIR / "concepts.npy")
        concept_index  = json.loads((EMBEDDINGS_DIR / "concepts.json").read_text())
        return model, chunk_matrix, chunk_index, concept_matrix, concept_index
    except Exception as e:
        print(f"WARNING: Could not load embeddings: {{e}}", file=sys.stderr)
        return None, None, None, None, None


def _vec_search(query: str, model, matrix, index, n: int = 5) -> list[tuple]:
    """Cosine similarity search. Returns [(meta_dict, score), ...]."""
    qv = model.encode([query], convert_to_numpy=True).astype(np.float32)
    qv = qv / np.maximum(np.linalg.norm(qv), 1e-8)
    scores = matrix @ qv[0]
    top = np.argsort(scores)[::-1][:n]
    return [(index[i], float(scores[i])) for i in top]


print(f"Loading {{TOPIC}} knowledge graph...", file=sys.stderr)
G = _load_graph()
_emb_model, _chunk_mat, _chunk_idx, _concept_mat, _concept_idx = _load_embeddings()
print(
    f"Ready: {{G.number_of_nodes()}} nodes, {{G.number_of_edges()}} edges",
    file=sys.stderr,
)

# ---------------------------------------------------------------------------
# Concept node helpers
# ---------------------------------------------------------------------------

def _norm(name: str) -> str:
    name = name.strip()
    name = re.sub(r\'\\(\\s*\\)$\', \'\', name)
    name = name.lower()
    name = re.sub(r\'[ _\\-]+\', \' \', name)
    return re.sub(r\'\\s+\', \' \', name).strip().rstrip(\'.,;:\')


def _concept_id(name: str) -> str:
    return f"concept:{{_norm(name)}}"


def _concept_data(name: str) -> dict | None:
    cid = _concept_id(name)
    if G.has_node(cid):
        return dict(G.nodes[cid])
    # Semantic fallback: find closest concept by embedding
    if _emb_model is not None and _concept_mat is not None:
        hits = _vec_search(name, _emb_model, _concept_mat, _concept_idx, n=1)
        if hits and hits[0][1] > 0.7:
            norm = hits[0][0].get("norm_name", "")
            node_id = f"concept:{{norm}}"
            if G.has_node(node_id):
                return dict(G.nodes[node_id])
    return None


def _chunks_for_concept(concept_id: str) -> list[dict]:
    """Return chunk node dicts that MENTION this concept."""
    chunks = []
    for pred, _, d in G.in_edges(concept_id, data=True):
        if d.get("type") in ("MENTIONS", "REQUIRES") and G.nodes[pred].get("type") == "chunk":
            chunks.append(dict(G.nodes[pred]))
    return chunks


# ---------------------------------------------------------------------------
# Tool implementations
# ---------------------------------------------------------------------------

def _tool_search(query: str, n_results: int = 5) -> list[dict]:
    if _emb_model is None or _chunk_mat is None:
        return [{{"error": "Embedding index not available. Run build_embeddings.py first."}}]
    results = []
    for meta, score in _vec_search(query, _emb_model, _chunk_mat, _chunk_idx, n=n_results):
        # Retrieve full text from graph node if available
        chunk_node = G.nodes.get(meta.get("id", ""), {{}})
        text = chunk_node.get("text", "")
        results.append({{
            "score":      round(score, 3),
            "section":    meta.get("section", ""),
            "page_title": meta.get("page_title", ""),
            "url":        meta.get("url", ""),
            "tags":       meta.get("tags", ""),
            "has_code":   meta.get("has_code", False),
            "text":       text[:500] + ("..." if len(text) > 500 else ""),
        }})
    return results


def _tool_get_concept(name: str) -> dict:
    data = _concept_data(name)
    if not data:
        return {{"error": f"Concept {{name!r}} not found. Try list_concepts() to browse."}}
    cid = f"concept:{{_norm(name)}}"
    chunks = _chunks_for_concept(cid)
    return {{
        "name":          data.get("name", name),
        "description":   data.get("description", ""),
        "mention_count": data.get("mention_count", 0),
        "chunks": [
            {{
                "section":  c.get("section", ""),
                "url":      c.get("url", ""),
                "text":     c.get("text", "")[:400],
                "has_code": bool(c.get("code_examples")),
            }}
            for c in chunks[:5]
        ],
    }}


def _tool_get_related(concept: str, n: int = 5) -> list[dict]:
    cid = _concept_id(concept)
    if not G.has_node(cid):
        return [{{"error": f"Concept {{concept!r}} not found."}}]
    related = []
    # Outgoing RELATED edges (concept → other_concept)
    for _, tgt, d in G.out_edges(cid, data=True):
        if d.get("type") == "RELATED" and G.nodes[tgt]["type"] == "concept":
            tgt_data = G.nodes[tgt]
            related.append({{
                "name":        tgt_data.get("name", tgt),
                "description": tgt_data.get("description", ""),
                "relation":    d.get("relation", "related_to"),
                "direction":   "→",
            }})
    # Incoming RELATED edges (other_concept → concept)
    for src, _, d in G.in_edges(cid, data=True):
        if d.get("type") == "RELATED" and G.nodes[src]["type"] == "concept":
            src_data = G.nodes[src]
            related.append({{
                "name":        src_data.get("name", src),
                "description": src_data.get("description", ""),
                "relation":    d.get("relation", "related_to"),
                "direction":   "←",
            }})
    return related[:n]


def _tool_get_learning_path(start: str, goal: str) -> dict:
    start_id = _concept_id(start)
    goal_id  = _concept_id(goal)
    if not G.has_node(start_id):
        return {{"error": f"Start concept {{start!r}} not found."}}
    if not G.has_node(goal_id):
        return {{"error": f"Goal concept {{goal!r}} not found."}}
    try:
        path = nx.shortest_path(G, start_id, goal_id)
        return {{
            "path": [G.nodes[n].get("name", n) for n in path],
            "length": len(path) - 1,
        }}
    except nx.NetworkXNoPath:
        return {{"error": f"No path found from {{start!r}} to {{goal!r}}."}}
    except nx.NodeNotFound as e:
        return {{"error": str(e)}}


def _tool_get_prerequisites(concept: str) -> list[str]:
    cid = _concept_id(concept)
    if not G.has_node(cid):
        return [f"Concept {{concept!r}} not found."]
    prereqs = set()
    # Find chunks that REQUIRE this concept
    for src, _, d in G.in_edges(cid, data=True):
        if d.get("type") == "REQUIRES" and G.nodes[src]["type"] == "chunk":
            # Also look at what OTHER concepts are required by the same chunks
            for _, tgt2, d2 in G.out_edges(src, data=True):
                if (d2.get("type") == "REQUIRES"
                        and tgt2 != cid
                        and G.nodes[tgt2]["type"] == "concept"):
                    prereqs.add(G.nodes[tgt2].get("name", tgt2))
    return sorted(prereqs)


def _tool_get_examples(concept: str) -> list[dict]:
    chunks = _chunks_for_concept(_concept_id(concept))
    examples = []
    for chunk in chunks:
        for ex in chunk.get("code_examples", []):
            examples.append({{
                "code":        ex.get("code", ""),
                "description": ex.get("description", ""),
                "source":      chunk.get("section", ""),
                "url":         chunk.get("url", ""),
            }})
    return examples[:10]


def _tool_list_concepts(tag: str = "", limit: int = 20) -> list[dict]:
    results = []
    for nid, data in G.nodes(data=True):
        if data.get("type") != "concept":
            continue
        # Tag filter: check chunks that mention this concept
        if tag:
            chunk_tags = []
            for chunk_data in _chunks_for_concept(nid):
                raw = chunk_data.get("tags", [])
                if isinstance(raw, str):
                    chunk_tags += [t.strip() for t in raw.split(",")]
                else:
                    chunk_tags += raw
            if tag.lower() not in [t.lower() for t in chunk_tags]:
                continue
        results.append({{
            "name":          data.get("name", nid),
            "description":   data.get("description", "")[:120],
            "mention_count": data.get("mention_count", 0),
        }})
    results.sort(key=lambda x: -x["mention_count"])
    return results[:limit]


def _tool_get_page(url: str) -> dict:
    page_id = f"page:{{url}}"
    if not G.has_node(page_id):
        # Try prefix match
        matches = [n for n in G.nodes if n.startswith("page:") and url in n]
        if not matches:
            return {{"error": f"Page {{url!r}} not found."}}
        page_id = matches[0]
    page_data = dict(G.nodes[page_id])
    chunks = []
    for _, chunk_id, d in G.out_edges(page_id, data=True):
        if d.get("type") == "HAS_CHUNK":
            cdata = dict(G.nodes[chunk_id])
            chunks.append({{
                "section":       cdata.get("section", ""),
                "text":          cdata.get("text", "")[:600],
                "word_count":    cdata.get("word_count", 0),
                "tags":          cdata.get("tags", []),
                "has_code":      bool(cdata.get("code_examples")),
            }})
    return {{
        "url":    page_data.get("url", url),
        "title":  page_data.get("title", ""),
        "chunks": chunks,
    }}


# ---------------------------------------------------------------------------
# MCP server definition
# ---------------------------------------------------------------------------

server = Server(f"{{TOPIC.lower().replace(' ', '-')}}-knowledge")


@server.list_tools()
async def list_tools() -> list[types.Tool]:
    return [
        types.Tool(
            name="search",
            description=(
                f"Semantic search over all {{TOPIC}} documentation and tutorials. "
                "Returns the most relevant content sections for a natural-language query."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "query":     {{"type": "string",  "description": "What to search for"}},
                    "n_results": {{"type": "integer", "description": "Max results (default 5)", "default": 5}},
                }},
                "required": ["query"],
            }},
        ),
        types.Tool(
            name="get_concept",
            description=(
                f"Look up a {{TOPIC}} concept by name. Returns its description, "
                "mention count, and the content chunks where it appears."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "name": {{"type": "string", "description": "Concept name to look up"}},
                }},
                "required": ["name"],
            }},
        ),
        types.Tool(
            name="get_related",
            description=(
                "Find concepts related to a given concept via the knowledge graph. "
                "Returns directed relationships extracted from the documentation."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "concept": {{"type": "string"}},
                    "n":       {{"type": "integer", "default": 5}},
                }},
                "required": ["concept"],
            }},
        ),
        types.Tool(
            name="get_learning_path",
            description=(
                "Find the shortest conceptual path between two topics. "
                "Useful for planning what to learn in order to understand a target concept."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "start": {{"type": "string", "description": "Starting concept"}},
                    "goal":  {{"type": "string", "description": "Target concept to reach"}},
                }},
                "required": ["start", "goal"],
            }},
        ),
        types.Tool(
            name="get_prerequisites",
            description="List concepts that are prerequisites for understanding a given concept.",
            inputSchema={{
                "type": "object",
                "properties": {{
                    "concept": {{"type": "string"}},
                }},
                "required": ["concept"],
            }},
        ),
        types.Tool(
            name="get_examples",
            description=(
                f"Find code examples related to a {{TOPIC}} concept. "
                "Returns runnable code snippets with descriptions."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "concept": {{"type": "string"}},
                }},
                "required": ["concept"],
            }},
        ),
        types.Tool(
            name="list_concepts",
            description=(
                f"List all indexed {{TOPIC}} concepts, sorted by how often they appear "
                "in the documentation. Optionally filter by topic tag."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "tag":   {{"type": "string",  "description": "Filter by tag (optional)"}},
                    "limit": {{"type": "integer", "description": "Max results (default 20)", "default": 20}},
                }},
            }},
        ),
        types.Tool(
            name="get_page",
            description=(
                f"Return all content chunks for a specific {{TOPIC}} documentation page, "
                "identified by URL."
            ),
            inputSchema={{
                "type": "object",
                "properties": {{
                    "url": {{"type": "string", "description": "Page URL"}},
                }},
                "required": ["url"],
            }},
        ),
    ]


@server.call_tool()
async def call_tool(
    name: str, arguments: dict
) -> list[types.TextContent]:
    result: object

    if name == "search":
        result = _tool_search(
            arguments["query"],
            int(arguments.get("n_results", 5)),
        )
    elif name == "get_concept":
        result = _tool_get_concept(arguments["name"])
    elif name == "get_related":
        result = _tool_get_related(
            arguments["concept"],
            int(arguments.get("n", 5)),
        )
    elif name == "get_learning_path":
        result = _tool_get_learning_path(arguments["start"], arguments["goal"])
    elif name == "get_prerequisites":
        result = _tool_get_prerequisites(arguments["concept"])
    elif name == "get_examples":
        result = _tool_get_examples(arguments["concept"])
    elif name == "list_concepts":
        result = _tool_list_concepts(
            arguments.get("tag", ""),
            int(arguments.get("limit", 20)),
        )
    elif name == "get_page":
        result = _tool_get_page(arguments["url"])
    else:
        result = {{"error": f"Unknown tool: {{name}}"}}

    return [types.TextContent(type="text", text=json.dumps(result, indent=2))]


# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------

async def _main():
    async with stdio_server() as (read_stream, write_stream):
        await server.run(
            read_stream,
            write_stream,
            server.create_initialization_options(),
        )


if __name__ == "__main__":
    asyncio.run(_main())
'''


# ---------------------------------------------------------------------------
# Claude Desktop config snippet
# ---------------------------------------------------------------------------

_CONFIG_TEMPLATE = """\
{{
  "mcpServers": {{
    "{slug}": {{
      "command": "{python_cmd}",
      "args": ["{server_path}"]
    }}
  }}
}}
"""


# ---------------------------------------------------------------------------
# Writer
# ---------------------------------------------------------------------------

def generate(input_dir: Path, model: str, force: bool = False):
    server_path = input_dir / "server.py"
    config_path = input_dir / "mcp_config.json"

    if server_path.exists() and not force:
        print(f"server.py already exists. Use --force to overwrite.")
        sys.exit(0)

    cfg = _infer_config(input_dir, model)

    # Fill template
    server_code = _SERVER_TEMPLATE.format(
        topic=cfg["topic"],
        slug=cfg["slug"],
        embedding_model=cfg["embedding_model"],
    )
    server_path.write_text(server_code)

    # Write config snippet — use sys.executable so the venv Python is baked in
    config_json = _CONFIG_TEMPLATE.format(
        slug=cfg["slug"],
        server_path=server_path.resolve(),
        python_cmd=sys.executable,
    )
    config_path.write_text(config_json)

    stats = cfg["stats"]
    print()
    print("━" * 62)
    print("  MCP server generated!")
    print(f"  Topic:   {cfg['topic']}")
    print(f"  Server:  {server_path}")
    print(f"  Config:  {config_path}")
    print()
    print(f"  Graph:   {stats.get('pages', '?')} pages, "
          f"{stats.get('chunks', '?')} chunks, "
          f"{stats.get('concepts', '?')} concepts")
    print()
    print("  Next steps:")
    print(f"  1. Add the contents of mcp_config.json to:")
    print(f"     ~/Library/Application Support/Claude/claude_desktop_config.json")
    print(f"  2. Restart Claude Desktop")
    print(f"  3. The '{cfg['slug']}' MCP server will appear in Claude's tool list")
    print("━" * 62)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Generate a self-contained MCP server from the graph + embeddings (M5)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument("--input", required=True, metavar="DIR",
                        help="MCP output dir (e.g. ./output/strudel-cc-mcp)")
    parser.add_argument("--model", default="all-MiniLM-L6-v2", metavar="MODEL",
                        help="Embedding model used in M4 (default: all-MiniLM-L6-v2)")
    parser.add_argument("--force", action="store_true",
                        help="Overwrite existing server.py")
    args = parser.parse_args()

    input_dir = Path(args.input)
    if not input_dir.exists():
        print(f"ERROR: Input directory not found: {input_dir}", file=sys.stderr)
        sys.exit(1)

    print(f"Generating MCP server for: {input_dir.name}")
    generate(input_dir, model=args.model, force=args.force)


if __name__ == "__main__":
    main()


path: scripts/build_graph.py
content: #!/usr/bin/env python3
"""
build_graph.py — Knowledge Graph Construction (M3)

Reads all extracted/<url_hash>.json files produced by extract_concepts.py
and assembles a directed knowledge graph:

  Node types:
    page     — one per crawled URL
    chunk    — one per content section (from a page)
    concept  — one per unique extracted concept (deduplicated)

  Edge types:
    HAS_CHUNK   page → chunk
    MENTIONS    chunk → concept  (concept appears in this chunk)
    REQUIRES    chunk → concept  (chunk lists this as a prerequisite)
    RELATED     concept → concept  (from LLM-extracted relationships)
    LINKS_TO    page → page  (hyperlinks from the crawl)

Outputs:
  graph.json       — serialized MultiDiGraph (networkx node-link format)

Usage:
    python scripts/build_graph.py --input ./output/strudel-cc-mcp
    python scripts/build_graph.py --input ./output/strudel-cc-mcp --force
"""

import argparse
import json
import os
import re
import sys
from datetime import datetime
from pathlib import Path

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------

def _ensure_deps():
    deps = {"networkx": "networkx"}
    missing = [pkg for mod, pkg in deps.items() if not _import_ok(mod)]
    if missing:
        print(f"Installing: {', '.join(missing)}")
        os.system(f"pip install {' '.join(missing)} --break-system-packages -q")

def _import_ok(mod):
    try:
        __import__(mod)
        return True
    except ImportError:
        return False

_ensure_deps()

import networkx as nx


# ---------------------------------------------------------------------------
# Concept normalization
# ---------------------------------------------------------------------------

def normalize_concept(name: str) -> str:
    """Normalize a concept name for deduplication.

    Examples:
        'Mini Notation'  → 'mini notation'
        'note()'         → 'note'
        'FM Synthesis'   → 'fm synthesis'
        '  pitch  '      → 'pitch'
    """
    name = name.strip()
    if not name:
        return ""
    name = re.sub(r'\(\s*\)$', '', name)   # strip trailing ()
    name = name.lower()
    name = re.sub(r'[_\-]+', ' ', name)    # hyphens/underscores → space
    name = re.sub(r'\s+', ' ', name)       # collapse whitespace
    name = name.rstrip('.,;:')
    return name.strip()


# ---------------------------------------------------------------------------
# Graph builder
# ---------------------------------------------------------------------------

class GraphBuilder:

    def __init__(self, input_dir: Path):
        self.input_dir = input_dir
        self.extracted_dir = input_dir / "extracted"
        self.raw_dir = input_dir / "raw_content"

        self.G = nx.MultiDiGraph()

        # Accumulators filled during load
        self._concept_meta: dict[str, dict] = {}     # norm → {name, desc, mention_count}
        self._chunk_mentions: list[tuple] = []        # (chunk_id, norm)
        self._chunk_requires: list[tuple] = []        # (chunk_id, norm)
        self._concept_rels: list[tuple] = []          # (src_norm, relation, tgt_norm, chunk_id)

    # ------------------------------------------------------------------
    # Phase 1: load extracted pages → page + chunk nodes
    # ------------------------------------------------------------------

    def _load_extracted(self) -> int:
        files = sorted(self.extracted_dir.glob("*.json"))
        if not files:
            print("  WARNING: No extracted JSON files found.",
                  "Run extract_concepts.py first.", file=sys.stderr)
            return 0

        loaded = skipped_dry = 0

        for f in files:
            data = json.loads(f.read_text())

            if data.get("model") == "dry-run":
                skipped_dry += 1
                continue

            url = data["url"]
            page_id = f"page:{url}"

            self.G.add_node(
                page_id,
                type="page",
                url=url,
                title=data.get("title", ""),
                processed_at=data.get("processed_at", ""),
            )

            for chunk in data.get("chunks", []):
                chunk_id = chunk["id"]

                self.G.add_node(
                    chunk_id,
                    type="chunk",
                    url=chunk["url"],
                    page_title=chunk.get("page_title", ""),
                    section=chunk.get("section", ""),
                    text=chunk.get("text", ""),
                    word_count=chunk.get("word_count", 0),
                    tags=chunk.get("tags", []),
                    code_examples=chunk.get("code_examples", []),
                )

                self.G.add_edge(page_id, chunk_id, type="HAS_CHUNK")

                # Collect concepts
                for concept in chunk.get("concepts", []):
                    cname = concept.get("name", "").strip()
                    norm = normalize_concept(cname)
                    if not norm:
                        continue
                    desc = concept.get("description", "")
                    if norm not in self._concept_meta:
                        self._concept_meta[norm] = {
                            "name": cname,
                            "description": desc,
                            "mention_count": 0,
                        }
                    else:
                        # Keep the longer description
                        if len(desc) > len(self._concept_meta[norm]["description"]):
                            self._concept_meta[norm]["description"] = desc
                        # Prefer title-cased canonical name
                        existing = self._concept_meta[norm]["name"]
                        if cname[0].isupper() and not existing[0].isupper():
                            self._concept_meta[norm]["name"] = cname
                    self._concept_meta[norm]["mention_count"] += 1
                    self._chunk_mentions.append((chunk_id, norm))

                # Collect prerequisites
                for prereq in chunk.get("prerequisites", []):
                    norm = normalize_concept(prereq)
                    if norm:
                        self._chunk_requires.append((chunk_id, norm))

                # Collect concept→concept relationships
                for rel in chunk.get("relationships", []):
                    src = normalize_concept(rel.get("source", ""))
                    tgt = normalize_concept(rel.get("target", ""))
                    relation = (rel.get("relation") or "related_to").strip() or "related_to"
                    if src and tgt:
                        self._concept_rels.append((src, relation, tgt, chunk_id))

            loaded += 1

        if skipped_dry:
            print(f"  Skipped {skipped_dry} dry-run file(s) — "
                  "re-run extract_concepts.py with a real API key to populate concepts")

        return loaded

    # ------------------------------------------------------------------
    # Phase 2: materialize concept nodes + all concept edges
    # ------------------------------------------------------------------

    def _build_concept_graph(self):
        # Create concept nodes
        for norm, meta in self._concept_meta.items():
            self.G.add_node(
                f"concept:{norm}",
                type="concept",
                name=meta["name"],
                norm_name=norm,
                description=meta["description"],
                mention_count=meta["mention_count"],
            )

        # chunk → concept (MENTIONS)
        for chunk_id, norm in self._chunk_mentions:
            cid = f"concept:{norm}"
            if self.G.has_node(cid):
                self.G.add_edge(chunk_id, cid, type="MENTIONS")

        # chunk → concept (REQUIRES) — create concept node if new
        for chunk_id, norm in self._chunk_requires:
            cid = f"concept:{norm}"
            if not self.G.has_node(cid):
                self.G.add_node(
                    cid,
                    type="concept",
                    name=norm,
                    norm_name=norm,
                    description="",
                    mention_count=0,
                )
            self.G.add_edge(chunk_id, cid, type="REQUIRES")

        # concept → concept (RELATED)
        for src_norm, relation, tgt_norm, chunk_id in self._concept_rels:
            src_id = f"concept:{src_norm}"
            tgt_id = f"concept:{tgt_norm}"
            for nid, nname in [(src_id, src_norm), (tgt_id, tgt_norm)]:
                if not self.G.has_node(nid):
                    self.G.add_node(
                        nid,
                        type="concept",
                        name=nname,
                        norm_name=nname,
                        description="",
                        mention_count=0,
                    )
            self.G.add_edge(
                src_id, tgt_id,
                type="RELATED",
                relation=relation,
                via_chunk=chunk_id,
            )

    # ------------------------------------------------------------------
    # Phase 3: page → page hyperlinks (from raw_content/*.json)
    # ------------------------------------------------------------------

    def _build_page_links(self):
        if not self.raw_dir.exists():
            return

        # Build url → page node id lookup
        url_to_nid = {
            data["url"]: nid
            for nid, data in self.G.nodes(data=True)
            if data.get("type") == "page"
        }

        for f in self.raw_dir.glob("*.json"):
            raw = json.loads(f.read_text())
            src_url = raw.get("url", "")
            src_id = url_to_nid.get(src_url)
            if not src_id:
                continue
            for link_url in raw.get("outgoing_links", []):
                tgt_id = url_to_nid.get(link_url)
                if tgt_id and tgt_id != src_id:
                    # Avoid duplicate hyperlink edges
                    existing = [
                        d for _, _, d in self.G.out_edges(src_id, data=True)
                        if d.get("type") == "LINKS_TO"
                        and self.G.nodes[_]["url"] == link_url
                        if False  # placeholder — use has_edge check below
                    ]
                    # nx.MultiDiGraph allows parallel edges; deduplicate via attribute
                    already = any(
                        d.get("type") == "LINKS_TO"
                        for _, t, d in self.G.out_edges(src_id, data=True)
                        if t == tgt_id
                    )
                    if not already:
                        self.G.add_edge(src_id, tgt_id, type="LINKS_TO")

    # ------------------------------------------------------------------
    # Main build
    # ------------------------------------------------------------------

    def build(self) -> dict:
        print("  Phase 1: Loading extracted pages...")
        n_pages = self._load_extracted()
        print(f"           {n_pages} page(s) with LLM data loaded")

        print("  Phase 2: Building concept nodes and edges...")
        self._build_concept_graph()

        print("  Phase 3: Building page hyperlink edges...")
        self._build_page_links()

        return self._stats()

    def _stats(self) -> dict:
        node_types: dict[str, int] = {}
        for _, d in self.G.nodes(data=True):
            t = d.get("type", "unknown")
            node_types[t] = node_types.get(t, 0) + 1

        edge_types: dict[str, int] = {}
        for _, _, d in self.G.edges(data=True):
            t = d.get("type", "unknown")
            edge_types[t] = edge_types.get(t, 0) + 1

        return {
            "pages":       node_types.get("page", 0),
            "chunks":      node_types.get("chunk", 0),
            "concepts":    node_types.get("concept", 0),
            "total_nodes": self.G.number_of_nodes(),
            "total_edges": self.G.number_of_edges(),
            "edge_types":  edge_types,
        }

    # ------------------------------------------------------------------
    # Serialization
    # ------------------------------------------------------------------

    def save(self, output_path: Path, stats: dict):
        data = nx.node_link_data(self.G, edges="edges")
        data["meta"] = {
            "created_at": datetime.utcnow().isoformat() + "Z",
            "stats": stats,
        }
        output_path.write_text(
            json.dumps(data, ensure_ascii=False, indent=2)
        )


# ---------------------------------------------------------------------------
# Output formatting
# ---------------------------------------------------------------------------

def _print_stats(stats: dict):
    print()
    print("━" * 62)
    print("  Graph built!")
    print(f"  Pages:        {stats['pages']:>6}")
    print(f"  Chunks:       {stats['chunks']:>6}")
    print(f"  Concepts:     {stats['concepts']:>6}")
    print(f"  ─────────────────────────")
    print(f"  Total nodes:  {stats['total_nodes']:>6}")
    print(f"  Total edges:  {stats['total_edges']:>6}")
    if stats["edge_types"]:
        print()
        print("  Edge breakdown:")
        for etype, count in sorted(stats["edge_types"].items(), key=lambda x: -x[1]):
            print(f"    {etype:<20} {count:>6}")
    print("━" * 62)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Build knowledge graph from extracted concept data (M3)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument("--input", required=True, metavar="DIR",
                        help="MCP output dir (e.g. ./output/strudel-cc-mcp)")
    parser.add_argument("--force", action="store_true",
                        help="Overwrite existing graph.json")
    args = parser.parse_args()

    input_dir = Path(args.input)
    if not input_dir.exists():
        print(f"ERROR: Input directory not found: {input_dir}", file=sys.stderr)
        sys.exit(1)

    output_path = input_dir / "graph.json"
    if output_path.exists() and not args.force:
        print(f"graph.json already exists. Use --force to overwrite.")
        sys.exit(0)

    print(f"Building knowledge graph")
    print(f"  Input:  {input_dir}")
    print(f"  Output: {output_path}")
    print()

    builder = GraphBuilder(input_dir)
    stats = builder.build()
    builder.save(output_path, stats)

    _print_stats(stats)
    print(f"  Saved: {output_path}")
    print()


if __name__ == "__main__":
    main()


path: scripts/build_embeddings.py
content: #!/usr/bin/env python3
"""
build_embeddings.py — Embedding Index Construction (M4)

Reads extracted/<url_hash>.json files and builds two numpy embedding indexes:

  chunks    — one vector per content chunk, for semantic search
  concepts  — one vector per unique concept, for concept lookup

Embeddings are generated with sentence-transformers (local, ~80MB download once).
Vectors are L2-normalized so cosine similarity = dot product.

Storage layout (in <input>/embeddings/):
  chunks.npy       float32 matrix (N_chunks × D)
  chunks.json      list of chunk metadata dicts
  concepts.npy     float32 matrix (N_concepts × D)
  concepts.json    list of concept metadata dicts
  meta.json        model name, dimensions, counts, created_at

No ChromaDB or other database — pure numpy, compatible with any Python version.

Usage:
    python scripts/build_embeddings.py --input ./output/strudel-cc-mcp
    python scripts/build_embeddings.py --input ./output/strudel-cc-mcp --force
    python scripts/build_embeddings.py --input ./output/strudel-cc-mcp --smoke-test
"""

import argparse
import json
import os
import re
import sys
from datetime import datetime
from pathlib import Path

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------

def _ensure_deps():
    deps = {"sentence_transformers": "sentence-transformers", "numpy": "numpy"}
    missing = [pkg for mod, pkg in deps.items() if not _import_ok(mod)]
    if missing:
        print(f"Installing: {', '.join(missing)}...")
        os.system(f"pip install {' '.join(missing)} --break-system-packages -q")

def _import_ok(mod):
    try:
        __import__(mod)
        return True
    except ImportError:
        return False

_ensure_deps()

import numpy as np
from sentence_transformers import SentenceTransformer


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def _normalize(name: str) -> str:
    name = name.strip()
    name = re.sub(r'\(\s*\)$', '', name)
    name = name.lower()
    name = re.sub(r'[_\-]+', ' ', name)
    name = re.sub(r'\s+', ' ', name).strip()
    return name.rstrip('.,;:')


def _chunk_doc_text(chunk: dict) -> str:
    section = chunk.get("section", "").strip()
    text = chunk.get("text", "").strip()
    if section and not text.startswith(section):
        return f"{section}\n\n{text}"
    return text


def _concept_doc_text(name: str, description: str) -> str:
    return f"{name}: {description}" if description else name


def _tags_to_str(tags) -> str:
    if isinstance(tags, list):
        return ", ".join(str(t) for t in tags if t)
    return str(tags) if tags else ""


# ---------------------------------------------------------------------------
# Load extracted data
# ---------------------------------------------------------------------------

def load_extracted(extracted_dir: Path) -> tuple[list[dict], list[dict]]:
    """Return (chunks, concepts) from all non-dry-run extracted JSON files."""
    chunks: list[dict] = []
    concept_map: dict[str, dict] = {}
    skipped = 0

    for f in sorted(extracted_dir.glob("*.json")):
        data = json.loads(f.read_text())
        if data.get("model") == "dry-run":
            skipped += 1
            continue

        page_url = data.get("url", "")
        for chunk in data.get("chunks", []):
            chunk = dict(chunk)
            chunk["page_url"] = page_url
            chunks.append(chunk)

            for concept in chunk.get("concepts", []):
                name = concept.get("name", "").strip()
                if not name:
                    continue
                norm = _normalize(name)
                desc = concept.get("description", "")
                if norm not in concept_map:
                    concept_map[norm] = {
                        "norm_name": norm,
                        "name": name,
                        "description": desc,
                        "mention_count": 0,
                    }
                else:
                    if len(desc) > len(concept_map[norm]["description"]):
                        concept_map[norm]["description"] = desc
                concept_map[norm]["mention_count"] += 1

    if skipped:
        print(f"  Skipped {skipped} dry-run file(s). "
              "Run extract_concepts.py with a real API key first.")

    return chunks, list(concept_map.values())


# ---------------------------------------------------------------------------
# Embed + save
# ---------------------------------------------------------------------------

def _l2_normalize(matrix: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(matrix, axis=1, keepdims=True)
    return matrix / np.maximum(norms, 1e-8)


def build_index(
    input_dir: Path,
    embedding_model: str = "all-MiniLM-L6-v2",
    force: bool = False,
) -> dict:
    extracted_dir = input_dir / "extracted"
    emb_dir = input_dir / "embeddings"
    emb_dir.mkdir(parents=True, exist_ok=True)

    meta_path      = emb_dir / "meta.json"
    chunk_npy      = emb_dir / "chunks.npy"
    chunk_json     = emb_dir / "chunks.json"
    concept_npy    = emb_dir / "concepts.npy"
    concept_json   = emb_dir / "concepts.json"

    if not force and meta_path.exists():
        print("  Embeddings already built. Use --force to rebuild.")
        meta = json.loads(meta_path.read_text())
        return meta

    # Load data
    print("  Loading extracted pages...")
    chunks, concepts = load_extracted(extracted_dir)
    print(f"  → {len(chunks)} chunks, {len(concepts)} concepts")

    if not chunks:
        print("  Nothing to embed. Run extract_concepts.py with a real API key first.")
        return {"chunks_indexed": 0, "concepts_indexed": 0}

    # Load model
    print(f"  Loading embedding model '{embedding_model}'...")
    print(f"  (First run downloads ~80MB — cached after that)")
    model = SentenceTransformer(embedding_model)

    # --- Embed chunks ---
    print(f"  Embedding {len(chunks)} chunks...", end="", flush=True)
    chunk_texts = [_chunk_doc_text(c) for c in chunks]
    chunk_matrix = model.encode(
        chunk_texts,
        batch_size=64,
        show_progress_bar=False,
        convert_to_numpy=True,
    ).astype(np.float32)
    chunk_matrix = _l2_normalize(chunk_matrix)
    np.save(chunk_npy, chunk_matrix)

    # Slim metadata (no raw text in index — server loads from graph.json)
    chunk_meta = [
        {
            "id":         c.get("id", ""),
            "url":        c.get("url", ""),
            "page_title": c.get("page_title", ""),
            "section":    c.get("section", ""),
            "word_count": c.get("word_count", 0),
            "tags":       _tags_to_str(c.get("tags", [])),
            "has_code":   bool(c.get("code_examples") or c.get("code_examples_raw")),
        }
        for c in chunks
    ]
    chunk_json.write_text(json.dumps(chunk_meta, ensure_ascii=False))
    print(f" done ({chunk_matrix.shape})")

    # --- Embed concepts ---
    print(f"  Embedding {len(concepts)} concepts...", end="", flush=True)
    concept_texts = [_concept_doc_text(c["name"], c["description"]) for c in concepts]
    concept_matrix = model.encode(
        concept_texts,
        batch_size=64,
        show_progress_bar=False,
        convert_to_numpy=True,
    ).astype(np.float32)
    concept_matrix = _l2_normalize(concept_matrix)
    np.save(concept_npy, concept_matrix)

    concept_json.write_text(
        json.dumps(concepts, ensure_ascii=False)
    )
    print(f" done ({concept_matrix.shape})")

    # --- Save meta ---
    meta = {
        "model":            embedding_model,
        "dimensions":       int(chunk_matrix.shape[1]),
        "chunks_indexed":   len(chunks),
        "concepts_indexed": len(concepts),
        "created_at":       datetime.utcnow().isoformat() + "Z",
    }
    meta_path.write_text(json.dumps(meta, indent=2))

    return meta


# ---------------------------------------------------------------------------
# Smoke test
# ---------------------------------------------------------------------------

def _smoke_test(input_dir: Path, model_name: str):
    emb_dir   = input_dir / "embeddings"
    meta_path = emb_dir / "meta.json"

    if not meta_path.exists():
        print("  No embeddings found — run without --smoke-test first.")
        return

    model      = SentenceTransformer(model_name)
    chunk_mat  = np.load(emb_dir / "chunks.npy")
    chunk_idx  = json.loads((emb_dir / "chunks.json").read_text())
    concept_mat = np.load(emb_dir / "concepts.npy")
    concept_idx = json.loads((emb_dir / "concepts.json").read_text())

    def search(query, matrix, index, n=3):
        qv = model.encode([query], convert_to_numpy=True).astype(np.float32)
        qv = qv / np.maximum(np.linalg.norm(qv), 1e-8)
        scores = (matrix @ qv[0])
        top = np.argsort(scores)[::-1][:n]
        return [(index[i], float(scores[i])) for i in top]

    print("\n── Smoke test ──────────────────────────────────────────")

    query = "how do I play a note pattern?"
    print(f"Chunk search: '{query}'")
    for meta, score in search(query, chunk_mat, chunk_idx):
        print(f"  [{score:.3f}] {meta['page_title']} / {meta['section']}")

    query2 = "musical rhythm and timing"
    print(f"\nConcept search: '{query2}'")
    for meta, score in search(query2, concept_mat, concept_idx):
        print(f"  [{score:.3f}] {meta['name']}")

    print("────────────────────────────────────────────────────────\n")


# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------

def _print_stats(stats: dict):
    print()
    print("━" * 62)
    print("  Embedding index built!")
    print(f"  Chunks indexed:   {stats.get('chunks_indexed', 0):>6}")
    print(f"  Concepts indexed: {stats.get('concepts_indexed', 0):>6}")
    print(f"  Dimensions:       {stats.get('dimensions', '?'):>6}")
    print(f"  Model:            {stats.get('model', '?')}")
    print("━" * 62)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Build numpy embedding index from extracted concepts (M4)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument("--input", required=True, metavar="DIR",
                        help="MCP output dir (e.g. ./output/strudel-cc-mcp)")
    parser.add_argument("--model", default="all-MiniLM-L6-v2", metavar="MODEL",
                        help="Sentence-transformers model name (default: all-MiniLM-L6-v2)")
    parser.add_argument("--force", action="store_true",
                        help="Rebuild existing index")
    parser.add_argument("--smoke-test", action="store_true",
                        help="Run test queries after building")
    args = parser.parse_args()

    input_dir = Path(args.input)
    if not input_dir.exists():
        print(f"ERROR: Input directory not found: {input_dir}", file=sys.stderr)
        sys.exit(1)

    print(f"Building embedding index")
    print(f"  Input:  {input_dir}")
    print(f"  Model:  {args.model}")
    print()

    stats = build_index(input_dir, embedding_model=args.model, force=args.force)
    _print_stats(stats)

    if args.smoke_test and stats.get("chunks_indexed", 0) > 0:
        _smoke_test(input_dir, args.model)


if __name__ == "__main__":
    main()


path: scripts/extract_concepts.py
content: #!/usr/bin/env python3
"""
extract_concepts.py — Content Extraction + LLM Concept Extraction (M2)

Two phases in one script:
  Phase 2: HTML → clean chunked text (split at h2 boundaries, code blocks
           extracted from <pre> tags alongside their section text)
  Phase 3: Each chunk → Claude API call → structured concepts, tags,
           prerequisites, relationships, code example descriptions

Usage:
    python extract_concepts.py --input ./output/strudel-cc-mcp
    python extract_concepts.py --input ./output/strudel-cc-mcp --model sonnet
    python extract_concepts.py --input ./output/strudel-cc-mcp --max-pages 5 --dry-run
"""

import argparse
import json
import os
import re
import sys
import time
from pathlib import Path
from datetime import datetime

# ---------------------------------------------------------------------------
# Dependency bootstrap
# ---------------------------------------------------------------------------

def _ensure_deps():
    deps = {"bs4": "beautifulsoup4", "lxml": "lxml", "anthropic": "anthropic"}
    missing = [pkg for mod, pkg in deps.items()
               if not __import_ok(mod)]
    if missing:
        print(f"Installing: {', '.join(missing)}")
        os.system(f"pip install {' '.join(missing)} --break-system-packages -q")

def __import_ok(mod):
    try:
        __import__(mod)
        return True
    except ImportError:
        return False

_ensure_deps()

from bs4 import BeautifulSoup, NavigableString, Tag
import anthropic


# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------

# Pages to skip regardless of content
SKIP_URL_PATTERNS = [
    "/de/",          # German translations
    "/fr/",          # French (future-proofing)
    "/bakery",       # Interactive gallery, no prose
    "/intro/showcase",  # Short gallery page
]

# Primary content selector — works for all Strudel docs pages
CONTENT_SELECTOR = ("article", {"class": "content"})

# Model shortcuts → full API strings
MODEL_MAP = {
    "haiku":  "claude-haiku-4-5-20251001",
    "sonnet": "claude-sonnet-4-5-20250929",
    "opus":   "claude-opus-4-5-20251101",
}

# Max words per chunk before we force-split a section
MAX_CHUNK_WORDS = 700

# Drop chunks with fewer than this many words (footers, stubs).
# 30 is intentionally low to preserve function-reference pages where
# each h2 section is a single function with a one-liner + code example.
MIN_CHUNK_WORDS = 30

# Max code blocks to include in the LLM prompt per chunk
MAX_CODES_IN_PROMPT = 6


# ---------------------------------------------------------------------------
# Phase 2: HTML → structured chunks
# ---------------------------------------------------------------------------

def extract_article(html: str) -> Tag | None:
    """Return the <article class='content'> element or None."""
    soup = BeautifulSoup(html, "lxml")
    return soup.find(*CONTENT_SELECTOR)


def _word_count(text: str) -> int:
    return len(text.split())


def split_into_chunks(article: Tag, url: str, page_title: str) -> list[dict]:
    """
    Strudel docs structure:
        <article class='content'>
          <section>            ← all prose (h1/h2/p/pre/ul...)
          <nav>                ← footer ("Edit this page") — skipped
        </article>

    We recursively walk the <section>, treating h1 and h2 headings as chunk
    boundaries. h3/h4 stay within their parent chunk.

    The first h1 we encounter is used as the canonical page title (cleaner
    than the browser <title> tag which includes emoji and site suffix).

    Returns list of chunk dicts with keys:
        id, url, page_title, section, text, code_examples_raw, word_count
    """
    url_hash = _url_hash(url)

    # Content root: the <section> child of the article.
    # Fall back to the article itself for non-standard pages.
    content_root = article.find("section") or article

    chunks: list[dict] = []
    chunk_idx = 0
    inferred_title = page_title   # overwritten by first h1 we find
    current_section = page_title
    current_parts: list[str] = []
    current_codes: list[str] = []

    def _flush():
        nonlocal chunk_idx
        text = "\n".join(p for p in current_parts if p.strip()).strip()
        if _word_count(text) < MIN_CHUNK_WORDS:
            return
        for sub_text, sub_codes in _force_split(text, list(current_codes), MAX_CHUNK_WORDS):
            if _word_count(sub_text) >= MIN_CHUNK_WORDS:
                chunks.append({
                    "id": f"{url_hash}_chunk_{chunk_idx:03d}",
                    "url": url,
                    "page_title": inferred_title,
                    "section": current_section,
                    "text": sub_text,
                    "code_examples_raw": sub_codes,
                    "word_count": _word_count(sub_text),
                })
                chunk_idx += 1

    def _walk(node: Tag) -> None:
        nonlocal current_section, inferred_title

        for el in node.children:
            if isinstance(el, NavigableString):
                continue
            if not hasattr(el, "name") or not el.name:
                continue

            tag = el.name

            # Skip footer nav and non-content elements
            if tag in ("nav", "style", "script"):
                continue

            if tag in ("h1", "h2"):
                _flush()
                current_parts.clear()
                current_codes.clear()
                heading = el.get_text(strip=True)
                # First h1 becomes the canonical page title
                if tag == "h1" and not chunks and chunk_idx == 0 and not current_parts:
                    inferred_title = heading
                current_section = heading
                current_parts.append(f"{'#' * int(tag[1])} {heading}")

            elif tag in ("h3", "h4"):
                current_parts.append(f"{'#' * int(tag[1])} {el.get_text(strip=True)}")

            elif tag == "pre":
                code = el.get_text(strip=True)
                current_parts.append(f"\n```strudel\n{code}\n```\n")
                current_codes.append(code)

            elif tag == "p":
                text = el.get_text(separator=" ", strip=True)
                if text:
                    current_parts.append(text)

            elif tag in ("ul", "ol"):
                for li in el.find_all("li"):
                    current_parts.append(f"- {li.get_text(separator=' ', strip=True)}")

            elif tag in ("table",):
                # Simple table → flatten to text
                current_parts.append(el.get_text(separator=" ", strip=True))

            elif tag not in ("img", "iframe", "input", "button", "form",
                             "svg", "canvas", "audio", "video"):
                # Recurse into ANY other element (div, section, astro-island,
                # custom-repl, details, summary, blockquote, etc.)
                # This future-proofs against new component types.
                _walk(el)

    _walk(content_root)
    _flush()
    return chunks


def _force_split(text: str, codes: list[str], max_words: int) -> list[tuple[str, list[str]]]:
    """
    If text exceeds max_words, split at paragraph boundaries.
    Returns list of (text_fragment, code_list) tuples.
    """
    if _word_count(text) <= max_words:
        return [(text, codes)]

    paragraphs = re.split(r"\n{2,}", text)
    parts = []
    current: list[str] = []
    current_wc = 0

    for para in paragraphs:
        para_wc = _word_count(para)
        if current_wc + para_wc > max_words and current:
            parts.append(("\n\n".join(current), []))
            current = [para]
            current_wc = para_wc
        else:
            current.append(para)
            current_wc += para_wc

    if current:
        parts.append(("\n\n".join(current), []))

    # Attach codes to first sub-chunk only
    if parts:
        parts[0] = (parts[0][0], codes)
    return parts


def _url_hash(url: str) -> str:
    import hashlib
    return hashlib.sha256(url.encode()).hexdigest()[:16]


# ---------------------------------------------------------------------------
# Phase 3: LLM concept extraction
# ---------------------------------------------------------------------------

SYSTEM_PROMPT = """\
You are a knowledge extraction assistant specializing in technical documentation
for Strudel, a JavaScript live-coding music pattern library.

Your job: given a section of documentation, extract structured knowledge that
will be used to build a searchable knowledge graph and MCP server.

Always respond with valid JSON only — no markdown fences, no explanation.
"""

def _build_user_prompt(chunk: dict) -> str:
    # Truncate code examples to avoid token overflow
    codes_text = ""
    if chunk["code_examples_raw"]:
        shown = chunk["code_examples_raw"][:MAX_CODES_IN_PROMPT]
        codes_text = "\n\nCode examples in this section:\n" + "\n---\n".join(
            f"```\n{c[:400]}\n```" for c in shown
        )

    return f"""Documentation section from: {chunk['page_title']} ({chunk['url']})
Section heading: {chunk['section']}

--- Content ---
{chunk['text'][:3000]}
{codes_text}
--- End ---

Extract the following as a JSON object:

{{
  "concepts": [
    {{"name": "short concept name", "definition": "one clear sentence"}}
  ],
  "tags": ["tag1", "tag2"],
  "code_examples": [
    {{"code": "...", "description": "one sentence describing what this demonstrates"}}
  ],
  "prerequisites": ["concept names the reader should already understand"],
  "relationships": [
    {{"from": "concept A", "to": "concept B", "type": "requires|related_to|explains|part_of"}}
  ]
}}

Guidelines:
- concepts: name the specific ideas taught here (2-8 concepts, not generic terms)
- tags: broad categories (e.g. "rhythm", "synthesis", "beginner", "pattern")
- code_examples: only include examples that clearly demonstrate a concept
- prerequisites: be specific, use exact concept names where possible
- relationships: only include confident relationships, 0-5 is fine
"""


def _extract_json(response_text: str) -> dict:
    """
    Robustly extract a JSON object from LLM response text.
    Handles cases where the model adds explanation or markdown fences.
    """
    text = response_text.strip()

    # Try direct parse first
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass

    # Strip markdown code fences
    text = re.sub(r"^```(?:json)?\s*", "", text, flags=re.MULTILINE)
    text = re.sub(r"\s*```$", "", text, flags=re.MULTILINE)

    # Find the outermost JSON object
    start = text.find("{")
    end = text.rfind("}") + 1
    if start >= 0 and end > start:
        try:
            return json.loads(text[start:end])
        except json.JSONDecodeError:
            pass

    return {}


def _empty_extraction() -> dict:
    return {
        "concepts": [],
        "tags": [],
        "code_examples": [],
        "prerequisites": [],
        "relationships": [],
    }


def extract_concepts_llm(
    chunk: dict,
    client: anthropic.Anthropic,
    model: str,
    retries: int = 3,
) -> dict:
    """
    Call Claude API to extract concepts from a single chunk.
    Returns the parsed extraction dict, or an empty dict on failure.
    """
    for attempt in range(retries):
        try:
            response = client.messages.create(
                model=model,
                max_tokens=1024,
                system=SYSTEM_PROMPT,
                messages=[{"role": "user", "content": _build_user_prompt(chunk)}],
            )
            raw = response.content[0].text
            parsed = _extract_json(raw)
            if parsed:
                return parsed
            print(f"         ⚠  JSON parse failed, attempt {attempt+1}/{retries}")
            print(f"            Response: {raw[:200]}")

        except anthropic.RateLimitError:
            wait = (attempt + 1) * 10
            print(f"         ⚠  Rate limit hit. Waiting {wait}s...")
            time.sleep(wait)
        except anthropic.APIError as e:
            print(f"         ✗  API error: {e}")
            time.sleep(2)
        except Exception as e:
            print(f"         ✗  Unexpected error: {e}")
            time.sleep(2)

    return _empty_extraction()


# ---------------------------------------------------------------------------
# State management (mirrors CrawlDB from crawl.py)
# ---------------------------------------------------------------------------

def load_state(output_path: Path) -> dict:
    state_file = output_path / "crawl.json"
    if state_file.exists():
        return json.loads(state_file.read_text())
    return {"pages": [], "meta": []}


def save_state(output_path: Path, state: dict):
    state_file = output_path / "crawl.json"
    state_file.write_text(json.dumps(state, indent=2))


def get_page_status(state: dict, url: str) -> str | None:
    for p in state["pages"]:
        if p["url"] == url:
            return p["status"]
    return None


def set_page_status(state: dict, url: str, status: str):
    for p in state["pages"]:
        if p["url"] == url:
            p["status"] = status
            return
    state["pages"].append({"url": url, "status": status})


# ---------------------------------------------------------------------------
# Main pipeline
# ---------------------------------------------------------------------------

def should_skip(url: str) -> bool:
    return any(pat in url for pat in SKIP_URL_PATTERNS)


def process_page(
    raw_file: Path,
    output_dir: Path,
    client: anthropic.Anthropic | None,
    model: str,
    dry_run: bool,
    api_delay: float,
) -> dict:
    """
    Process a single raw_content JSON file.
    Returns a summary dict with counts.
    """
    d = json.loads(raw_file.read_text())
    url = d["url"]
    page_title = d.get("title", url.split("/")[-1])

    # Skip conditions
    if should_skip(url):
        return {"url": url, "skipped": "url_pattern", "chunks": 0}

    html = d.get("raw_html", "")
    article = extract_article(html)
    if not article:
        return {"url": url, "skipped": "no_article", "chunks": 0}

    # Phase 2: chunking
    chunks = split_into_chunks(article, url, page_title)
    if not chunks:
        return {"url": url, "skipped": "no_chunks", "chunks": 0}

    # Phase 3: LLM extraction (or dry-run pass-through)
    extracted_chunks = []
    for i, chunk in enumerate(chunks):
        print(f"           chunk {i+1}/{len(chunks)}: \"{chunk['section'][:50]}\"  "
              f"({chunk['word_count']} words, {len(chunk['code_examples_raw'])} code blocks)")

        if dry_run or client is None:
            extraction = _empty_extraction()
        else:
            extraction = extract_concepts_llm(chunk, client, model)
            if api_delay > 0:
                time.sleep(api_delay)

        extracted_chunks.append({**chunk, **extraction})

    # Save to extracted/
    extracted_dir = output_dir / "extracted"
    extracted_dir.mkdir(exist_ok=True)
    out_file = extracted_dir / f"{_url_hash(url)}.json"
    out_data = {
        "url": url,
        "title": page_title,
        "processed_at": datetime.now().isoformat(),
        "model": model if not dry_run else "dry-run",
        "chunks": extracted_chunks,
    }
    out_file.write_text(json.dumps(out_data, indent=2, ensure_ascii=False))

    return {"url": url, "skipped": None, "chunks": len(chunks)}


def run(
    input_dir: str,
    model_shorthand: str = "haiku",
    max_pages: int | None = None,
    dry_run: bool = False,
    force: bool = False,
    api_delay: float = 0.2,
):
    output_path = Path(input_dir)
    raw_dir = output_path / "raw_content"

    if not raw_dir.exists():
        print(f"✗  raw_content/ not found at {raw_dir}")
        sys.exit(1)

    # Resolve model
    model = MODEL_MAP.get(model_shorthand, model_shorthand)
    print(f"\n{'━'*62}")
    print(f"  GraphRAG Builder — Concept Extraction (M2)")
    print(f"{'━'*62}")
    print(f"  Input:   {output_path}")
    print(f"  Model:   {model}{' (dry-run, no API calls)' if dry_run else ''}")
    if max_pages:
        print(f"  Limit:   first {max_pages} pages")
    print(f"{'━'*62}\n")

    # Set up Anthropic client
    client = None
    if not dry_run:
        [REDACTED_SECRET]"ANTHROPIC_API_KEY")
        if not [REDACTED_SECRET]"✗  ANTHROPIC_API_KEY not set. Use --dry-run to test without API calls.")
            sys.exit(1)
        client = anthropic.Anthropic([REDACTED_SECRET])

    # Load crawl state
    state = load_state(output_path)

    raw_files = sorted(raw_dir.glob("*.json"))
    if max_pages:
        raw_files = raw_files[:max_pages]

    stats = {"processed": 0, "skipped": 0, "total_chunks": 0, "failed": 0}

    for raw_file in raw_files:
        d = json.loads(raw_file.read_text())
        url = d["url"]
        status = get_page_status(state, url)

        # Incremental: skip if already extracted (unless --force)
        if not force and status in ("extracted", "embedded", "graphed", "complete"):
            print(f"  ✓  already extracted — skip  {url.replace('https://strudel.cc', '')}")
            stats["skipped"] += 1
            continue

        short_url = url.replace("https://strudel.cc", "") or "/"
        print(f"  Processing: {short_url}")

        result = process_page(
            raw_file=raw_file,
            output_dir=output_path,
            client=client,
            model=model,
            dry_run=dry_run,
            api_delay=api_delay,
        )

        if result["skipped"]:
            print(f"           → skipped ({result['skipped']})")
            stats["skipped"] += 1
        else:
            stats["processed"] += 1
            stats["total_chunks"] += result["chunks"]
            # Advance status in state
            set_page_status(state, url, "extracted")

    save_state(output_path, state)

    print(f"\n{'━'*62}")
    print(f"  Done!")
    print(f"  Pages processed:   {stats['processed']}")
    print(f"  Pages skipped:     {stats['skipped']}")
    print(f"  Total chunks:      {stats['total_chunks']}")
    print(f"  Output:            {output_path}/extracted/")
    print(f"{'━'*62}\n")


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="GraphRAG Builder — Concept Extraction (M2)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  # Dry run — test chunking without API calls:
  python extract_concepts.py --input ./output/strudel-cc-mcp --dry-run

  # Extract 5 pages with haiku (cheap, fast):
  python extract_concepts.py --input ./output/strudel-cc-mcp --max-pages 5

  # Full extraction with sonnet (better quality):
  python extract_concepts.py --input ./output/strudel-cc-mcp --model sonnet

  # Re-run everything, ignoring prior extracted status:
  python extract_concepts.py --input ./output/strudel-cc-mcp --force
        """,
    )
    parser.add_argument("--input", required=True,
                        help="Path to the MCP output folder (contains raw_content/)")
    parser.add_argument("--model", default="haiku",
                        help="Model shorthand: haiku, sonnet, opus, or full model string (default: haiku)")
    parser.add_argument("--max-pages", type=int, default=None, metavar="N",
                        help="Process only the first N pages (for testing)")
    parser.add_argument("--dry-run", action="store_true",
                        help="Run chunking only, skip LLM calls. Good for testing.")
    parser.add_argument("--force", action="store_true",
                        help="Re-extract pages even if already marked as extracted")
    parser.add_argument("--api-delay", type=float, default=0.2, metavar="SEC",
                        help="Seconds between API calls to avoid rate limiting (default: 0.2)")

    args = parser.parse_args()
    run(
        input_dir=args.input,
        model_shorthand=args.model,
        max_pages=args.max_pages,
        dry_run=args.dry_run,
        force=args.force,
        api_delay=args.api_delay,
    )


if __name__ == "__main__":
    main()


path: evals/evals.json
content: {
  "skill_name": "GraphRAGBuilderSkill",
  "evals": [
    {
      "id": 1,
      "prompt": "Build an MCP server from the Strudel live-coding library docs at https://strudel.cc/workshop/getting-started — I want to be able to ask Claude questions about it",
      "expected_output": "A fully runnable server.py and mcp_config.json in output/strudel-cc-mcp/, with clear instructions for installing into Claude Desktop",
      "files": [],
      "assertions": [
        "Claude asks for or confirms the target URL before running anything",
        "Claude provides the crawl.py command with --url and --max-depth flags",
        "Claude suggests running extract_concepts.py with --dry-run first before spending API budget",
        "Claude provides the full sequence of 5 pipeline commands (M1-M5)",
        "Claude explains that ANTHROPIC_API_KEY is needed for M2",
        "Claude mentions the --smoke-test flag for M4 to verify the embedding index",
        "Claude provides clear instructions for installing the server into Claude Desktop",
        "Claude explains what the 8 MCP tools do after the server is built"
      ]
    },
    {
      "id": 2,
      "prompt": "I want to index the React documentation so I can search it semantically in Claude. The docs are at https://react.dev/learn",
      "expected_output": "Clear pipeline commands for the React docs, with appropriate --max-depth recommendation for a large doc site",
      "files": [],
      "assertions": [
        "Claude provides M1 crawl command with the React docs URL",
        "Claude suggests --max-depth 2 or warns that React docs may be large",
        "Claude mentions the haiku model for cost-efficiency on a large site",
        "Claude suggests using --max-pages 20 or similar to test before full run",
        "Claude provides the complete pipeline sequence ending with server.py generation",
        "Claude explains the output slug will be derived from the domain (react-dev)"
      ]
    },
    {
      "id": 3,
      "prompt": "I already ran the crawl and got 56 pages. Now I want to extract the concepts and build the MCP server. The output is in ./output/strudel-cc-mcp",
      "expected_output": "Commands for M2 through M5 only, skipping M1, with dry-run suggestion",
      "files": [],
      "assertions": [
        "Claude does NOT tell the user to re-run the crawl",
        "Claude provides the extract_concepts.py --dry-run command first",
        "Claude provides the live LLM extraction command with ANTHROPIC_API_KEY",
        "Claude provides build_graph.py command pointing to ./output/strudel-cc-mcp",
        "Claude provides build_embeddings.py command with --smoke-test",
        "Claude provides generate_mcp_server.py command",
        "All commands use --input ./output/strudel-cc-mcp"
      ]
    },
    {
      "id": 4,
      "prompt": "The extract_concepts.py script ran but it says 'Skipped 46 dry-run files' when I run build_graph.py and the graph has 0 nodes. What's wrong?",
      "expected_output": "Clear diagnosis that dry-run mode was used and instructions to re-run with real API key",
      "files": [],
      "assertions": [
        "Claude correctly identifies that extract_concepts.py was run with --dry-run flag",
        "Claude explains that dry-run outputs have empty concept arrays (model: 'dry-run')",
        "Claude provides the corrected command with ANTHROPIC_API_KEY set",
        "Claude does NOT suggest deleting the extracted/ folder (--force flag is sufficient)",
        "Claude provides the --force flag to overwrite dry-run outputs"
      ]
    },
    {
      "id": 5,
      "prompt": "Make the Tailwind CSS docs searchable in Claude. Site: https://tailwindcss.com/docs",
      "expected_output": "Pipeline commands with appropriate settings for a large CSS framework doc site",
      "files": [],
      "assertions": [
        "Claude provides M1 command with the Tailwind docs URL",
        "Claude recommends a reasonable max-depth (2-3) for a large docs site",
        "Claude provides the full 5-step pipeline sequence",
        "Claude mentions the final output will be in output/tailwindcss-com-mcp/ or similar",
        "Claude mentions the dry-run step for cost verification before M2 LLM extraction"
      ]
    }
  ]
}


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)