Skill: jonathangu/crabpath (version 11.2.1)

--- name: crabpath description: Memory graph engine with caller-provided embed and LLM callbacks; core is pure, with real-time correction flow and optional OpenAI integration. metadata: openclaw: emoji: "🦀" requires: python: ">=3.10" --- # CrabPath Pure graph core: zero required deps and no network calls. Caller provides callbacks. ## Design Tenets - No network calls in core - No secret discovery (no dotfiles, keychain, or env probing) - No subprocess provider wrappers - Embedder identity in state metadata; dimension mismatches are errors - One canonical state format (`state.json`) ## Quick Start ```python from crabpath import split_workspace, HashEmbedder, VectorIndex graph, texts = split_workspace("./workspace") embedder = HashEmbedder() index = VectorIndex() for nid, content in texts.items(): index.upsert(nid, embedder.embed(content)) ``` ## Embeddings and LLM callbacks - Default: `HashEmbedder` (hash-v1, 1024-dim) - Real: callback `embed_fn` / `embed_batch_fn` (e.g., `text-embedding-3-small`) - LLM routing: callback `llm_fn` using `gpt-5-mini` (example) ## Session Replay `replay_queries(graph, queries)` can warm-start from historical turns. ## CLI `--state` is preferred: `crabpath query TEXT --state S [--top N] [--json]` `crabpath query TEXT --state S --chat-id CID` `crabpath doctor --state S` `crabpath info --state S` `crabpath init --workspace W --output O --embedder openai` `crabpath query TEXT --state S --llm openai` `crabpath inject --state S --type TEACHING [--type DIRECTIVE]` Real-time correction flow: `python3 query_brain.py --chat-id CHAT_ID` `python3 learn_correction.py --chat-id CHAT_ID` ## Quick Reference - `crabpath init/query/learn/inject/health/doctor/info` - `query_brain.py --chat-id` and `learn_correction.py` for real-time correction pipelines - `query_brain.py` traversal limits: `beam_width=8`, `max_hops=30`, `fire_threshold=0.01` - Hard traversal caps: `max_fired_nodes` and `max_context_chars` (defaults `None`; `query_brain.py` defaults `max_context_chars=20000`) - `examples/correction_flow/`, `examples/cold_start/`, `examples/openai_embedder/` ## API Reference - Core lifecycle: - `split_workspace` - `load_state` - `save_state` - `ManagedState` - `VectorIndex` - Traversal and learning: - `traverse` - `TraversalConfig` - `TraversalConfig.beam_width`, `.max_hops`, `.fire_threshold`, `.max_fired_nodes`, `.max_context_chars`, `.reflex_threshold`, `.habitual_range`, `.inhibitory_threshold` - `TraversalResult` - `apply_outcome` - Runtime injection APIs: - `inject_node` - `inject_correction` - `inject_batch` - Maintenance helpers: - `suggest_connections`, `apply_connections` - `suggest_merges`, `apply_merge` - `measure_health`, `autotune`, `replay_queries` - Embedding utilities: - `HashEmbedder` - `OpenAIEmbedder` - `default_embed` - `default_embed_batch` - `openai_llm_fn` - LLM routing callbacks: - `chat_completion` - Graph primitives: - `Node` - `Edge` - `Graph` - `split_workspace` - `generate_summaries` ## CLI Commands - `crabpath init --workspace W --output O [--sessions S] [--embedder openai]` - `crabpath query TEXT --state S [--top N] [--json] [--chat-id CHAT_ID]` - `crabpath learn --state S --outcome N --fired-ids a,b,c [--json]` - `crabpath inject --state S --id NODE_ID --content TEXT [--type CORRECTION|TEACHING|DIRECTIVE] [--json] [--connect-min-sim 0.0]` - `crabpath inject --state S --id NODE_ID --content TEXT --type TEACHING` - `crabpath inject --state S --id NODE_ID --content TEXT --type DIRECTIVE` - `crabpath health --state S` - `crabpath doctor --state S` - `crabpath info --state S` - `crabpath replay --state S --sessions S` - `crabpath merge --state S [--llm openai]` - `crabpath connect --state S [--llm openai]` - `crabpath journal [--stats]` - `query_brain.py --chat-id CHAT_ID` - `learn_correction.py --chat-id CHAT_ID` ## Traversal defaults - `beam_width=8` - `max_hops=30` - `fire_threshold=0.01` - `reflex_threshold=0.6` - `habitual_range=0.2-0.6` - `inhibitory_threshold=-0.01` - `max_fired_nodes` (hard node-count cap, default `None`) - `max_context_chars` (hard context cap, default `None`; `query_brain.py` default is `20000`) ## Paper https://jonathangu.com/crabpath/

path: CHANGELOG.md
content: ## v11.2.0 (2026-02-27)

### New: Persistent worker daemon (`crabpath daemon`)
- `crabpath/daemon.py`: loads state once, keeps graph+index hot in memory
- JSON-RPC over stdin/stdout (NDJSON protocol)
- Methods: query, learn, maintain, health, info, save, reload, shutdown
- Query responses include timing: `embed_query_ms`, `traverse_ms`, `total_ms`
- Auto-save after N write ops, graceful SIGTERM/SIGINT handling
- Eliminates per-call state reload (~100-800ms savings)
- `examples/ops/client_example.py`: Python client demo

### New: Constitutional anchors (`crabpath anchor`)
- Authority levels: constitutional (never decay/prune/merge), canonical (slow decay), overlay (default)
- `crabpath maintain` respects authority levels during all structural ops
- CLI: `crabpath anchor --state PATH --node-id ID --authority constitutional|canonical`

### New: Incremental file sync (`crabpath sync`)
- `crabpath/sync.py`: detect file changes, re-embed only what changed
- Default authority map for common files (SOUL.md → constitutional, USER.md → canonical)
- CLI: `crabpath sync --state PATH --workspace DIR`

### New: Daily note compaction (`crabpath compact`)
- `crabpath/compact.py`: old notes → extract facts → inject into graph → shrink files
- Optional LLM summarization, deterministic fallback
- CLI: `crabpath compact --state PATH --memory-dir DIR`

### Fix: save_state() embedder metadata preservation
- Now reads existing meta before writing; no more silent hash-v1/1024 overwrite
- Dimension mismatch raises ValueError (prevents corrupt state)
- Fixes GitHub Issue #1 item #7 (CormorantAI production data corruption)

### Fix: Rebuild preserves injected nodes
- `--preserve-injected` flag (default True) in init_agent_brain.py and rebuild_all_brains.py
- Snapshots + restores CORRECTION/TEACHING nodes across rebuilds
- Fixes GitHub Issue #1 item #5

### Docs
- README overhaul: all 17 CLI commands, apply_outcome_pg section, "Why CrabPath", write policy
- `docs/architecture.md`: persistent worker, context lifecycle, constitutional anchors
- `docs/setup-guide.md`: steps 7-9 (sync, anchors, compact)
- Production stats: 4 brains (MAIN, PELICAN, BOUNTIFUL, CORMORANT)
- `jonathangu.com/crabpath/gu2016/`: accessible policy gradient derivation page

### Tests
- 246 tests (was 220), 15 simulations

## v11.1.0 (2026-02-27)

### New: Maintenance engine (`crabpath maintain`)
- `crabpath/maintain.py`: `run_maintenance()`, `prune_edges()`, `prune_orphan_nodes()`
- CLI: `crabpath maintain --state PATH --tasks health,decay,prune,merge [--dry-run]`
- Scheduler-agnostic: call from cron, systemd, Airflow, or any batch runner

### New: Two-timescale architecture docs
- `docs/architecture.md`: online learning (fast loop) vs maintenance (slow loop)
- Integration contract: what frameworks provide, what CrabPath produces
- Brain directory layout specification

### New: Framework-agnostic examples (`examples/ops/`)
- `callbacks.py`: `make_embed_fn()` and `make_llm_fn()` factory functions
  - Default: OpenAI text-embedding-3-small + gpt-5-mini
  - Fallback: HashEmbedder (zero-dep, for testing)
- `query_and_learn.py`: fast loop example with --embedder flag
- `run_maintenance.py`: slow loop example with --embedder and --llm flags

### New: True REINFORCE policy gradient (`apply_outcome_pg`)
- Full derivation: softmax score function ∇log π = (1/τ)(eₐ - π)
- Updates ALL outgoing edges at each visited node (not just traversed)
- Conservation property: weight updates sum to zero per node
- Temperature scaling, baseline support, discount along trajectory

### New: 12 simulations + expanded benchmarks
- PG vs heuristic ablation (weight conservation)
- Noise robustness (graceful to 20%, degrades at 30%)
- Static vs learning baseline
- Scaling analysis (sub-linear: 50→2000 nodes)
- Internal benchmark: 45 queries, 8 methods
- External benchmarks: MultiHop-RAG (n=1000) + HotPotQA (n=500) with OpenAI embeddings

### Paper updates
- Title: "Learned Graph Traversal for Retrieval Routing"
- Full PG derivation with numerical examples
- External benchmark results (cold traverse +7.7% MultiHop, +33.6% HotPotQA)
- De-marketing pass: removed metaphors, added crisp definitions
- Honest finding: naive online learning degrades on non-repeating queries

# CHANGELOG

## Unreleased

### Maintenance loop and ops docs
- Added `crabpath.maintain` with `run_maintenance`, `prune_edges`, and `prune_orphan_nodes`.
- Added `crabpath maintain` CLI command with `--state`, `--tasks`, `--dry-run`, `--max-merges`, and `--prune-below`.
- Added framework-agnostic examples in `examples/ops/` for fast loop and maintenance loop.
- Added `docs/architecture.md` documenting fast/slow loops, integration contracts, and scheduler guidance.
- Added `tests/test_maintain.py` covering maintenance execution, dry-run behavior, prune operations, merge effects, and report accounting.

## v10.4.0 (2026-02-27)

### True REINFORCE policy gradient
- Added `apply_outcome_pg` in `crabpath.learn` for true policy-gradient updates:
  - numerically stable softmax over outgoing edges plus synthetic STOP
  - updates for all outgoing edges at each step using `(1[chosen] - π)` score gradient
  - optional baseline and temperature support
  - weight clipping + Hebbian co-firing preserved
- Exported `apply_outcome_pg` from `crabpath.__init__`.
- Extended `LearningConfig` with `temperature` and `baseline`.

### External benchmark additions
- Added external benchmark harness support for MultiHop-RAG and HotPotQA.
- Added OpenAI embedding cache support for benchmark scripts to support fast repeat runs.

## v10.3.0 (2026-02-27)

### Traversal tiers recalibrated from production data
- **Defaults changed:** `reflex_threshold: 0.8 -> 0.6`, `habitual_range: 0.3-0.8 -> 0.2-0.6`, `inhibitory_threshold: -0.3 -> -0.01`.
- **Production root-cause fix:** inhibitory edges were non-functional because the default threshold was `-0.3` while negatives were near `-0.035`.
- **Tests and sims updated:** adjusted tier-classification expectations and added coverage for `-0.02` as inhibitory.
- Test count increased to **204** in docs to match new coverage.

## v10.2.0 (2026-02-27)

### Traversal engine hardening
- **Traversal termination updated:** stops on hop ceiling, fired-node budget, and context-character budget.
- **Defaults changed:** `beam_width: 3 → 8`, `max_hops: 15 → 30`, `fire_threshold: 0.0 → 0.01`.
- **New control:** `max_fired_nodes` hard-cap added for traversal loops.
- **Context budget enforcement:** `max_context_chars` is enforced during traversal, not only at render time.
- **Default CLI / script behavior:** `query_brain.py` now defaults to `max_context_chars=20000` for logged session flow.
- Test count updated to **203** (from 200).

## v10.1.0 (2026-02-27)

### Real-time correction flow
- **Fired-node logging:** `query_brain.py --chat-id` persists fired nodes to `fired_log.jsonl` per conversation, with 7-day rolling prune. Corrections can now find what to penalize even hours after the original query.
- **Same-turn corrections:** new `learn_correction.py` reads fired log by chat_id, penalizes via `apply_outcome`, and optionally injects a CORRECTION node — all in one command.
- **Dedup with batch harvester:** `injected_corrections.jsonl` prevents double-injection when the 2-hour harvester later processes the same correction. `init_agent_brain.py` reads the dedup log on rebuild.

### CLI DX improvements
- **Query text output shows node IDs** — no more `--json` required for the learn workflow. Format: `node_id / ~~~ / content`.
- **`learn --json` returns summary** (`{"edges_updated": N, "max_weight_delta": X}`) instead of dumping the entire graph.
- **`health` text output is human-readable** — `Brain health: / Nodes / Edges / Reflex% Habitual% Dormant% / Orphans / Cross-file edges`.
- **`inject` auto-detects hash-v1 embedder** and defaults `--connect-min-sim` to 0.0 (was 0.3, causing zero connections for users without OpenAI).

### OpenAI integration
- `crabpath/openai_embeddings.py` — `OpenAIEmbedder` class wrapping `text-embedding-3-small`.
- `crabpath/openai_llm.py` — `openai_llm_fn` and `chat_completion` for GPT-5-mini routing/scoring.
- CLI flags: `--embedder openai` for init/query/inject, `--llm openai` for init/merge/connect.
- Zero new required dependencies — `openai` is an optional runtime import.

### Documentation overhaul
- **README rewritten:** value prop first, 5-minute quickstart with A→B learning story, "Correcting mistakes" and "Adding new knowledge" sections promoted to top.
- **TEACHING documented:** all three injection types (CORRECTION, TEACHING, DIRECTIVE) explained with examples. Previously only CORRECTION was documented — three production agents independently thought rebuild was required to add new knowledge.
- **Competition table:** CrabPath vs Plain RAG vs Reflexion vs MemGPT.
- **New sections:** State lifecycle, Cost control, "How CrabPath differs from related tools".
- **Session replay demoted** to "Optional: warm start" with skip note for new users.

### New examples
- `examples/cold_start/` — guided zero-session walkthrough with canned queries and expected output.
- `examples/correction_flow/` — generic fired-node logging pattern for any agent framework (hash embedder, no API key).
- `examples/sample_workspace/` expanded from 3 to 5 files (added `incidents.md`, `onboarding.md`) for cross-file routing demos.
- `examples/openclaw_adapter/agents_hook.md` updated with full inject + correction + chat-id logging template.

### Tests
- 200 tests (was 188), 8 simulations all passing.
- New tests: hash-embedder inject default, query text node IDs, health readable output, learn summary, correction flow integration (3 tests).

## v10.0.0
### What Changed
- Added CLI and API flows so CrabPath can inject and persist correction/teaching signals without rebuilding the entire state.
- Expanded operational guidance and verification workflow so feedback and correction behavior can be validated deterministically.

- Added live injection APIs (`inject_node`, `inject_correction`, `inject_batch`) with CLI support via `crabpath inject`, enabling runtime updates during operation.
- Added direct graph node injection paths for TEACHING, DIRECTIVE, and CORRECTION workflows, plus lightweight injection stats payloads from the CLI.

- Expanded reproducibility docs with live injection verification steps so users can confirm graph updates and health signals after corrective actions.

## v9.3.1
- Hardened command entrypoints and replay/logging behavior for edge cases in graph indexing and state workflows.
- Improved docs around runtime injection, exports, and deterministic test paths to make troubleshooting and operator handoff faster.

## v9.3.0
- Introduced live injection primitives (`inject_*`) and correction-node inhibitory edge behavior.
- Added the `crabpath inject` command path and supporting test coverage for repeatable feedback workflows.

## v9.1.0
- Added adversarial tests and latency benchmark harness updates to surface stability issues before release.
- Added interaction extraction and benchmark cleanup paths so simulation artifacts are predictable across runs.

## v9.0.0
- Delivered 20 user-feedback fixes focused on inhibitory suppression, file-type handling, `max_chars`, and `doctor/info` behavior.
- Tuned graph traversal defaults and diagnostics so noisy retrieval cases now recover more gracefully in production-like inputs.

## v8.0.0
- Unified state format and added dimension validation to prevent accidental embedding mismatches.
- Documented core design tenets to make production constraints explicit for integrators.

## v7.0.0
- Brought `HashEmbedder` into the core implementation as a built-in default path.
- This removed external required dependencies from baseline operation while preserving callback flexibility.

## v6.0-6.1
- Removed subprocess-based paths and deduplicated helper logic to simplify execution flow.
- This reduced operational complexity and improved reliability for smaller deployments.

## v5.0-5.3
- Removed provider-specific integrations and standardized pure callback behavior.
- This made core graph operations less opinionated and easier to adapt across embedding/LLM stacks.

## v4.0-4.5
- Added LLM-based features and batch callback execution paths.
- Improved how route extraction and scoring can combine deterministic traversal with optional model guidance.

## v3.0.0
- Rewrote the graph engine core with zero dependencies.
- Added a stable, dependency-light baseline for local, offline usage and reproducible graph state files.


path: CONTRIBUTING.md
content: # Contributing

## Development setup

1. Clone the repository and enter it:
   - `git clone <repo-url>`
   - `cd <repo>`
2. Create or activate your environment, then install editable:
   - `pip install -e .`
3. Run the test suite:
   - `pytest`

## Code style

- No linter is currently required by the project.
- Public functions must have docstrings.
- Keep interfaces focused and backward-compatible where possible.

## Test requirements

- All tests must pass before opening a PR.
- The test suite currently has 200 tests.
- Run the full suite with:
  - `python3 -m pytest tests/ -x -q`
- Add tests for new features and bug fixes when behavior changes or new API
  shapes are introduced.

## Running simulations

Use the deterministic simulation harness to validate behavior after graph and CLI updates:

- Build and run all simulation suites with:
  - `python3 sims/run_all.py`
- The script prints per-simulation pass/fail summaries and should complete with
  exit code `0` before merging changes that affect learning behavior.

## Release process

1. Bump version in both:
   - `pyproject.toml`
   - `crabpath/__init__.py`
2. Create a release commit/tag:
   - `git tag vX.Y.Z`
3. Push commits and tags:
   - `git push`
   - `git push --tags`

## Releasing

Releases are published via GitHub Actions with OIDC (trusted publishing):

- Ensure `secrets` are not used for package credentials.
- Push a signed tag that matches `vX.Y.Z` to trigger `.github/workflows/publish.yml`.
- The workflow requires `permissions: id-token: write` and publishes through
  `pypa/gh-action-pypi-publish` in the `pypi` environment.
- Confirm a successful publish in the Actions tab after the tag workflow completes.


path: README.md
content: # CrabPath — learned retrieval routing for AI agents

> Your retrieval routes become the prompt — assembled by learned routing, not top-k similarity.

**Current release: v11.2.0**

**Setup:** [Setup Guide](docs/setup-guide.md)

**CrabPath learns from your agent feedback, so wrong answers get suppressed instead of resurfacing.** It builds a memory graph over your workspace, remembers what worked, and routes future answers through learned paths.

- Zero dependencies. Pure Python 3.10+.
- Works offline with built-in hash embeddings.
- Builds a **`state.json`** brain from your workspace.
- Queries follow learned routes instead of only similarity matches.
- Positive feedback (`+1`) strengthens routes, negative (`-1`) creates inhibitory edges.
- Over time, less noise appears and recurring mistakes are less likely.

- CrabPath integrates with your agent's file-based workspace through incremental sync, constitutional anchors, and automatic compaction.
- See the [context lifecycle](docs/architecture.md) for details.

## Install

```bash
pip install crabpath
```

See also: [Setup Guide](docs/setup-guide.md) for a complete local configuration walkthrough.

## Why CrabPath

- Static retrieval vs learned routing: CrabPath continuously updates node-to-node edges so good routes strengthen and bad routes decay.
- No correction propagation vs inhibitory edges: incorrect context can be actively suppressed and forgotten less often than in similarity-only systems.
- Bulk context load vs targeted traversal: context windows stay focused (roughly 52KB → 3-13KB in typical sessions) by following likely retrieval routes.
- No structural maintenance vs prune/merge/compact: CrabPath includes scheduled maintenance commands to keep the graph healthy and compact.
- No protection vs constitutional anchors: anchor critical nodes with authority so operational instructions do not drift.

## 5-minute quickstart (A→B learning story)

```bash
# 1. Build a brain from the sample workspace
crabpath init --workspace examples/sample_workspace --output /tmp/brain

# 2. Check state health
crabpath doctor --state /tmp/brain/state.json
# output
# PASS: python_version
# PASS: state_file_exists
# PASS: state_json_valid
# Summary: 8/9 checks passed

# 3. Query (text output includes node IDs)
crabpath query "how do I deploy" --state /tmp/brain/state.json --top 3 --json
# output (abbrev.)
# {"fired": ["deploy.md::0", "deploy.md::1", "deploy.md::2"], ...}

# 4. Teach it (good path)
crabpath learn --state /tmp/brain/state.json --outcome 1.0 --fired-ids "deploy.md::0,deploy.md::1"
# output
# {"edges_updated": 2, "max_weight_delta": 0.155}

# 5. Inject a correction
crabpath inject --state /tmp/brain/state.json \
  --id "fix::1" --content "Never skip CI for hotfixes" --type CORRECTION

# 5b. Add new knowledge (no correction needed, just a new fact)
crabpath inject --state /tmp/brain/state.json \
  --id "teaching::monitoring-tip" \
  --content "Check Grafana dashboards before every deploy" \
  --type TEACHING

# 6. Query again and see the route change
crabpath query "can I skip CI" --state /tmp/brain/state.json --top 3
# output
# fix::1
# ~~~~~~
# Never skip CI for hotfixes
# ...

# 7. Re-check health for a quick signal
crabpath health --state /tmp/brain/state.json
```

## Correcting mistakes (the main workflow)

When your agent retrieves wrong context, teach CrabPath in one command:

```bash
crabpath inject --state brain/state.json \
  --id "correction::42" \
  --content "Never show API keys in chat messages" \
  --type CORRECTION
```

What happens:
1. CrabPath creates a new node with your correction text
2. It connects that node to the most related workspace chunks
3. It adds **inhibitory edges** — negative-weight links that suppress those chunks
4. Next query touching that topic: the correction appears, the bad route is dampened

To add knowledge without suppressing anything, use `--type TEACHING` instead.

## Adding new knowledge (no rebuild needed)

When you learn something that isn't in any workspace file, inject it directly:

```bash
crabpath inject --state brain/state.json \
  --id "teaching::codex-spark" \
  --content "Use Codex CLI with gpt-5.3-codex-spark for coding tasks — free on Pro plan" \
  --type TEACHING
```

TEACHING nodes connect to related workspace chunks just like CORRECTION nodes,
but without inhibitory edges — they add knowledge instead of suppressing it.

Three injection types:
- **CORRECTION** — creates inhibitory edges that suppress related wrong paths
- **TEACHING** — adds knowledge with normal positive connections
- **DIRECTIVE** — same as TEACHING (use for standing instructions)

For agent frameworks that need to correlate corrections with earlier queries,
see `examples/correction_flow/` for the fired-node logging pattern.

You can also reinforce good retrievals:

```bash
# After a query returns helpful context, strengthen those paths
crabpath learn --state brain/state.json --outcome 1.0 \
  --fired-ids "deploy.md::0,deploy.md::1"
```

Or weaken bad ones:

```bash
crabpath learn --state brain/state.json --outcome -1.0 \
  --fired-ids "monitoring.md::2"
```

## What it looks like in practice

```bash
# Before learning
crabpath query "how do we handle incidents" --state /tmp/brain/state.json --top 3

# After one good learn on the best route
crabpath learn --state /tmp/brain/state.json --outcome 1.0 --fired-ids "incidents.md::0,deploy.md::1"

# After one negative learn on a bad route
crabpath learn --state /tmp/brain/state.json --outcome -1.0 --fired-ids "monitoring.md::2,incidents.md::0"

# Query again to observe new routing
crabpath query "incident runbook for deploy failures" --state /tmp/brain/state.json --top 4
```

## How it compares

| | Plain RAG | CrabPath |
|---|-----------|----------|
| Retrieval | Similarity search | Learned graph traversal |
| Feedback | None | `learn +1/-1` updates edge weights |
| Wrong answers | Can keep resurfacing | Inhibitory edges suppress them |
| Adding knowledge | Re-index/re-embed | `inject --type TEACHING` (no rebuild) |
| Over time | Same results for same query | Routes become habitual behavior |
| Dependencies | Vector DB or service | Zero dependencies |

## How CrabPath differs from related tools

| | CrabPath | Plain RAG | Reflexion | MemGPT |
|---|----------|-----------|-----------|--------|
| What it learns | Retrieval routes | Nothing | Reasoning via self-reflection text | Memory read/write policies |
| Negative feedback | Inhibitory edges suppress bad paths | None | None (additive only) | None |
| New knowledge | `inject` node (no rebuild) | Re-embed corpus | Add to reflection prompt | Update tier config |
| Integration | Standalone library, any agent | Vector DB required | Tied to agent loop | Tied to agent architecture |
| Cold start | Hash embeddings, no API key | Needs embedding service | Needs prior episodes | Needs configured tiers |
| State | Single `state.json` file | External DB | Prompt history | Multi-tier storage |

## Real embeddings + LLM routing (OpenAI)

Offline hash embeddings work for trying the product, but real deployments generally use:

- **Embeddings:** `text-embedding-3-small` (1536-dim)
- **LLM routing/scoring:** `gpt-5-mini`

```python
from openai import OpenAI
from crabpath import split_workspace, VectorIndex

client = OpenAI()


def embed(text):
    return client.embeddings.create(
        model="text-embedding-3-small", input=[text]
    ).data[0].embedding


def llm(system, user):
    return client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "system", "content": system}, {"role": "user", "content": user}],
    ).choices[0].message.content

graph, texts = split_workspace("./workspace", llm_fn=llm)
index = VectorIndex()
for nid, content in texts.items():
    index.upsert(nid, embed(content))
```

See `examples/openai_embedder/` for a complete example.

## CLI Reference

| Command | Description |
|---------|-------------|
| `init` | Build a brain from workspace files |
| `query` | Traverse graph and return context |
| `learn` | Apply outcome feedback to fired edges |
| `merge` | Suggest and apply node merges |
| `anchor` | Set/list/remove constitutional authority on nodes |
| `connect` | Connect learning nodes to workspace neighborhoods |
| `maintain` | Run structural maintenance (health, decay, prune, merge) |
| `compact` | Compact old daily notes into graph nodes |
| `sync` | Incremental re-embed after file changes |
| `inject` | Add CORRECTION/TEACHING/DIRECTIVE nodes |
| `replay` | Replay session queries into brain |
| `health` | Show graph health metrics |
| `journal` | Show event journal |
| `doctor` | Run diagnostic checks |
| `info` | Show brain info (nodes, edges, embedder) |
| `daemon` | Start persistent worker (JSON-RPC over stdio, state loaded once) |

## Persistent Worker (`crabpath daemon`)

For production use, run CrabPath as a long-lived daemon that keeps state hot in memory:

```bash
crabpath daemon --state ~/.crabpath/main/state.json
```

- Loads `state.json` once at startup (~100-800ms saved per call)
- Accepts NDJSON requests over stdin, responds on stdout
- Methods: `query`, `learn`, `maintain`, `health`, `info`, `save`, `reload`, `shutdown`
- Query responses include timing: `embed_query_ms`, `traverse_ms`, `total_ms`
- Auto-saves after N write operations; graceful SIGTERM/SIGINT shutdown

Production timing (Mac Mini M4 Pro, OpenAI embeddings):
- MAIN (1,158 nodes): 397ms embed + 107ms traverse = **504ms total**
- PELICAN (582 nodes): 634ms embed + 51ms traverse = **685ms total**
- BOUNTIFUL (285 nodes): 404ms embed + 27ms traverse = **431ms total**

See `examples/ops/client_example.py` for a Python client and `docs/architecture.md` for protocol details.

## True Policy Gradient (apply_outcome_pg)

`apply_outcome_pg` implements a full REINFORCE policy-gradient update.

- It updates **all outgoing edges** for each visited node on the fired trajectory, not only traversed edges.
- It uses the update:
  `Δw = (η(z-b)γ^ℓ)/τ · (𝟙[j=a] - π(j|i))`
  where:
  - `η` = learning rate
  - `z` = outcome reward
  - `b` = baseline
  - `γ` = discount
  - `ℓ` = trajectory depth
  - `τ` = temperature
  - `π(j|i)` = action probability from softmax (including STOP)
  - `𝟙[j=a]` = 1 for the taken action, else 0
- Conservation property: for each source node `i`, outgoing updates sum to zero, so total outgoing mass is preserved.
- Use `apply_outcome_pg` when you want smoother, probability-based updates across alternatives; use `apply_outcome` for a simpler sparse update that only touches traversed edges.

```python
from crabpath import apply_outcome_pg, LearningConfig

config = LearningConfig(learning_rate=0.1, temperature=1.0, baseline=0.0)
updates = apply_outcome_pg(graph, fired_nodes=["a", "b", "c"], outcome=1.0, config=config)
```

Full derivation: https://jonathangu.com/crabpath/gu2016/

## Write policy summary

| Situation | Action |
|-----------|--------|
| Durable fact | Edit file → sync re-embeds |
| Correction | Edit file + learn_correction.py |
| Soft teaching | crabpath inject --type TEACHING |
| Wrong retrieval | learn_correction.py (graph-only) |
| New rule | Edit AGENTS.md or SOUL.md |

## Production stats (current)

- MAIN: 1,160 nodes, 2,551 edges, 43 learnings
- PELICAN: 555 nodes, 2,211 edges, 181 learnings
- BOUNTIFUL: 289 nodes, 1,101 edges, 35 learnings
- CORMORANT: 1,672 nodes, ~7,100 edges, 22 learnings (first external user!)

## Traversal defaults

| Setting | Default | Purpose |
|---------|---------|---------|
| `beam_width` | `8` | Frontier size per hop (wider = reaches farther routes) |
| `max_hops` | `30` | Safety ceiling; damping controls convergence |
| `fire_threshold` | `0.01` | Minimum score required to fire a candidate node |
| `reflex_threshold` | `0.6` | Edges with weight `>= 0.6` auto-follow (no route function) |
| `habitual_range` | `0.2 - 0.6` | Edges in this band run through route function |
| `inhibitory_threshold` | `-0.01` | Edges at or below suppress targets |
| `max_fired_nodes` | `None` | Hard stop on fired node count |
| `max_context_chars` | `None` | Hard stop on rendered traversal context |
| `edge_damping` | `0.3` | Per-reuse decay (`weight × 0.3^k`) |

```python
from crabpath import traverse, TraversalConfig

result = traverse(
    graph,
    seeds,
    config=TraversalConfig(max_context_chars=20000, max_fired_nodes=30),
)
```

`query` and `query_brain.py` honor these budgets and stop as soon as any termination condition is met.

## External benchmarks

External retrieval benchmarks are optional and use separately downloaded datasets.
CrabPath ships a quick-start workflow for MultiHop-RAG and HotPotQA in
`benchmarks/external/README.md`, but the datasets are not in the repository.

Quick run (from project root):

```bash
mkdir -p benchmarks/external
curl -L https://huggingface.co/datasets/yixuantt/MultiHopRAG/raw/main/MultiHopRAG.json -o benchmarks/external/multihop_rag.json
curl -L https://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json -o benchmarks/external/hotpotqa_dev_distractor.json
python3 benchmarks/external/run_multihop_rag.py --limit 50
python3 benchmarks/external/run_hotpotqa.py --limit 50
```

## Python API

```python
from crabpath import (
    split_workspace,
    traverse,
    apply_outcome,
    inject_node,
    inject_correction,
    inject_batch,
    VectorIndex,
    HashEmbedder,
    TraversalConfig,
    save_state,
    load_state,
    ManagedState,
    measure_health,
    replay_queries,
    score_retrieval,
)
```

## State lifecycle

- **Where it lives:** a single `state.json` file (portable, version-controllable)
- **How big:** ~180KB for 20 nodes (hash), ~60MB for 1,600 nodes (OpenAI embeddings)
- **When to rebuild:** after major workspace restructuring or embedder changes
- **Embedder changes:** CrabPath stores the embedder name + dimension in state metadata and hard-fails on mismatch — no silent corruption
- **Merging:** use `crabpath merge` to consolidate similar nodes as the graph grows

## Cost control

- **Free tier:** hash embeddings work offline with zero API calls. Good for trying CrabPath and small workspaces.
- **Budget tier:** use OpenAI `text-embedding-3-small` (~$0.02/1M tokens). Embed once at init, cache in state.json.
- **LLM routing:** optional. `gpt-5-mini` for routing/scoring decisions. Only called during query, not at rest.
- **Batch init:** `crabpath init` embeds all workspace files in one batch call. Subsequent queries reuse cached vectors.
- **Upgrade path:** start with hash, switch to real embeddings later by rebuilding state with `crabpath init`.

## Optional: warm start from sessions

If you have prior conversation logs, replay them:

```bash
crabpath replay --state /tmp/brain/state.json --sessions ./sessions/
```

Skip this if you are just getting started.

## Production experience

Three brains run in production on a Mac Mini M4 Pro:

| Brain | Nodes | Edges | Learning Corrections | Sessions Replayed |
|-------|-------|-------|---------------------|-------------------|
| MAIN | 1,142 | 2,814 | 43 | 215 |
| PELICAN | 512 | 1,984 | 181 | 183 |
| BOUNTIFUL | 273 | 1,073 | 35 | 300 |

## Design Tenets

- No network calls in core.
- No secret discovery (no dotfiles, no keychain lookup).
- Embedder identity stored in state metadata; hard-fail on dimension mismatch.
- One canonical state format (`state.json`).
- Traversal defaults are budget-first for safety: `beam_width=8`, `max_hops=30`, `fire_threshold=0.01`.

## Paper + links

[jonathangu.com/crabpath](https://jonathangu.com/crabpath/) — 8 deterministic simulations + production deployment data.

- PyPI: `pip install crabpath`
- GitHub: [jonathangu/crabpath](https://github.com/jonathangu/crabpath)
- ClawHub: `clawhub install crabpath`
- Benchmarks: `python3 benchmarks/run_benchmark.py` (deterministic per commit; timings vary by machine)


path: REPRODUCE.md
content: # Reproduce CrabPath Results

All paper claims are verified by deterministic simulations and one benchmark harness. No API keys are needed for simulation.

## Quick Start

```bash
git clone https://github.com/jonathangu/crabpath.git
cd crabpath
pip install -e .
python sims/run_all.py
```

## Expected Output

```
simulation                 claim        summary
-------------------------------------------------------------------------------
deploy_pipeline            PASS         final=1.000
negation                   PASS         bad=-0.940
context_reduction          PASS         30.0->2.6
forgetting                 PASS         dormant=0.933
edge_damping               PASS         dr=True ur=False
domain_separation          PASS         cross=5
brain_death                PASS         dormant=0.975
individuation              PASS         diff_edges=27
```

All eight simulations are deterministic and all PASS.

## Table: Simulations → Paper Claims

| Sim | Paper Claim | What It Tests |
|-----|-------------|---------------|
| `deploy_pipeline` | Repeated edges compile to reflex | 50 queries, 4 pipeline edges, reflex-tier convergence |
| `negation` | Corrective negatives suppress bad guidance | Inhibitory learning (`-0.940`) on stale edge |
| `context_reduction` | Context becomes focused over time | 30→2.6 fired nodes |
| `forgetting` | Selective dormancy is learned | Dormant shares reach 93.3% |
| `edge_damping` | Damped traversal avoids loops | `λ=0.3` reaches target; `λ=1.0` loops |
| `domain_separation` | Sparse bridges while preserving clusters | 5 cross-file edges emerge |
| `brain_death` | Health checks surface recovery pressure | Dormancy and recovery control behavior |
| `individuation` | Same topology with workload-specific weights | 27 edges differ by >0.05 after divergent workloads |

## Run Tests

```bash
pip install -e .
python3 -m pytest tests/ -x -q
```

Expected: 204 passed.

## Benchmarks

CrabPath includes a deterministic benchmark harness at `benchmarks/run_benchmark.py`.

### Run

```bash
python3 benchmarks/run_benchmark.py
python3 -m pytest tests/ -x -q
```

Benchmark output compares:

1. keyword overlap
2. hash embedding
3. CrabPath traversal
4. CrabPath traversal + session replay

Run `python3 benchmarks/run_benchmark.py` to see current deterministic results for this commit.

## External benchmarks (optional, dataset downloads required)

Benchmarks for external corpora are documented in `benchmarks/external/README.md`.
The external datasets are optional and are **not included** in this repository.

Download the datasets and run a quick end-to-end check:

```bash
mkdir -p benchmarks/external
curl -L https://huggingface.co/datasets/yixuantt/MultiHopRAG/raw/main/MultiHopRAG.json -o benchmarks/external/multihop_rag.json
curl -L https://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json -o benchmarks/external/hotpotqa_dev_distractor.json
python3 benchmarks/external/run_multihop_rag.py --limit 50
python3 benchmarks/external/run_hotpotqa.py --limit 50
```

For full benchmark runs and tuning options, see `benchmarks/external/README.md`.

## Live Injection Verification

Live injection is covered by `tests/test_inject.py` and can also be exercised directly with a simple state payload.

```bash
python3 -m pytest tests/test_inject.py -x -q
```

Run an end-to-end CLI smoke test:

```bash
python3 - <<'PY'
from pathlib import Path

from crabpath import VectorIndex, save_state, HashEmbedder, inject_node
from crabpath.graph import Graph, Node

graph = Graph()
index = VectorIndex()
graph.add_node(Node("seed", "Seed policy guidance", metadata={"file": "seed.md"}))
index.upsert("seed", HashEmbedder().embed("Seed policy guidance"))

state_path = Path("/tmp/crabpath_live_inject_state.json")
save_state(graph=graph, index=index, path=state_path)

result = inject_node(
    graph=graph,
    index=index,
    node_id="learning::manual",
    content="Prefer deterministic tests over assumptions.",
    connect_top_k=1,
    connect_min_sim=0.0,
)

save_state(graph=graph, index=index, path=state_path)
print(result)
print(f"node_count={graph.node_count()}")
print(f"edges={graph.edge_count()}")
PY

crabpath inject \
  --state /tmp/crabpath_live_inject_state.json \
  --id learning::manual2 \
  --content "Never expose secrets in plain text." \
  --type TEACHING \
  --json \
  --connect-min-sim 0.0
```


path: benchmarks/external/README.md
content: # External benchmarks

These scripts run the two external corpora used in repo papers and internal experiments.

## Datasets

* MultiHop-RAG (`multihop_rag.json`) for 2556 multi-hop queries across web and news corpora.
* HotPotQA dev distractor (`hotpotqa_dev_distractor.json`) for explainable two-hop QA with supporting facts.

## Download

```bash
mkdir -p benchmarks/external
curl -L https://huggingface.co/datasets/yixuantt/MultiHopRAG/raw/main/MultiHopRAG.json -o benchmarks/external/multihop_rag.json
curl -L https://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json -o benchmarks/external/hotpotqa_dev_distractor.json
```

## Quick vs full runs

Quick run (default limits):
```bash
python3 benchmarks/external/run_multihop_rag.py
python3 benchmarks/external/run_hotpotqa.py
```

Full run:
```bash
python3 benchmarks/external/run_multihop_rag.py --limit 2556
python3 benchmarks/external/run_hotpotqa.py --limit 0
```

## Expected outputs

* `benchmarks/external/*_results.json` (run output)
* Cached query/node embeddings written to: `benchmarks/external/.embedding_cache_*.json`

## Notes

- Set `--limit` smaller for faster smoke tests. Example: `--limit 50`.
- Use `--seed-top-k`, `--embed-top-k`, `--max-hops`, and `--beam-width` to tune traversal and embedding breadth.
- Embeddings are the only expected paid API calls (`text-embedding-3-small`). Query and traversal phases are in-memory once cache is populated.
```


path: benchmarks/external/run_hotpotqa.py
content: from __future__ import annotations

import argparse
import json
import os
import re
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

from openai import OpenAI

from crabpath import Edge, Graph, LearningConfig, Node, TraversalConfig, VectorIndex, apply_outcome, apply_outcome_pg, traverse


EMBED_MODEL = "text-embedding-3-small"
CACHE_PATH = Path("benchmarks/external/.embedding_cache_hotpot.json")
LEARNING_CONFIG = LearningConfig(learning_rate=0.3, discount=0.9)
DEFAULT_EMBED_TOP_K = 10
DEFAULT_SEED_TOP_K = 8
DEFAULT_MAX_HOPS = 3
DEFAULT_BEAM_WIDTH = 4
DEFAULT_LIMIT = 500
CHECKPOINTS = [50, 100, 200, 500]
RUN_METRICS = ["sp_recall_at_10", "sp_recall_at_5", "mrr", "nodes_fired"]


def _slugify(title: str) -> str:
    return re.sub(r"[^a-z0-9]+", "_", title.lower()).strip("_")[:80]


def _load_json_list(path: Path) -> list[dict[str, Any]]:
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, list):
        raise SystemExit(f"Expected a JSON list: {path}")
    return data


def _load_dataset(path: Path, limit: int = 500) -> list[dict[str, Any]]:
    rows = _load_json_list(path)
    if limit <= 0:
        return rows
    return rows[:limit]


def _add_or_boost_edge(graph: Graph, source_id: str, target_id: str, weight: float) -> None:
    if source_id == target_id:
        return
    current = graph._edges.get(source_id, {}).get(target_id)
    if current is None or weight > current.weight:
        graph.add_edge(Edge(source=source_id, target=target_id, weight=weight, kind="sibling"))


def _build_graph(rows: list[dict[str, Any]]) -> tuple[Graph, dict[str, str]]:
    graph = Graph()
    title_to_id: dict[str, str] = {}

    question_context_nodes: list[set[str]] = []
    for row in rows:
        context_ids: set[str] = set()
        for paragraph in row.get("context", []) if isinstance(row.get("context"), list) else []:
            if not isinstance(paragraph, list) or len(paragraph) < 2:
                continue
            title = str(paragraph[0]).strip()
            if not title:
                continue
            node_id = _slugify(title)
            sentences = [str(sentence).strip() for sentence in paragraph[1] if str(sentence).strip()]
            content = " ".join(sentences)
            if node_id not in title_to_id.values():
                graph.add_node(Node(id=node_id, content=content, metadata={"title": title}))
                title_to_id[title] = node_id
            context_ids.add(node_id)
        question_context_nodes.append(context_ids)

    for row_context_ids in question_context_nodes:
        ids = sorted(row_context_ids)
        for i in range(len(ids)):
            for j in range(i + 1, len(ids)):
                _add_or_boost_edge(graph, ids[i], ids[j], 0.3)
                _add_or_boost_edge(graph, ids[j], ids[i], 0.3)

    for row in rows:
        titles = [
            str(fact[0]).strip()
            for fact in row.get("supporting_facts", []) if isinstance(fact, list) and fact
            if str(fact[0]).strip() in title_to_id
        ]
        title_ids = [title_to_id[title] for title in dict.fromkeys(titles)]
        for i in range(len(title_ids)):
            for j in range(i + 1, len(title_ids)):
                _add_or_boost_edge(graph, title_ids[i], title_ids[j], 0.5)
                _add_or_boost_edge(graph, title_ids[j], title_ids[i], 0.5)

    return graph, title_to_id


def _clone_graph(graph: Graph) -> Graph:
    copy = Graph()
    copy._nodes = {node_id: node for node_id, node in graph._nodes.items()}
    copy._edges = {source: dict(targets) for source, targets in graph._edges.items()}
    return copy


def _load_cache(path: Path) -> dict[str, list[float]]:
    if not path.exists():
        return {}
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict):
        return {}
    return {str(text): list(vec) for text, vec in data.items() if isinstance(text, str) and isinstance(vec, list)}


def _save_cache(path: Path, cache: dict[str, list[float]]) -> None:
    path.write_text(json.dumps(cache), encoding="utf-8")


def _chunked(values: list[str], size: int) -> list[list[str]]:
    return [values[i : i + size] for i in range(0, len(values), size)]


def _embed_batch(client: OpenAI | None, texts: list[str]) -> list[list[float]]:
    if client is None:
        raise SystemExit("OPENAI_API_KEY is required to create missing embeddings")
    try:
        response = client.embeddings.create(model=EMBED_MODEL, input=texts)
        return [list(record.embedding) for record in response.data]
    except Exception:
        time.sleep(2)
        response = client.embeddings.create(model=EMBED_MODEL, input=texts)
        return [list(record.embedding) for record in response.data]


def _cache_embeddings(client: OpenAI | None, cache: dict[str, list[float]], texts: list[str]) -> None:
    pending = list(dict.fromkeys([text for text in texts if text and text not in cache]))
    if not pending:
        return
    for chunk in _chunked(pending, 100):
        vectors = _embed_batch(client, chunk)
        for text, vector in zip(chunk, vectors):
            cache[text] = vector


def _prepare_cache(cache: dict[str, list[float]], texts: list[str], client: OpenAI | None) -> dict[str, list[float]]:
    pending = list(dict.fromkeys([text for text in texts if text and text not in cache]))
    if pending:
        if client is None:
            raise SystemExit("OPENAI_API_KEY is required to create missing embeddings")
        _cache_embeddings(client, cache, pending)
        _save_cache(CACHE_PATH, cache)
    return cache


def _required_texts(graph: Graph, rows: list[dict[str, Any]]) -> list[str]:
    texts = [node.content for node in graph.nodes() if node.content]
    queries = [str(row.get("question", "")) for row in rows if str(row.get("question", ""))]
    return list(dict.fromkeys([text for text in texts + queries if text]))


def _build_embeddings(client: OpenAI | None, graph: Graph) -> dict[str, list[float]]:
    cache = _load_cache(CACHE_PATH)
    _prepare_cache(cache, [node.content for node in graph.nodes()], client)
    _save_cache(CACHE_PATH, cache)
    return cache


def _ensure_query_embedding(client: OpenAI | None, cache: dict[str, list[float]], query: str) -> list[float]:
    if query not in cache:
        if client is None:
            raise SystemExit("OPENAI_API_KEY is required to create missing query embeddings")
        _cache_embeddings(client, cache, [query])
        _save_cache(CACHE_PATH, cache)
    vector = cache.get(query)
    if vector is None:
        raise SystemExit(f"could not embed query: {query}")
    return vector


def _build_index_from_nodes(graph: Graph, cache: dict[str, list[float]]) -> VectorIndex:
    index = VectorIndex()
    for node in graph.nodes():
        vector = cache.get(node.content)
        if vector is None:
            raise SystemExit(f"missing embedding for node {node.id}")
        index.upsert(node.id, vector)
    return index


def _ground_truth_ids(row: dict[str, Any], title_to_id: dict[str, str]) -> set[str]:
    ids: set[str] = set()
    for fact in row.get("supporting_facts", []) if isinstance(row.get("supporting_facts"), list) else []:
        if not isinstance(fact, list) or not fact:
            continue
        title = str(fact[0]).strip()
        node_id = title_to_id.get(title)
        if node_id:
            ids.add(node_id)
    return ids


def _row_metrics(retrieved: list[str], relevant: set[str]) -> dict[str, float]:
    top5 = retrieved[:5]
    top10 = retrieved[:10]
    recall_at_5 = 1.0 if relevant.issubset(set(top5)) else 0.0
    recall_at_10 = 1.0 if relevant.issubset(set(top10)) else 0.0
    single_hit_5 = 1.0 if set(top5).intersection(relevant) else 0.0

    denominator = float(max(len(top10), 1))
    distractor_suppression = (len(top10) - len([node_id for node_id in top10 if node_id in relevant])) / denominator

    mrr = 0.0
    for rank, node_id in enumerate(retrieved, start=1):
        if node_id in relevant:
            mrr = 1.0 / rank
            break

    return {
        "sp_recall_at_5": recall_at_5,
        "sp_recall_at_10": recall_at_10,
        "single_sp_hit_at_5": single_hit_5,
        "distractor_suppression": distractor_suppression,
        "mrr": mrr,
        "nodes_fired": float(len(retrieved)),
    }


def _avg(metrics: list[dict[str, float]]) -> dict[str, float]:
    if not metrics:
        return {
            "sp_recall_at_5": 0.0,
            "sp_recall_at_10": 0.0,
            "single_sp_hit_at_5": 0.0,
            "distractor_suppression": 0.0,
            "mrr": 0.0,
            "nodes_fired": 0.0,
        }
    count = len(metrics)
    return {key: sum(row[key] for row in metrics) / count for key in metrics[0]}


def _log_progress(method_name: str, processed: int, metrics: dict[str, float], keys: list[str]) -> None:
    print(
        f"[{method_name}] processed={processed} "
        + ", ".join(f"{key}={metrics[key]:.3f}" for key in keys if key in metrics)
    )


def _embedding_topk(
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    query: str,
    embed_top_k: int,
) -> list[str]:
    vector = _ensure_query_embedding(client, cache, query)
    return [node_id for node_id, _ in index.search(vector, top_k=embed_top_k)]


def _traverse_topk(
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    query: str,
    seed_top_k: int,
    traversal_config: TraversalConfig,
) -> list[str]:
    vector = _ensure_query_embedding(client, cache, query)
    seeds = [(node_id, score) for node_id, score in index.search(vector, top_k=seed_top_k)]
    if not seeds:
        return []
    return traverse(graph=graph, seeds=seeds, config=traversal_config, query_text=query).fired


def _run_static(
    rows: list[dict[str, Any]],
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    title_to_id: dict[str, str],
    use_traversal: bool,
    method_name: str,
    embed_top_k: int,
    seed_top_k: int,
    traversal_config: TraversalConfig,
) -> dict[str, float]:
    collected = []
    for idx, row in enumerate(rows, start=1):
        query = str(row.get("question", ""))
        retrieved = (
            _traverse_topk(graph, index, client, cache, query, seed_top_k=seed_top_k, traversal_config=traversal_config)
            if use_traversal
            else _embedding_topk(graph, index, client, cache, query, embed_top_k=embed_top_k)
        )
        collected.append(_row_metrics(retrieved, _ground_truth_ids(row, title_to_id)))
        if idx % 100 == 0:
            _log_progress(method_name, idx, _avg(collected), RUN_METRICS)
    return _avg(collected)


def _run_learning(
    rows: list[dict[str, Any]],
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    title_to_id: dict[str, str],
    use_pg: bool,
    method_name: str,
    seed_top_k: int,
    traversal_config: TraversalConfig,
) -> tuple[dict[str, float], dict[str, dict[str, float]]]:
    history: list[dict[str, float]] = []
    checkpoints = {point for point in CHECKPOINTS if point <= len(rows)}
    learning_curve: dict[str, dict[str, float]] = {}

    for idx, row in enumerate(rows, start=1):
        query = str(row.get("question", ""))
        relevant = _ground_truth_ids(row, title_to_id)
        retrieved = _traverse_topk(graph, index, client, cache, query, seed_top_k=seed_top_k, traversal_config=traversal_config)
        scores = _row_metrics(retrieved, relevant)
        history.append(scores)

        outcome = 1.0 if scores["sp_recall_at_10"] >= 1.0 else -1.0
        if use_pg:
            apply_outcome_pg(graph=graph, fired_nodes=retrieved, outcome=outcome, config=LEARNING_CONFIG, temperature=1.0)
        else:
            apply_outcome(graph=graph, fired_nodes=retrieved, outcome=outcome, config=LEARNING_CONFIG)

        if idx in checkpoints:
            learning_curve[str(idx)] = _avg(history)
        if idx % 100 == 0:
            _log_progress(method_name, idx, _avg(history), RUN_METRICS)

    return _avg(history), learning_curve


def _print_ascii(methods: dict[str, dict[str, float]], learning_curves: dict[str, dict[str, dict[str, float]]]) -> None:
    print("\nMethod                   SP@5  SP@10  SingleHit@5  Distractor  MRR   NodesFired")
    print("----------------------------------------------------------------------------")
    for key in ["embedding_topk", "crabpath_cold", "crabpath_learning", "crabpath_pg_learning"]:
        values = methods[key]
        print(
            f"{key:<24}"
            f"{values['sp_recall_at_5']:.3f}  "
            f"{values['sp_recall_at_10']:.3f}    "
            f"{values['single_sp_hit_at_5']:.3f}       "
            f"{values['distractor_suppression']:.3f}     "
            f"{values['mrr']:.3f}   "
            f"{values['nodes_fired']:.2f}"
        )

    print("\nLearning Curves")
    for name in ["crabpath_learning", "crabpath_pg_learning"]:
        print(f"\n{name}")
        print("query  SP@5  SP@10  SingleHit@5  Distractor  MRR   NodesFired")
        for checkpoint in sorted(learning_curves.get(name, {}), key=lambda value: int(value)):
            values = learning_curves[name][checkpoint]
            print(
                f"{checkpoint:>5} "
                f"{values['sp_recall_at_5']:.3f}  "
                f"{values['sp_recall_at_10']:.3f}    "
                f"{values['single_sp_hit_at_5']:.3f}       "
                f"{values['distractor_suppression']:.3f}     "
                f"{values['mrr']:.3f}   "
                f"{values['nodes_fired']:.2f}"
            )


def main() -> None:
    parser = argparse.ArgumentParser(description="Run HotPotQA benchmark.")
    parser.add_argument(
        "--input",
        default="benchmarks/external/hotpotqa_dev_distractor.json",
        help="Path to HotPotQA dev distractor JSON",
    )
    parser.add_argument(
        "--output",
        default="benchmarks/external/hotpotqa_results.json",
        help="Path to write benchmark output JSON",
    )
    parser.add_argument("--limit", type=int, default=DEFAULT_LIMIT, help="Number of dataset queries to evaluate")
    parser.add_argument(
        "--seed-top-k",
        type=int,
        default=DEFAULT_SEED_TOP_K,
        help="Seed top-k for graph traversal",
    )
    parser.add_argument(
        "--embed-top-k",
        type=int,
        default=DEFAULT_EMBED_TOP_K,
        help="Top-k for retrieval-only embedding baseline",
    )
    parser.add_argument(
        "--max-hops",
        type=int,
        default=DEFAULT_MAX_HOPS,
        help="Traversal max_hops",
    )
    parser.add_argument(
        "--beam-width",
        type=int,
        default=DEFAULT_BEAM_WIDTH,
        help="Traversal beam_width",
    )
    args = parser.parse_args()

    input_path = Path(args.input)
    if not input_path.exists():
        raise SystemExit(f"dataset not found: {input_path}")
    output_path = Path(args.output)

    rows = _load_dataset(input_path, limit=args.limit)
    graph, title_to_id = _build_graph(rows)
    if not graph.node_count():
        raise SystemExit("no nodes were built from dataset")

    required_texts = _required_texts(graph, rows)
    cache = _load_cache(CACHE_PATH)
    missing = [text for text in required_texts if text not in cache]
    client: OpenAI | None = OpenAI() if missing else None
    if missing and not os.getenv("OPENAI_API_KEY"):
        raise SystemExit("OPENAI_API_KEY is required to build missing embeddings")
    cache = _prepare_cache(cache, required_texts, client)
    index = _build_index_from_nodes(graph, cache)
    traversal_config = TraversalConfig(max_hops=args.max_hops, beam_width=args.beam_width)

    methods = {}
    learning_curves: dict[str, dict[str, dict[str, float]]] = {}

    methods["embedding_topk"] = _run_static(
        rows=rows,
        graph=graph,
        index=index,
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_traversal=False,
        method_name="embedding_topk",
        embed_top_k=args.embed_top_k,
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    cold_graph = _clone_graph(graph)
    methods["crabpath_cold"] = _run_static(
        rows=rows,
        graph=cold_graph,
        index=_build_index_from_nodes(cold_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_traversal=True,
        method_name="crabpath_cold",
        embed_top_k=args.embed_top_k,
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    learning_graph = _clone_graph(graph)
    methods["crabpath_learning"], learning_curves["crabpath_learning"] = _run_learning(
        rows=rows,
        graph=learning_graph,
        index=_build_index_from_nodes(learning_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_pg=False,
        method_name="crabpath_learning",
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    pg_graph = _clone_graph(graph)
    methods["crabpath_pg_learning"], learning_curves["crabpath_pg_learning"] = _run_learning(
        rows=rows,
        graph=pg_graph,
        index=_build_index_from_nodes(pg_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_pg=True,
        method_name="crabpath_pg_learning",
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    _print_ascii(methods, learning_curves)

    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(
        json.dumps(
            {
                "generated_at": datetime.now(timezone.utc).isoformat(),
                "dataset": "HotPotQA",
                "dataset_path": str(input_path),
                "dataset_queries": len(rows),
                "graph_nodes": graph.node_count(),
                "graph_edges": graph.edge_count(),
                "cache_path": str(CACHE_PATH),
                "parameters": {
                    "limit": args.limit,
                    "seed_top_k": args.seed_top_k,
                    "embed_top_k": args.embed_top_k,
                    "max_hops": args.max_hops,
                    "beam_width": args.beam_width,
                },
                "method_metrics": methods,
                "learning_curves": learning_curves,
                "checkpoints": CHECKPOINTS,
                "traversal": {"max_hops": traversal_config.max_hops, "beam_width": traversal_config.beam_width},
            },
            indent=2,
        ),
        encoding="utf-8",
    )
    print(f"\nSaved: {output_path}")


if __name__ == "__main__":
    main()


path: benchmarks/external/run_multihop_rag.py
content: from __future__ import annotations

import argparse
import json
import os
import re
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

from openai import OpenAI

from crabpath import Edge, Graph, LearningConfig, Node, TraversalConfig, VectorIndex, apply_outcome, apply_outcome_pg, traverse


EMBED_MODEL = "text-embedding-3-small"
CACHE_PATH = Path("benchmarks/external/.embedding_cache_mhrag.json")
LEARNING_CONFIG = LearningConfig(learning_rate=0.3, discount=0.9)
DEFAULT_EMBED_TOP_K = 10
DEFAULT_SEED_TOP_K = 8
DEFAULT_MAX_HOPS = 3
DEFAULT_BEAM_WIDTH = 4
DEFAULT_LIMIT = 1000
CHECKPOINTS = [100, 250, 500, 1000, 2556]
RUN_METRICS = ["full_hit_rate", "partial_hit_rate", "evidence_recall_at_10", "mrr", "nodes_fired"]


def _slugify(title: str) -> str:
    return re.sub(r"[^a-z0-9]+", "_", title.lower()).strip("_")[:80]


def _load_json_list(path: Path) -> list[dict[str, Any]]:
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, list):
        raise SystemExit(f"Expected a JSON list: {path}")
    return data


def _load_dataset(path: Path, limit: int) -> list[dict[str, Any]]:
    rows = _load_json_list(path)
    filtered = [row for row in rows if str(row.get("question_type", "")) != "null_query"]
    if limit <= 0:
        return filtered
    return filtered[:limit]


def _add_or_boost_edge(graph: Graph, source_id: str, target_id: str, weight: float) -> None:
    if source_id == target_id:
        return
    current = graph._edges.get(source_id, {}).get(target_id)
    if current is None or weight > current.weight:
        graph.add_edge(Edge(source=source_id, target=target_id, weight=weight, kind="sibling"))


def _build_graph(rows: list[dict[str, Any]]) -> tuple[Graph, dict[str, str], list[str]]:
    graph = Graph()
    title_to_id: dict[str, str] = {}
    source_to_nodes: dict[str, set[str]] = {}
    category_to_nodes: dict[str, set[str]] = {}

    for row in rows:
        for evidence in row.get("evidence_list", []) if isinstance(row.get("evidence_list"), list) else []:
            if not isinstance(evidence, dict):
                continue
            title = str(evidence.get("title", "")).strip()
            if not title:
                continue
            node_id = _slugify(title)
            fact = str(evidence.get("fact", "")).strip()
            source = str(evidence.get("source", "")).strip()
            category = str(evidence.get("category", "")).strip()
            if node_id not in title_to_id.values():
                graph.add_node(
                    Node(
                        id=node_id,
                        content=fact,
                        metadata={"title": title, "source": source, "category": category},
                    )
                )
                title_to_id[title] = node_id
            else:
                node = graph.get_node(node_id)
                if node is not None and not node.content and fact:
                    node.content = fact

            if source:
                source_to_nodes.setdefault(source, set()).add(node_id)
            if category:
                category_to_nodes.setdefault(category, set()).add(node_id)

    for group in source_to_nodes.values():
        ids = sorted(group)
        for i in range(len(ids)):
            for j in range(i + 1, len(ids)):
                _add_or_boost_edge(graph, ids[i], ids[j], 0.35)
                _add_or_boost_edge(graph, ids[j], ids[i], 0.35)

    for group in category_to_nodes.values():
        ids = sorted(group)
        for i in range(len(ids)):
            for j in range(i + 1, len(ids)):
                _add_or_boost_edge(graph, ids[i], ids[j], 0.35)
                _add_or_boost_edge(graph, ids[j], ids[i], 0.35)

    for row in rows:
        evidence_ids = sorted(
            {
                title_to_id[str(evidence.get("title", "")).strip()]
                for evidence in row.get("evidence_list", []) if isinstance(evidence, dict)
                if str(evidence.get("title", "")).strip() in title_to_id
            }
        )
        for i in range(len(evidence_ids)):
            for j in range(i + 1, len(evidence_ids)):
                _add_or_boost_edge(graph, evidence_ids[i], evidence_ids[j], 0.5)
                _add_or_boost_edge(graph, evidence_ids[j], evidence_ids[i], 0.5)

    return graph, title_to_id, list(title_to_id.keys())


def _clone_graph(graph: Graph) -> Graph:
    copy = Graph()
    copy._nodes = {node_id: node for node_id, node in graph._nodes.items()}
    copy._edges = {source: dict(targets) for source, targets in graph._edges.items()}
    return copy


def _build_index_from_nodes(graph: Graph, cache: dict[str, list[float]]) -> VectorIndex:
    index = VectorIndex()
    for node in graph.nodes():
        vector = cache.get(node.content)
        if vector is None:
            raise SystemExit(f"missing embedding for node {node.id}")
        index.upsert(node.id, vector)
    return index


def _load_cache(path: Path) -> dict[str, list[float]]:
    if not path.exists():
        return {}
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict):
        return {}
    return {str(text): list(vec) for text, vec in data.items() if isinstance(text, str) and isinstance(vec, list)}


def _save_cache(path: Path, cache: dict[str, list[float]]) -> None:
    path.write_text(json.dumps(cache), encoding="utf-8")


def _chunked(values: list[str], size: int) -> list[list[str]]:
    return [values[i : i + size] for i in range(0, len(values), size)]


def _embed_batch(client: OpenAI | None, texts: list[str]) -> list[list[float]]:
    if client is None:
        raise SystemExit("OPENAI_API_KEY is required to create missing embeddings")
    try:
        response = client.embeddings.create(model=EMBED_MODEL, input=texts)
        return [list(record.embedding) for record in response.data]
    except Exception:
        time.sleep(2)
        response = client.embeddings.create(model=EMBED_MODEL, input=texts)
        return [list(record.embedding) for record in response.data]


def _cache_embeddings(client: OpenAI | None, cache: dict[str, list[float]], texts: list[str]) -> None:
    pending = list(dict.fromkeys([text for text in texts if text and text not in cache]))
    if not pending:
        return
    for chunk in _chunked(pending, 100):
        vectors = _embed_batch(client, chunk)
        for text, vector in zip(chunk, vectors):
            cache[text] = vector


def _prepare_cache(cache: dict[str, list[float]], texts: list[str], client: OpenAI | None) -> dict[str, list[float]]:
    pending = list(dict.fromkeys([text for text in texts if text and text not in cache]))
    if pending:
        if client is None:
            raise SystemExit("OPENAI_API_KEY is required to create missing embeddings")
        _cache_embeddings(client, cache, pending)
        _save_cache(CACHE_PATH, cache)
    return cache


def _required_texts(graph: Graph, rows: list[dict[str, Any]]) -> list[str]:
    texts = [node.content for node in graph.nodes() if node.content]
    queries = [str(row.get("query", "")) for row in rows if str(row.get("query", ""))]
    return list(dict.fromkeys([text for text in texts + queries if text]))


def _build_embeddings(client: OpenAI | None, graph: Graph) -> dict[str, list[float]]:
    cache = _load_cache(CACHE_PATH)
    _prepare_cache(cache, [node.content for node in graph.nodes()], client)
    _save_cache(CACHE_PATH, cache)
    return cache


def _ensure_query_embedding(client: OpenAI | None, cache: dict[str, list[float]], query: str) -> list[float]:
    if query not in cache:
        if client is None:
            raise SystemExit("OPENAI_API_KEY is required to create missing query embeddings")
        _cache_embeddings(client, cache, [query])
        _save_cache(CACHE_PATH, cache)
    vector = cache.get(query)
    if vector is None:
        raise SystemExit(f"could not embed query: {query}")
    return vector


def _ground_truth_ids(row: dict[str, Any], title_to_id: dict[str, str]) -> set[str]:
    ids: set[str] = set()
    for evidence in row.get("evidence_list", []) if isinstance(row.get("evidence_list"), list) else []:
        if not isinstance(evidence, dict):
            continue
        title = str(evidence.get("title", "")).strip()
        if not title:
            continue
        node_id = title_to_id.get(title)
        if node_id:
            ids.add(node_id)
    return ids


def _row_metrics(retrieved: list[str], relevant: set[str], embed_top_k: int) -> dict[str, float]:
    top10 = retrieved[:min(embed_top_k, 10)]
    relevant_found_top10 = len([node_id for node_id in top10 if node_id in relevant])
    full_hit = 1.0 if relevant.issubset(set(retrieved)) else 0.0
    partial_hit = 1.0 if set(retrieved).intersection(relevant) else 0.0
    recall_at_10 = relevant_found_top10 / len(relevant) if relevant else 0.0
    mrr = 0.0
    for rank, node_id in enumerate(retrieved, start=1):
        if node_id in relevant:
            mrr = 1.0 / rank
            break
    return {
        "full_hit_rate": full_hit,
        "partial_hit_rate": partial_hit,
        "evidence_recall_at_10": recall_at_10,
        "mrr": mrr,
        "nodes_fired": float(len(retrieved)),
    }


def _avg(metrics: list[dict[str, float]]) -> dict[str, float]:
    if not metrics:
        return {"full_hit_rate": 0.0, "partial_hit_rate": 0.0, "evidence_recall_at_10": 0.0, "mrr": 0.0, "nodes_fired": 0.0}
    count = len(metrics)
    return {key: sum(row[key] for row in metrics) / count for key in metrics[0]}


def _log_progress(method_name: str, processed: int, metrics: dict[str, float], keys: list[str]) -> None:
    print(
        f"[{method_name}] processed={processed} "
        + ", ".join(f"{key}={metrics[key]:.3f}" for key in keys if key in metrics)
    )


def _embedding_topk(
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    query: str,
    embed_top_k: int,
) -> list[str]:
    vector = _ensure_query_embedding(client, cache, query)
    return [node_id for node_id, _ in index.search(vector, top_k=embed_top_k)]


def _traverse_topk(
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    query: str,
    seed_top_k: int,
    traversal_config: TraversalConfig,
) -> list[str]:
    vector = _ensure_query_embedding(client, cache, query)
    seeds = [(node_id, score) for node_id, score in index.search(vector, top_k=seed_top_k)]
    if not seeds:
        return []
    result = traverse(graph=graph, seeds=seeds, config=traversal_config, query_text=query)
    return result.fired


def _run_static(
    rows: list[dict[str, Any]],
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    title_to_id: dict[str, str],
    use_traversal: bool,
    method_name: str,
    embed_top_k: int,
    seed_top_k: int,
    traversal_config: TraversalConfig,
) -> dict[str, float]:
    per_query = []
    for idx, row in enumerate(rows, start=1):
        query = str(row.get("query", ""))
        retrieved = (
            _traverse_topk(graph, index, client, cache, query, seed_top_k=seed_top_k, traversal_config=traversal_config)
            if use_traversal
            else _embedding_topk(graph, index, client, cache, query, embed_top_k=embed_top_k)
        )
        per_query.append(_row_metrics(retrieved, _ground_truth_ids(row, title_to_id), embed_top_k=embed_top_k))
        if idx % 100 == 0:
            _log_progress(method_name, idx, _avg(per_query), RUN_METRICS)
    return _avg(per_query)


def _run_learning(
    rows: list[dict[str, Any]],
    graph: Graph,
    index: VectorIndex,
    client: OpenAI | None,
    cache: dict[str, list[float]],
    title_to_id: dict[str, str],
    use_pg: bool,
    method_name: str,
    seed_top_k: int,
    traversal_config: TraversalConfig,
    embed_top_k: int,
) -> tuple[dict[str, float], dict[str, dict[str, float]]]:
    history: list[dict[str, float]] = []
    checkpoints = {point for point in CHECKPOINTS if point <= len(rows)}
    learning_curve: dict[str, dict[str, float]] = {}

    for idx, row in enumerate(rows, start=1):
        query = str(row.get("query", ""))
        retrieved = _traverse_topk(graph, index, client, cache, query, seed_top_k=seed_top_k, traversal_config=traversal_config)
        relevant = _ground_truth_ids(row, title_to_id)
        scores = _row_metrics(retrieved, relevant, embed_top_k=embed_top_k)
        history.append(scores)

        outcome = 1.0 if scores["full_hit_rate"] >= 1.0 else -1.0
        if use_pg:
            apply_outcome_pg(graph=graph, fired_nodes=retrieved, outcome=outcome, config=LEARNING_CONFIG, temperature=1.0)
        else:
            apply_outcome(graph=graph, fired_nodes=retrieved, outcome=outcome, config=LEARNING_CONFIG)

        if idx in checkpoints:
            learning_curve[str(idx)] = _avg(history)
        if idx % 100 == 0:
            _log_progress(method_name, idx, _avg(history), RUN_METRICS)

    return _avg(history), learning_curve


def _print_ascii(methods: dict[str, dict[str, float]], learning_curves: dict[str, dict[str, dict[str, float]]]) -> None:
    print("\nMethod                  FullHit  Partial  Recall@10  MRR    NodesFired")
    print("-----------------------------------------------------------------------")
    for key in ["embedding_topk", "crabpath_cold", "crabpath_learning", "crabpath_pg_learning"]:
        values = methods[key]
        print(
            f"{key:<22}"
            f"{values['full_hit_rate']:.3f}    "
            f"{values['partial_hit_rate']:.3f}     "
            f"{values['evidence_recall_at_10']:.3f}     "
            f"{values['mrr']:.3f}   "
            f"{values['nodes_fired']:.2f}"
        )

    print("\nLearning Curves")
    for name in ["crabpath_learning", "crabpath_pg_learning"]:
        print(f"\n{name}")
        print("query  FullHit  Partial  Recall@10  MRR    NodesFired")
        for checkpoint in sorted(learning_curves.get(name, {}), key=lambda value: int(value)):
            values = learning_curves[name][checkpoint]
            print(
                f"{checkpoint:>5}  "
                f"{values['full_hit_rate']:.3f}    "
                f"{values['partial_hit_rate']:.3f}     "
                f"{values['evidence_recall_at_10']:.3f}     "
                f"{values['mrr']:.3f}   "
                f"{values['nodes_fired']:.2f}"
            )


def main() -> None:
    parser = argparse.ArgumentParser(description="Run MultiHop-RAG benchmark.")
    parser.add_argument(
        "--input",
        default="benchmarks/external/multihop_rag.json",
        help="Path to MultiHop-RAG dataset JSON",
    )
    parser.add_argument(
        "--output",
        default="benchmarks/external/multihop_rag_results.json",
        help="Path to write benchmark output JSON",
    )
    parser.add_argument("--limit", type=int, default=DEFAULT_LIMIT, help="Number of dataset queries to evaluate")
    parser.add_argument(
        "--seed-top-k",
        type=int,
        default=DEFAULT_SEED_TOP_K,
        help="Seed top-k for graph traversal",
    )
    parser.add_argument(
        "--embed-top-k",
        type=int,
        default=DEFAULT_EMBED_TOP_K,
        help="Top-k for retrieval-only embedding baseline",
    )
    parser.add_argument(
        "--max-hops",
        type=int,
        default=DEFAULT_MAX_HOPS,
        help="Traversal max_hops",
    )
    parser.add_argument(
        "--beam-width",
        type=int,
        default=DEFAULT_BEAM_WIDTH,
        help="Traversal beam_width",
    )
    args = parser.parse_args()

    input_path = Path(args.input)
    if not input_path.exists():
        raise SystemExit(f"dataset not found: {input_path}")
    output_path = Path(args.output)

    rows = _load_dataset(input_path, limit=args.limit)
    graph, title_to_id, titles = _build_graph(rows)
    if not graph.node_count():
        raise SystemExit("no nodes were built from dataset")

    required_texts = _required_texts(graph, rows)
    cache = _load_cache(CACHE_PATH)
    missing = [text for text in required_texts if text not in cache]
    client: OpenAI | None = OpenAI() if missing else None
    if missing and not os.getenv("OPENAI_API_KEY"):
        raise SystemExit("OPENAI_API_KEY is required to build missing embeddings")

    cache = _prepare_cache(cache, required_texts, client)
    index = _build_index_from_nodes(graph, cache)
    traversal_config = TraversalConfig(max_hops=args.max_hops, beam_width=args.beam_width)

    methods = {}
    learning_curves: dict[str, dict[str, dict[str, float]]] = {}

    methods["embedding_topk"] = _run_static(
        rows=rows,
        graph=graph,
        index=index,
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_traversal=False,
        method_name="embedding_topk",
        embed_top_k=args.embed_top_k,
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    cold_graph = _clone_graph(graph)
    methods["crabpath_cold"] = _run_static(
        rows=rows,
        graph=cold_graph,
        index=_build_index_from_nodes(cold_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_traversal=True,
        method_name="crabpath_cold",
        embed_top_k=args.embed_top_k,
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
    )

    learning_graph = _clone_graph(graph)
    methods["crabpath_learning"], learning_curves["crabpath_learning"] = _run_learning(
        rows=rows,
        graph=learning_graph,
        index=_build_index_from_nodes(learning_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_pg=False,
        method_name="crabpath_learning",
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
        embed_top_k=args.embed_top_k,
    )

    pg_graph = _clone_graph(graph)
    methods["crabpath_pg_learning"], learning_curves["crabpath_pg_learning"] = _run_learning(
        rows=rows,
        graph=pg_graph,
        index=_build_index_from_nodes(pg_graph, cache),
        client=client,
        cache=cache,
        title_to_id=title_to_id,
        use_pg=True,
        method_name="crabpath_pg_learning",
        seed_top_k=args.seed_top_k,
        traversal_config=traversal_config,
        embed_top_k=args.embed_top_k,
    )

    _print_ascii(methods, learning_curves)

    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(
        json.dumps(
            {
                "generated_at": datetime.now(timezone.utc).isoformat(),
                "dataset": "MultiHop-RAG",
                "dataset_path": str(input_path),
                "dataset_queries": len(rows),
                "titles_considered": len(titles),
                "graph_nodes": graph.node_count(),
                "graph_edges": graph.edge_count(),
                "cache_path": str(CACHE_PATH),
                "parameters": {
                    "limit": args.limit,
                    "seed_top_k": args.seed_top_k,
                    "embed_top_k": args.embed_top_k,
                    "max_hops": args.max_hops,
                    "beam_width": args.beam_width,
                },
                "method_metrics": methods,
                "learning_curves": learning_curves,
                "checkpoints": CHECKPOINTS,
                "traversal": {"max_hops": traversal_config.max_hops, "beam_width": traversal_config.beam_width},
            },
            indent=2,
        ),
        encoding="utf-8",
    )
    print(f"\nSaved: {output_path}")


if __name__ == "__main__":
    main()


path: benchmarks/queries.json
content: [
  {
    "query": "What is CrabPath and what is it designed for?",
    "relevant": ["README.md"]
  },
  {
    "query": "How does CrabPath stay pure with no network calls and zero Python dependencies by default?",
    "relevant": ["README.md", "SKILL.md", "examples/hash_embedder/README.md"]
  },
  {
    "query": "How do I split markdown workspace content into a graph of chunks?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "Show the quick start flow for split_workspace with HashEmbedder and VectorIndex.",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "How can I use the default hash-v1 embedder in a local workflow?",
    "relevant": ["README.md", "SKILL.md", "examples/hash_embedder/README.md"]
  },
  {
    "query": "How do I plug in OpenAI embeddings when indexing workspace files?",
    "relevant": ["README.md", "examples/openai_embedder/README.md"]
  },
  {
    "query": "How do I replay historical queries to warm start a graph?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "How do I initialize state from a workspace using CLI?",
    "relevant": ["README.md"]
  },
  {
    "query": "How do I run semantic query retrieval with index search plus graph traversal?",
    "relevant": ["README.md"]
  },
  {
    "query": "How do keyword-seeded retrieval and overlap scoring work?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "What CLI commands are available for query, learn, health, replay, merge, and connect?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "What is stored in CrabPath's canonical state.json format?",
    "relevant": ["README.md", "SKILL.md", "REPRODUCE.md"]
  },
  {
    "query": "Which simulation claims can be reproduced and what do they validate?",
    "relevant": ["REPRODUCE.md"]
  },
  {
    "query": "How is deterministic behavior guaranteed in CrabPath simulations and benchmarks?",
    "relevant": ["README.md", "REPRODUCE.md", "SKILL.md"]
  },
  {
    "query": "What does the hash embedder example tell me to run?",
    "relevant": ["examples/hash_embedder/README.md", "README.md"]
  },
  {
    "query": "How do I run the OpenAI embedder example and what env setup does it need?",
    "relevant": ["examples/openai_embedder/README.md", "README.md"]
  },
  {
    "query": "How can I use sessions as replay input when using query history?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "What is the default hash embedder name and vector dimension?",
    "relevant": ["README.md", "SKILL.md"]
  },
  {
    "query": "Where are the paper references and reproducibility claims documented?",
    "relevant": ["REPRODUCE.md"]
  },
  {
    "query": "Can I run all reproducibility checks from one command line script?",
    "relevant": ["REPRODUCE.md"]
  },
  {
    "query": "What is the exact initialization order in `crabpath init` when sessions are passed, including replay timing versus embedding/index creation?",
    "relevant": ["crabpath/cli.py", "crabpath/split.py", "crabpath/replay.py"]
  },
  {
    "query": "What is the query replay sequence in CLI and when does `last_replayed_ts` prevent duplicate session processing?",
    "relevant": ["crabpath/cli.py", "crabpath/replay.py"]
  },
  {
    "query": "How does `apply_outcome` react differently to a missing edge when outcome is negative versus positive?",
    "relevant": ["crabpath/learn.py", "crabpath/graph.py"]
  },
  {
    "query": "In the PG learner, how are taken edges and non-taken edges weighted when updating a traversal path?",
    "relevant": ["crabpath/learn.py", "crabpath/traverse.py"]
  },
  {
    "query": "What does `apply_outcome_pg` change compared with `apply_outcome` after `traverse` on the same fired route?",
    "relevant": ["crabpath/learn.py", "crabpath/traverse.py"]
  },
  {
    "query": "How are inhibitory edges activated by incoming suppression during seed scoring versus later expansion in traversal?",
    "relevant": ["crabpath/traverse.py"]
  },
  {
    "query": "What traversal candidates are skipped when tier is dormant or inhibitory, and what happens to suppression scores below the threshold?",
    "relevant": ["crabpath/traverse.py", "crabpath/learn.py"]
  },
  {
    "query": "How does edge_damping prevent runaway revisits when the same edge is traversed repeatedly in one query?",
    "relevant": ["crabpath/traverse.py", "README.md"]
  },
  {
    "query": "What are the exact stop conditions in traversal for max_fired_nodes, max_context_chars, and stop_expanding flags?",
    "relevant": ["crabpath/traverse.py", "README.md"]
  },
  {
    "query": "How is split chunking selected for `.md`, `.py`, `.json`, and config-like files during workspace split?",
    "relevant": ["crabpath/split.py"]
  },
  {
    "query": "Which paths and patterns does split_workspace skip, including defaults, CLI excludes, and parsed .gitignore rules?",
    "relevant": ["crabpath/split.py"]
  },
  {
    "query": "How are adjacent chunk sibling edges weighted and inserted when a file is split?",
    "relevant": ["crabpath/split.py", "crabpath/graph.py"]
  },
  {
    "query": "How does split choose between heuristic chunking and LLM chunking, and how are failed LLM splits handled?",
    "relevant": ["crabpath/split.py"]
  },
  {
    "query": "What is the `crabpath inject` difference between TEACHING and CORRECTION with respect to inhibitory edge creation?",
    "relevant": ["crabpath/inject.py", "crabpath/cli.py"]
  },
  {
    "query": "How are existing inhibitory edges updated inside `_apply_inhibitory_edges` when injection re-targets the same node again?",
    "relevant": ["crabpath/inject.py", "crabpath/graph.py"]
  },
  {
    "query": "How does CLI query fall back from index search to keyword overlap and which argument path triggers fallback?",
    "relevant": ["crabpath/cli.py"]
  },
  {
    "query": "How do cross-file edges get identified in health and why do they matter for autotune?",
    "relevant": ["crabpath/autotune.py", "crabpath/cli.py", "crabpath/graph.py"]
  },
  {
    "query": "When dormant edge ratio is high, what knob suggestions can `autotune` emit and why?",
    "relevant": ["crabpath/autotune.py"]
  },
  {
    "query": "How does `replay_queries` compare edge snapshots to capture which edges were reinforced and which were newly cross-file?",
    "relevant": ["crabpath/replay.py", "crabpath/graph.py"]
  },
  {
    "query": "What is the candidate selection process for connect suggestions and why are same-file pairs ignored?",
    "relevant": ["crabpath/connect.py"]
  },
  {
    "query": "How does merge decide candidate sibling pairs and preserve strongest edges when rewiring a merged node?",
    "relevant": ["crabpath/merge.py", "crabpath/graph.py"]
  },
  {
    "query": "What cold-start workflow sequence is documented in examples for init, doctor, query, correction, injection, query again, and health verification?",
    "relevant": ["examples/cold_start/README.md", "crabpath/cli.py"]
  },
  {
    "query": "How does correction flow align fired nodes by chat_id and apply delayed corrections without losing the target route?",
    "relevant": ["examples/correction_flow/README.md", "examples/correction_flow/correct.py", "examples/correction_flow/query_with_logging.py"]
  },
  {
    "query": "How does the openclaw adapter rebuild brains from sessions and learning DB while applying correction inhibition?",
    "relevant": ["examples/openclaw_adapter/README.md", "examples/openclaw_adapter/init_agent_brain.py"]
  },
  {
    "query": "How is `replay` related to `query` and `learn` in the lifecycle for production learning workflows?",
    "relevant": ["README.md", "examples/openclaw_adapter/README.md", "SKILL.md"]
  }
]


path: benchmarks/run_benchmark.py
content: from __future__ import annotations

import argparse
import json
import platform
import re
import time
from collections.abc import Callable
from copy import deepcopy
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

from crabpath import (
    HashEmbedder,
    TraversalConfig,
    VectorIndex,
    apply_outcome,
    apply_outcome_pg,
    split_workspace,
    traverse,
)
from crabpath.replay import default_keyword_seed_fn

WORD_RE = re.compile(r"[A-Za-z0-9']+")


def _tokenize(text: str) -> set[str]:
    return {match.group(0).lower() for match in WORD_RE.finditer(text or "")}


def _node_file(graph: Any, node_id: str) -> str | None:
    node = graph.get_node(node_id)
    if not node:
        return None
    value = node.metadata.get("file")
    return value if isinstance(value, str) else None


def _rank_to_files(graph: Any, node_ids: list[str], max_files: int | None = None) -> list[str]:
    files: list[str] = []
    seen: set[str] = set()
    for node_id in node_ids if max_files is None else node_ids[:max_files]:
        file = _node_file(graph, node_id)
        if file is None or file in seen:
            continue
        seen.add(file)
        files.append(file)
    return files


def _recall_at_k(predicted: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 0.0
    return len([f for f in predicted[:k] if f in relevant]) / len(relevant)


def _precision_at_k(predicted: list[str], relevant: set[str], k: int) -> float:
    if k <= 0:
        return 0.0
    return len([f for f in predicted[:k] if f in relevant]) / k


def _mrr(predicted: list[str], relevant: set[str]) -> float:
    for rank, file_name in enumerate(predicted, start=1):
        if file_name in relevant:
            return 1.0 / rank
    return 0.0


def _percentile(values: list[float], q: float) -> float:
    if not values:
        return 0.0
    if q <= 0.0:
        return float(min(values))
    if q >= 1.0:
        return float(max(values))

    ordered = sorted(values)
    index = int((len(ordered) - 1) * q)
    return float(ordered[index])


def _evaluate(ranked_nodes: list[str], graph: Any, relevant: list[str]) -> dict[str, float]:
    relevant_set = set(relevant)
    ranked_files = _rank_to_files(graph, ranked_nodes)
    return {
        "Recall@3": _recall_at_k(ranked_files, relevant_set, 3),
        "Recall@5": _recall_at_k(ranked_files, relevant_set, 5),
        "Precision@3": _precision_at_k(ranked_files, relevant_set, 3),
        "MRR": _mrr(ranked_files, relevant_set),
        "predicted_files": ranked_files[:5],
    }


def _run_query_set(
    name: str,
    queries: list[dict[str, object]],
    graph: Any,
    method: Callable[[str, Any], list[str]],
) -> tuple[dict[str, float], list[dict[str, object]], dict[str, float]]:
    totals = {
        "Recall@3": 0.0,
        "Recall@5": 0.0,
        "Precision@3": 0.0,
        "MRR": 0.0,
    }
    details: list[dict[str, object]] = []
    latencies_ms: list[float] = []

    for row in queries:
        query = str(row.get("query", ""))
        relevant = row.get("relevant")
        relevant_files = list(relevant) if isinstance(relevant, list) else []

        start = time.perf_counter()
        ranked_nodes = method(query, graph)
        latency_ms = (time.perf_counter() - start) * 1000
        scores = _evaluate(ranked_nodes, graph, relevant_files)
        latencies_ms.append(latency_ms)

        for key in totals:
            totals[key] += scores[key]

        details.append(
            {
                "query": query,
                "relevant": relevant_files,
                "predicted_files": scores.pop("predicted_files"),
                "scores": {k: scores[k] for k in totals},
                "latency_ms": latency_ms,
            }
        )

    query_count = len(queries) if queries else 1
    averaged = {key: value / query_count for key, value in totals.items()}
    latency_stats = {
        "p50_ms": _percentile(latencies_ms, 0.50),
        "p95_ms": _percentile(latencies_ms, 0.95),
    }
    return averaged, details, latency_stats


def _extract_fired_path(result: Any, graph: Any) -> list[str]:
    if not result:
        return []
    if result.steps:
        return [result.steps[0].from_node, *[step.to_node for step in result.steps]]

    fired = list(result.fired or [])
    if len(fired) <= 1:
        return fired

    valid: list[str] = []
    for node_id in fired:
        if graph.get_node(node_id) is not None:
            valid.append(node_id)
    return valid


def _run_learning_query_set(
    name: str,
    queries: list[dict[str, object]],
    graph: Any,
    config: TraversalConfig,
    seed_fn: Callable[[Any, str], list[tuple[str, float]]],
    learn_fn: Callable[[Any, list[str]], dict],
) -> tuple[dict[str, float], list[dict[str, object]], dict[str, float], dict[str, dict[str, float]]]:
    totals = {
        "Recall@3": 0.0,
        "Recall@5": 0.0,
        "Precision@3": 0.0,
        "MRR": 0.0,
    }
    details: list[dict[str, object]] = []
    latencies_ms: list[float] = []

    query_count = len(queries)
    checkpoints = [5, 10, 20, query_count] if query_count > 0 else []
    seen: set[int] = set()
    milestones: list[int] = []
    for point in checkpoints:
        if point <= 0:
            continue
        if point in seen:
            continue
        if point > query_count:
            continue
        seen.add(point)
        milestones.append(point)
    learning_curve: dict[str, dict[str, float]] = {}

    for idx, row in enumerate(queries, start=1):
        query = str(row.get("query", ""))
        relevant = row.get("relevant")
        relevant_files = list(relevant) if isinstance(relevant, list) else []
        start = time.perf_counter()
        seeded_nodes = seed_fn(graph, query)
        result = traverse(graph=graph, seeds=seeded_nodes, config=config, query_text=query)
        latency_ms = (time.perf_counter() - start) * 1000

        scores = _evaluate(result.fired, graph, relevant_files)
        latencies_ms.append(latency_ms)
        for key in totals:
            totals[key] += scores[key]
        details.append(
            {
                "query": query,
                "relevant": relevant_files,
                "predicted_files": scores.pop("predicted_files"),
                "scores": {k: scores[k] for k in totals},
                "latency_ms": latency_ms,
            }
        )

        fired_nodes = _extract_fired_path(result, graph)
        learn_fn(graph, fired_nodes)

        if idx in milestones:
            learning_curve[str(idx) if idx != query_count else "N"] = {
                key: value / idx for key, value in totals.items()
            }

    query_count = len(queries) if queries else 1
    averaged = {key: value / query_count for key, value in totals.items()}
    latency_stats = {
        "p50_ms": _percentile(latencies_ms, 0.50),
        "p95_ms": _percentile(latencies_ms, 0.95),
    }
    return averaged, details, latency_stats, learning_curve


def _keyword_overlap(graph: Any, query: str, top_k: int) -> list[str]:
    query_tokens = _tokenize(query)
    if not query_tokens:
        return []
    scores: list[tuple[str, float]] = []
    for node in graph.nodes():
        node_tokens = _tokenize(node.content)
        overlap = len(query_tokens & node_tokens) / len(query_tokens)
        scores.append((node.id, overlap))
    scores.sort(key=lambda item: (item[1], item[0]), reverse=True)
    return [node_id for node_id, _ in scores[:top_k]]


def _hash_overlap(index: VectorIndex, embedder: HashEmbedder, query: str, top_k: int) -> list[str]:
    query_vec = embedder.embed(query)
    return [node_id for node_id, _ in index.search(query_vec, top_k=top_k)]


def _weighted_hash_seeds(index: VectorIndex, embedder: HashEmbedder, query: str, top_k: int) -> list[tuple[str, float]]:
    seeds = _hash_overlap(index=index, embedder=embedder, query=query, top_k=top_k)
    if not seeds:
        return []
    return [(node_id, 1.0 - 0.05 * idx) for idx, node_id in enumerate(seeds)]


def _keyword_seeds(query: str, graph: Any, top_k: int) -> list[tuple[str, float]]:
    query_tokens = _tokenize(query)
    if not query_tokens:
        return []
    scores = [
        (node.id, len(query_tokens & _tokenize(node.content)) / len(query_tokens))
        for node in graph.nodes()
    ]
    scores.sort(key=lambda item: (item[1], item[0]), reverse=True)
    return scores[:top_k]


def _traverse(graph: Any, index: VectorIndex, embedder: HashEmbedder, query: str, seed_k: int, config: TraversalConfig) -> list[str]:
    seed_weights = _weighted_hash_seeds(index=index, embedder=embedder, query=query, top_k=seed_k)
    if not seed_weights:
        return []
    result = traverse(graph=graph, seeds=seed_weights, config=config, query_text=query)
    return result.fired


def _build_index(graph: Any, texts: dict[str, str], embedder: HashEmbedder) -> VectorIndex:
    index = VectorIndex()
    for node_id, content in texts.items():
        index.upsert(node_id, embedder.embed(content))
    return index


def _print_table(
    scores_by_method: dict[str, dict[str, float]],
    latency_by_method: dict[str, dict[str, float]],
) -> None:
    print("\nMethod                  Recall@3 Recall@5 Precision@3 MRR    p50_ms p95_ms")
    print("--------------------------------------------------------------")
    for name, scores in scores_by_method.items():
        latency = latency_by_method.get(name, {})
        print(
            f"{name:<22} "
            f"{scores['Recall@3']:.3f}    "
            f"{scores['Recall@5']:.3f}     "
            f"{scores['Precision@3']:.3f}       "
            f"{scores['MRR']:.3f}  "
            f"{latency.get('p50_ms', 0.0):7.3f} "
            f"{latency.get('p95_ms', 0.0):7.3f}"
    )


def main() -> None:
    parser = argparse.ArgumentParser(description="Run a reproducible CrabPath retrieval benchmark.")
    parser.add_argument("--workspace", default=".", help="Workspace directory to split")
    parser.add_argument("--queries-path", default="benchmarks/queries.json", help="Path to query definitions")
    parser.add_argument("--seed-top-k", type=int, default=20, help="Top-k for keyword overlap")
    parser.add_argument("--hash-top-k", type=int, default=20, help="Top-k for hash similarity search")
    parser.add_argument("--traverse-seed-k", type=int, default=8, help="Seed top-k for traverse")
    parser.add_argument("--traverse-max-hops", type=int, default=8, help="Traverse max_hops")
    parser.add_argument("--traverse-beam-width", type=int, default=4, help="Traverse beam_width")
    parser.add_argument("--replay-max-hops", type=int, default=2, help="Replay traverse max_hops")
    args = parser.parse_args()

    workspace_path = Path(args.workspace).resolve()
    queries_path = Path(args.queries_path)
    if not queries_path.exists():
        raise SystemExit(f"queries file missing: {queries_path}")

    queries = json.loads(queries_path.read_text(encoding="utf-8"))
    if not isinstance(queries, list):
        raise SystemExit("queries.json must contain a list")

    print(f"Splitting workspace: {workspace_path}")
    split_start = time.perf_counter()
    graph, texts = split_workspace(workspace_path)
    split_ms = (time.perf_counter() - split_start) * 1000

    embed_start = time.perf_counter()
    embedder = HashEmbedder()
    index = _build_index(graph, texts, embedder)
    embed_ms = (time.perf_counter() - embed_start) * 1000
    print(f"Graph nodes: {graph.node_count()} | Chunks: {len(texts)}")
    print(f"Init timings: split={split_ms:.3f}ms embed={embed_ms:.3f}ms")
    print("Benchmark timings are machine-dependent and vary by hardware, OS, and runtime conditions.")

    base_graph = graph
    traverse_config = TraversalConfig(max_hops=args.traverse_max_hops, beam_width=args.traverse_beam_width)
    replay_config = TraversalConfig(max_hops=args.replay_max_hops, beam_width=args.traverse_beam_width)
    latency_ms: dict[str, dict[str, float]] = {}
    learning_curves: dict[str, dict[str, dict[str, float]]] = {}

    scores: dict[str, dict[str, float]] = {}
    per_query: dict[str, list[dict[str, object]]] = {}
    weighted_hash_seeds = lambda query_graph, query_text: _weighted_hash_seeds(
        index=index,
        embedder=embedder,
        query=query_text,
        top_k=args.traverse_seed_k,
    )
    keyword_seed = lambda query_graph, query_text: default_keyword_seed_fn(graph=query_graph, query_text=query_text)

    scores["keyword_overlap"], per_query["keyword_overlap"], latency_ms["keyword_overlap"] = _run_query_set(
        name="keyword_overlap",
        queries=queries,
        graph=deepcopy(base_graph),
        method=lambda query, query_graph: _keyword_overlap(graph=query_graph, query=query, top_k=args.seed_top_k),
    )

    scores["hash_embed_similarity"], per_query["hash_embed_similarity"], latency_ms["hash_embed_similarity"] = _run_query_set(
        name="hash_embed_similarity",
        queries=queries,
        graph=deepcopy(base_graph),
        method=lambda query, query_graph: _hash_overlap(index=index, embedder=embedder, query=query, top_k=args.hash_top_k),
    )

    scores["static_traverse"], per_query["static_traverse"], latency_ms["static_traverse"] = _run_query_set(
        name="static_traverse",
        queries=queries,
        graph=deepcopy(base_graph),
        method=lambda query, query_graph: _traverse(
            graph=query_graph,
            index=index,
            embedder=embedder,
            query=query,
            seed_k=args.traverse_seed_k,
            config=traverse_config,
        ),
    )

    scores["crabpath_traverse"], per_query["crabpath_traverse"], latency_ms["crabpath_traverse"], learning_curves["crabpath_traverse"] = _run_learning_query_set(
        name="crabpath_traverse",
        queries=queries,
        graph=deepcopy(base_graph),
        config=traverse_config,
        seed_fn=weighted_hash_seeds,
        learn_fn=lambda learning_graph, fired_nodes: apply_outcome(graph=learning_graph, fired_nodes=fired_nodes, outcome=1.0),
    )

    no_inhibition_config = TraversalConfig(
        max_hops=args.traverse_max_hops,
        beam_width=args.traverse_beam_width,
        inhibitory_threshold=-999,
    )
    scores["crabpath_no_inhibition"], per_query["crabpath_no_inhibition"], latency_ms["crabpath_no_inhibition"], learning_curves["crabpath_no_inhibition"] = _run_learning_query_set(
        name="crabpath_no_inhibition",
        queries=queries,
        graph=deepcopy(base_graph),
        config=no_inhibition_config,
        seed_fn=weighted_hash_seeds,
        learn_fn=lambda learning_graph, fired_nodes: apply_outcome(graph=learning_graph, fired_nodes=fired_nodes, outcome=1.0),
    )

    scores["crabpath_pg"], per_query["crabpath_pg"], latency_ms["crabpath_pg"], learning_curves["crabpath_pg"] = _run_learning_query_set(
        name="crabpath_pg",
        queries=queries,
        graph=deepcopy(base_graph),
        config=traverse_config,
        seed_fn=weighted_hash_seeds,
        learn_fn=lambda learning_graph, fired_nodes: apply_outcome_pg(graph=learning_graph, fired_nodes=fired_nodes, outcome=1.0),
    )

    scores["crabpath_with_replay"], per_query["crabpath_with_replay"], latency_ms["crabpath_with_replay"], learning_curves["crabpath_with_replay"] = _run_learning_query_set(
        name="crabpath_with_replay",
        queries=queries,
        graph=deepcopy(base_graph),
        config=replay_config,
        seed_fn=keyword_seed,
        learn_fn=lambda learning_graph, fired_nodes: apply_outcome(graph=learning_graph, fired_nodes=fired_nodes, outcome=1.0),
    )

    scores["crabpath_pg_with_replay"], per_query["crabpath_pg_with_replay"], latency_ms["crabpath_pg_with_replay"], learning_curves["crabpath_pg_with_replay"] = _run_learning_query_set(
        name="crabpath_pg_with_replay",
        queries=queries,
        graph=deepcopy(base_graph),
        config=replay_config,
        seed_fn=keyword_seed,
        learn_fn=lambda learning_graph, fired_nodes: apply_outcome_pg(graph=learning_graph, fired_nodes=fired_nodes, outcome=1.0),
    )

    _print_table(scores, latency_ms)

    results_path = Path("benchmarks/results.json")
    output = {
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "workspace": str(workspace_path),
        "query_source": str(queries_path),
        "query_count": len(queries),
        "init_timing_ms": {"split": split_ms, "embed": embed_ms},
        "graph_nodes": graph.node_count(),
        "chunks": len(texts),
        "embedder": embedder.name,
        "embedder_dim": embedder.dim,
        "traversal": {
            "max_hops": args.traverse_max_hops,
            "beam_width": args.traverse_beam_width,
            "seed_k": args.traverse_seed_k,
        },
        "environment": {
            "python": platform.python_version(),
            "platform": platform.platform(),
            "machine_notes": "Benchmarks are timing-sensitive and may vary across machines.",
        },
        "methods": {},
    }

    for method_name, method_scores in scores.items():
        method_latency = latency_ms.get(method_name, {"p50_ms": 0.0, "p95_ms": 0.0})
        output["methods"][method_name] = {
            "scores": method_scores,
            "latency_ms": method_latency,
            "per_query": per_query[method_name],
            "learning_curve": learning_curves.get(method_name, {}),
        }

    results_path.write_text(json.dumps(output, indent=2), encoding="utf-8")
    print(f"Saved: {results_path}")


if __name__ == "__main__":
    main()


path: crabpath/__init__.py
content: """CrabPath public API."""

from .graph import Edge, Graph, Node, remove_from_state
from .index import VectorIndex
from ._batch import batch_or_single, batch_or_single_embed
from .autotune import GraphHealth, autotune, measure_health
from .decay import DecayConfig, apply_decay
from .maintain import MaintenanceReport, run_maintenance, prune_edges, prune_orphan_nodes
from .learn import LearningConfig, apply_outcome, apply_outcome_pg, maybe_create_node
from .inject import inject_batch, inject_correction, inject_node
from .hasher import HashEmbedder, default_embed, default_embed_batch
from .score import score_retrieval
from .connect import apply_connections, suggest_connections
from .merge import apply_merge, suggest_merges
from .replay import replay_queries
from .split import generate_summaries, split_workspace
from .sync import DEFAULT_AUTHORITY_MAP, SyncReport, sync_workspace
from .store import ManagedState, load_state, save_state
from .traverse import TraversalResult, TraversalConfig, traverse

__all__ = [
    "Node",
    "Edge",
    "Graph",
    "VectorIndex",
    "GraphHealth",
    "TraversalConfig",
    "TraversalResult",
    "HashEmbedder",
    "default_embed",
    "default_embed_batch",
    "DecayConfig",
    "LearningConfig",
    "MaintenanceReport",
    "measure_health",
    "autotune",
    "prune_edges",
    "prune_orphan_nodes",
    "traverse",
    "ManagedState",
    "split_workspace",
    "generate_summaries",
    "sync_workspace",
    "SyncReport",
    "DEFAULT_AUTHORITY_MAP",
    "apply_outcome",
    "apply_outcome_pg",
    "remove_from_state",
    "maybe_create_node",
    "apply_decay",
    "batch_or_single",
    "batch_or_single_embed",
    "save_state",
    "load_state",
    "score_retrieval",
    "suggest_merges",
    "apply_merge",
    "run_maintenance",
    "suggest_connections",
    "apply_connections",
    "inject_node",
    "inject_correction",
    "inject_batch",
    "replay_queries",
]

__version__ = "10.4.0"


path: crabpath/__main__.py
content: from .cli import main

if __name__ == "__main__":
    raise SystemExit(main())


path: crabpath/_batch.py
content: """Helpers for optional LLM/embedding batch execution."""

from __future__ import annotations

from collections.abc import Callable


def batch_or_single(
    requests: list[dict],
    llm_fn: Callable[[str, str], str] | None = None,
    llm_batch_fn: Callable[[list[dict]], list[dict]] | None = None,
    max_workers: int = 8,
) -> list[dict]:
    """Execute LLM requests using a batch callback when available, otherwise parallel single calls."""
    if llm_batch_fn is not None:
        return llm_batch_fn(requests)
    if llm_fn is None:
        raise ValueError("either llm_fn or llm_batch_fn must be provided")
    if not requests:
        return []

    from concurrent.futures import ThreadPoolExecutor

    def _call(req: dict) -> dict:
        """Call LLM for one request."""
        try:
            response = llm_fn(req["system"], req["user"])
            return {"id": req["id"], "response": response}
        except Exception as e:  # noqa: BLE001
            return {"id": req["id"], "response": "", "error": str(e)}

    with ThreadPoolExecutor(max_workers=max_workers) as pool:
        return list(pool.map(_call, requests))


def batch_or_single_embed(
    texts: list[tuple[str, str]],
    embed_fn: Callable[[str], list[float]] | None = None,
    embed_batch_fn: Callable[[list[tuple[str, str]]], dict[str, list[float]]] | None = None,
    max_workers: int = 8,
) -> dict[str, list[float]]:
    """Execute embedding requests with either batched or single-text callbacks.

    Contract:
    - ``texts`` is always ``list[tuple[node_id, text]]`` for both single and batch paths.
    - ``embed_fn`` receives only ``text`` and returns ``list[float]``.
    - ``embed_batch_fn`` receives ``list[tuple[node_id, text]]`` and returns
      ``dict[node_id, list[float]]``.
    """
    if embed_batch_fn is not None:
        return embed_batch_fn(texts)
    if embed_fn is None:
        raise ValueError("either embed_fn or embed_batch_fn must be provided")

    from concurrent.futures import ThreadPoolExecutor

    def _call(item: tuple[str, str]) -> tuple[str, list[float]]:
        """Call embedder for one text."""
        node_id, text = item
        return node_id, embed_fn(text)

    results: dict[str, list[float]] = {}
    with ThreadPoolExecutor(max_workers=max_workers) as pool:
        for node_id, vector in pool.map(_call, texts):
            results[node_id] = vector
    return results


path: crabpath/_util.py
content: """Internal shared utilities for parsing lightweight LLM and string helpers."""

from __future__ import annotations

import json
import re


_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.S)
_WORD_RE = re.compile(r"[A-Za-z0-9']+")


def _extract_json(raw: str) -> dict | list | str | None:
    """Extract JSON from raw model output."""
    text = (raw or "").strip()
    if not text:
        return None

    if text.startswith("```") and text.endswith("```"):
        text = "\n".join(text.splitlines()[1:-1]).strip()

    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass

    match = _JSON_OBJECT_RE.search(text)
    if not match:
        return None
    try:
        return json.loads(match.group(0))
    except json.JSONDecodeError:
        return None


def _first_line(text: str) -> str:
    """Return the first non-empty line."""
    return (text.splitlines() or [""])[0]


def _tokenize(text: str) -> set[str]:
    """Extract normalized tokens from text."""
    return {match.group(0).lower() for match in _WORD_RE.finditer(text or "")}


path: crabpath/autotune.py
content: """Graph health measurement and simple self-tuning suggestions."""

from __future__ import annotations

from dataclasses import dataclass

from .graph import Graph
from .decay import DecayConfig
from .learn import LearningConfig


@dataclass
class GraphHealth:
    """Readout of graph state for auto-tuning decisions."""

    dormant_pct: float
    habitual_pct: float
    reflex_pct: float
    cross_file_edge_pct: float
    orphan_nodes: int


def measure_health(graph: Graph) -> GraphHealth:
    """Measure aggregate weights and orphan ratio for tuning."""
    total_edges = graph.edge_count()
    if total_edges == 0:
        return GraphHealth(
            dormant_pct=0.0,
            habitual_pct=0.0,
            reflex_pct=0.0,
            cross_file_edge_pct=0.0,
            orphan_nodes=graph.node_count(),
        )

    reflex = habitual = dormant = cross_file = 0
    for source_edges in graph._edges.values():
        source_node = graph.get_node(source_edges and next(iter(source_edges.values())).source)
        source_file = source_node.metadata.get("file") if source_node else None
        for edge in source_edges.values():
            if edge.weight >= 0.6:
                reflex += 1
            elif 0.2 <= edge.weight < 0.6:
                habitual += 1
            else:
                dormant += 1

            target_node = graph.get_node(edge.target)
            target_file = target_node.metadata.get("file") if target_node else None
            if source_file is not None and target_file is not None and source_file != target_file:
                cross_file += 1

    dormant_pct = dormant / total_edges
    habitual_pct = habitual / total_edges
    reflex_pct = reflex / total_edges
    cross_file_edge_pct = cross_file / total_edges

    orphan = 0
    for node in graph.nodes():
        has_out = len(graph.outgoing(node.id)) > 0
        has_in = len(graph.incoming(node.id)) > 0
        if not has_out and not has_in:
            orphan += 1

    return GraphHealth(
        dormant_pct=dormant_pct,
        habitual_pct=habitual_pct,
        reflex_pct=reflex_pct,
        cross_file_edge_pct=cross_file_edge_pct,
        orphan_nodes=orphan,
    )


def autotune(graph: Graph, health: GraphHealth) -> list[dict]:
    """Return suggested adjustments for graph-level health tuning.

    ``graph`` is required because orphan and cross-file statistics come directly
    from structure traversal. ``health`` is a precomputed health snapshot used to
    decide which knobs to adjust.

    Knobs:
    - ``half_life``
    - ``hebbian_increment``
    - ``promotion_threshold``
    """
    deltas: list[dict] = []

    if health.dormant_pct > 0.65:
        deltas.append(
            {
                "knob": "half_life",
                "suggested_adjustment": "decrease",
                "value": 10,
                "reason": "Many dormant edges; slow decay to preserve low-signal paths longer.",
            }
        )
        deltas.append(
            {
                "knob": "promotion_threshold",
                "suggested_adjustment": "decrease",
                "value": 0.05,
                "reason": "Promote more edges from dormant into habitual processing.",
            }
        )

    if health.habitual_pct < 0.2:
        deltas.append(
            {
                "knob": "hebbian_increment",
                "suggested_adjustment": "increase",
                "value": 0.02,
                "reason": "Few habitual pathways; strengthen co-firing structure faster.",
            }
        )

    if health.reflex_pct > 0.8:
        deltas.append(
            {
                "knob": "half_life",
                "suggested_adjustment": "increase",
                "value": 15,
                "reason": "Predominantly reflex traffic; relax decay to avoid over-pruning.",
            }
        )

    if health.orphan_nodes > graph.node_count() * 0.2:
        deltas.append(
            {
                "knob": "promotion_threshold",
                "suggested_adjustment": "decrease",
                "value": 0.02,
                "reason": "Many orphan nodes; lower threshold to reconnect exploration.",
            }
        )

    if health.cross_file_edge_pct < 0.1 and graph.node_count() > 1:
        deltas.append(
            {
                "knob": "promotion_threshold",
                "suggested_adjustment": "decrease",
                "value": 0.05,
                "reason": "Promote more dormant edges and increase cross-file exploration.",
            }
        )

    return deltas


def _clamp_half_life(value: float) -> int:
    """Clamp half-life to a safe range for decay."""
    return max(10, min(500, int(round(value))))


def apply_autotune(
    graph: Graph,
    suggestions: list[dict],
    decay_config: DecayConfig | None = None,
    learning_config: LearningConfig | None = None,
) -> tuple[DecayConfig, LearningConfig]:
    """Apply suggestions and return the resulting tuning configurations."""
    del graph
    decay_cfg = DecayConfig() if decay_config is None else DecayConfig(**decay_config.__dict__)
    learning_cfg = LearningConfig() if learning_config is None else LearningConfig(**learning_config.__dict__)

    for suggestion in suggestions:
        knob = suggestion.get("knob")
        adjustment = suggestion.get("suggested_adjustment")
        value = suggestion.get("value")
        if knob is None or adjustment not in {"increase", "decrease"} or not isinstance(value, (int, float)):
            continue

        delta = value if adjustment == "increase" else -value
        if knob == "half_life":
            decay_cfg = DecayConfig(half_life=_clamp_half_life(decay_cfg.half_life + delta))
        elif knob == "hebbian_increment":
            learning_cfg = LearningConfig(**learning_cfg.__dict__ | {"hebbian_increment": learning_cfg.hebbian_increment + delta})
        elif knob == "promotion_threshold":
            learning_cfg = LearningConfig(
                **learning_cfg.__dict__
                | {"promotion_threshold": max(1, round(learning_cfg.promotion_threshold + delta))}
            )

    return decay_cfg, learning_cfg


path: crabpath/cli.py
content: """Pure graph-operations CLI for CrabPath."""

from __future__ import annotations

import argparse
import json
import os
import sys
import tempfile
import shutil
from collections.abc import Callable, Iterable
from dataclasses import asdict
from pathlib import Path
from types import SimpleNamespace

from .connect import apply_connections, suggest_connections
from .autotune import measure_health
from .graph import Edge, Graph, Node
from .index import VectorIndex
from .inject import inject_correction, inject_node
from .compact import compact_daily_notes
from .journal import (
    log_health,
    log_learn,
    log_query,
    log_replay,
    journal_stats,
    read_journal,
)
from .learn import apply_outcome
from .merge import apply_merge, suggest_merges
from .replay import (
    extract_interactions,
    extract_query_records,
    extract_query_records_from_dir,
    extract_queries,
    extract_queries_from_dir,
    replay_queries,
)
from .split import split_workspace
from .hasher import HashEmbedder
from .traverse import TraversalConfig, TraversalResult, traverse
from .sync import DEFAULT_AUTHORITY_MAP, sync_workspace
from ._util import _tokenize
from .maintain import run_maintenance
from .store import load_state, save_state
from . import __version__


def _build_parser() -> argparse.ArgumentParser:
    """ build parser."""
    parser = argparse.ArgumentParser(prog="crabpath")
    sub = parser.add_subparsers(dest="command", required=True)

    i = sub.add_parser("init")
    i.add_argument("--workspace", required=True)
    i.add_argument("--output", required=True)
    i.add_argument("--sessions")
    i.add_argument("--embedder", choices=["hash", "openai"], default=None)
    i.add_argument("--llm", choices=["none", "openai"], default="none")
    i.add_argument("--json", action="store_true")

    q = sub.add_parser("query")
    q.add_argument("text")
    q.add_argument("--state")
    q.add_argument("--graph")
    q.add_argument("--index")
    q.add_argument("--top", type=int, default=10)
    q.add_argument("--query-vector-stdin", action="store_true")
    q.add_argument("--embedder", choices=["hash", "openai"], default=None)
    q.add_argument("--max-context-chars", type=int, default=None)
    q.add_argument("--json", action="store_true")

    l = sub.add_parser("learn")
    l.add_argument("--state")
    l.add_argument("--graph")
    l.add_argument("--outcome", type=float, required=True)
    l.add_argument("--fired-ids", required=True)
    l.add_argument("--json", action="store_true")

    m = sub.add_parser("merge")
    m.add_argument("--state")
    m.add_argument("--graph")
    m.add_argument("--llm", choices=["none", "openai"], default="none")
    m.add_argument("--json", action="store_true")

    a = sub.add_parser("anchor")
    a.add_argument("--state", required=True)
    a.add_argument("--node-id")
    a.add_argument("--authority", choices=["constitutional", "canonical"])
    a.add_argument("--remove", action="store_true")
    a.add_argument("--list", action="store_true")
    a.add_argument("--json", action="store_true")

    c = sub.add_parser("connect")
    c.add_argument("--state")
    c.add_argument("--graph")
    c.add_argument("--llm", choices=["none", "openai"], default="none")
    c.add_argument("--json", action="store_true")

    p = sub.add_parser("maintain")
    p.add_argument("--state", required=True)
    p.add_argument("--tasks", default="health,decay,merge,prune")
    p.add_argument("--dry-run", action="store_true")
    p.add_argument("--max-merges", type=int, default=5)
    p.add_argument("--prune-below", type=float, default=0.01)
    p.add_argument("--llm", choices=["none", "openai"], default="none")
    p.add_argument("--json", action="store_true")

    z = sub.add_parser("compact")
    z.add_argument("--state", required=True)
    z.add_argument("--memory-dir", required=True)
    z.add_argument("--max-age-days", type=int, default=7)
    z.add_argument("--target-lines", type=int, default=15)
    z.add_argument("--llm", choices=["none", "openai"], default="none")
    z.add_argument("--dry-run", action="store_true")
    z.add_argument("--json", action="store_true")

    d = sub.add_parser("daemon")
    d.add_argument("--state", required=True)
    d.add_argument("--embed-model", default="text-embedding-3-small")
    d.add_argument("--auto-save-interval", type=int, default=10)

    x = sub.add_parser("inject")
    x.add_argument("--state", required=True)
    x.add_argument("--id", required=True)
    x.add_argument("--content", required=True)
    x.add_argument(
        "--type",
        choices=["CORRECTION", "TEACHING", "DIRECTIVE"],
        default="TEACHING",
    )
    x.add_argument("--summary")
    x.add_argument("--connect-top-k", type=int, default=3)
    x.add_argument("--connect-min-sim", type=float, default=None)
    x.add_argument("--embedder", choices=["hash", "openai"], default=None)
    x.add_argument("--vector-stdin", action="store_true")
    x.add_argument("--json", action="store_true")

    r = sub.add_parser("replay")
    r.add_argument("--state")
    r.add_argument("--graph")
    r.add_argument("--sessions", nargs="+", required=True)
    r.add_argument("--json", action="store_true")

    h = sub.add_parser("health")
    h.add_argument("--state")
    h.add_argument("--graph")
    h.add_argument("--json", action="store_true")

    j = sub.add_parser("journal")
    j.add_argument("--state")
    j.add_argument("--last", type=int, default=10)
    j.add_argument("--stats", action="store_true")
    j.add_argument("--json", action="store_true")

    doctor = sub.add_parser("doctor")
    doctor.add_argument("--state", required=True)

    info = sub.add_parser("info")
    info_group = info.add_mutually_exclusive_group(required=True)
    info_group.add_argument("--state")
    info_group.add_argument("--graph")
    info.add_argument("--json", action="store_true")

    sync = sub.add_parser("sync")
    sync.add_argument("--state", required=True)
    sync.add_argument("--workspace", required=True)
    sync.add_argument("--embedder", choices=["openai", "hash"], default=None)
    sync.add_argument(
        "--authority-map",
        help="JSON object mapping file name -> authority level",
    )
    sync.add_argument("--dry-run", action="store_true")
    sync.add_argument("--json", action="store_true")
    return parser


def _load_payload(path: str) -> dict:
    """ load payload."""
    payload_path = Path(os.path.expanduser(path))
    if payload_path.is_dir():
        payload_path = payload_path / "graph.json"
    if not payload_path.exists():
        raise SystemExit(f"missing graph file: {path}")
    return json.loads(payload_path.read_text(encoding="utf-8"))


def _load_graph(path: str) -> Graph:
    """ load graph."""
    payload = _load_payload(path)
    payload = payload["graph"] if "graph" in payload else payload
    graph = Graph()
    for node_data in payload.get("nodes", []):
        graph.add_node(
            Node(node_data["id"], node_data["content"], node_data.get("summary", ""), node_data.get("metadata", {}))
        )
    for edge_data in payload.get("edges", []):
        graph.add_edge(
            Edge(
                edge_data["source"],
                edge_data["target"],
                edge_data.get("weight", 0.5),
                edge_data.get("kind", "sibling"),
                edge_data.get("metadata", {}),
            )
        )
    return graph


def _resolve_graph_index(args: argparse.Namespace) -> tuple[Graph, VectorIndex | None, dict[str, object]]:
    """ resolve graph index."""
    if args.state is not None:
        graph, index, meta = load_state(args.state)
        return graph, index, meta
    if args.graph is None:
        raise SystemExit("--state or --graph is required")

    graph = _load_graph(args.graph)
    index_arg = getattr(args, "index", None)
    index = _load_index(index_arg) if index_arg is not None else None
    return graph, index, {}


def _graph_payload(graph: Graph) -> dict:
    """ graph payload."""
    return {
        "nodes": [
            {
                "id": n.id,
                "content": n.content,
                "summary": n.summary,
                "metadata": n.metadata,
            }
            for n in graph.nodes()
        ],
        "edges": [
            {"source": e.source, "target": e.target, "weight": e.weight, "kind": e.kind, "metadata": e.metadata}
            for source in graph._edges.values()
            for e in source.values()
        ],
    }


def _write_graph(
    path: str | Path,
    graph: Graph,
    *,
    include_meta: bool = False,
    meta: dict[str, object] | None = None,
) -> None:
    """Write graph payload to a JSON file."""
    destination = Path(path).expanduser()
    if destination.is_dir():
        destination = destination / "graph.json"
    destination.parent.mkdir(parents=True, exist_ok=True)
    payload: dict[str, object] = _graph_payload(graph)
    if include_meta:
        payload = {"graph": payload, "meta": meta or {}}
    destination.write_text(json.dumps(payload, indent=2), encoding="utf-8")


def _load_query_vector_from_stdin() -> list[float]:
    """ load query vector from stdin."""
    data = sys.stdin.read().strip()
    if not data:
        raise SystemExit("query vector JSON required on stdin")
    payload = json.loads(data)
    if not isinstance(payload, list):
        raise SystemExit("query vector stdin payload must be a JSON array")
    return [float(v) for v in payload]


def _load_session_queries(session_paths: str | Iterable[str]) -> list[str]:
    """ load session queries."""
    if isinstance(session_paths, str):
        session_paths = [session_paths]
    queries: list[str] = []
    for session_path in session_paths:
        path = Path(session_path).expanduser()
        if path.is_dir():
            queries.extend(extract_queries_from_dir(path))
        elif path.is_file():
            queries.extend(extract_queries(path))
        else:
            raise SystemExit(f"invalid sessions path: {path}")
    return queries


def _resolve_journal_path(args: argparse.Namespace) -> str | None:
    """ resolve journal path."""
    if args.state is not None:
        path = Path(args.state).expanduser()
        return str(path.parent / "journal.jsonl")
    if getattr(args, "graph", None) is not None:
        graph_path = Path(args.graph).expanduser()
        if graph_path.is_dir():
            return str(graph_path / "journal.jsonl")
        return str(graph_path.parent / "journal.jsonl")
    return None


def _load_session_query_records(session_paths: str | Iterable[str], since_ts: float | None = None) -> list[tuple[str, float | None]]:
    """ load session query records."""
    if isinstance(session_paths, str):
        session_paths = [session_paths]
    records: list[tuple[str, float | None]] = []
    for session_path in session_paths:
        path = Path(session_path).expanduser()
        if path.is_dir():
            records.extend(extract_query_records_from_dir(path, since_ts=since_ts))
        elif path.is_file():
            records.extend(extract_query_records(path, since_ts=since_ts))
        else:
            raise SystemExit(f"invalid sessions path: {path}")
    return records


def _load_session_interactions(session_paths: str | Iterable[str], since_ts: float | None = None) -> list[dict[str, object]]:
    """ load session interactions."""
    if isinstance(session_paths, str):
        session_paths = [session_paths]
    interactions: list[dict[str, object]] = []
    for session_path in session_paths:
        path = Path(session_path).expanduser()
        if path.is_dir():
            for session_file in sorted(path.glob("*.jsonl")):
                interactions.extend(extract_interactions(session_file, since_ts=since_ts))
        elif path.is_file():
            interactions.extend(extract_interactions(path, since_ts=since_ts))
        else:
            raise SystemExit(f"invalid sessions path: {path}")
    return interactions


def _state_meta(meta: dict[str, object] | None, fallback_name: str | None = None, fallback_dim: int | None = None) -> dict[str, object]:
    """ state meta."""
    base = dict(meta or {})
    embedder_name, embedder_dim = _state_embedder_meta(base)
    if fallback_name is not None:
        base["embedder_name"] = embedder_name or fallback_name
    if fallback_dim is not None:
        base["embedder_dim"] = embedder_dim if embedder_dim is not None else fallback_dim
    return base


def _keyword_seeds(graph: Graph, text: str, top_k: int) -> list[tuple[str, float]]:
    """ keyword seeds."""
    query_tokens = _tokenize(text)
    if not query_tokens:
        return []
    scores = [
        (node.id, len(query_tokens & _tokenize(node.content)) / len(query_tokens))
        for node in graph.nodes()
    ]
    scores.sort(key=lambda item: (item[1], item[0]), reverse=True)
    return scores[:top_k]


def _load_index(path: str) -> VectorIndex:
    """ load index."""
    payload = json.loads(Path(path).expanduser().read_text(encoding="utf-8"))
    if not isinstance(payload, dict):
        raise SystemExit("index payload must be a JSON object")
    index = VectorIndex()
    for node_id, vector in payload.items():
        if not isinstance(vector, list):
            raise SystemExit("index payload vectors must be arrays")
        index.upsert(str(node_id), [float(v) for v in vector])
    return index


def _state_embedder_meta(meta: dict[str, object]) -> tuple[str | None, int | None]:
    """ state embedder meta."""
    embedder_name = meta.get("embedder_name")
    if not isinstance(embedder_name, str):
        embedder_name = meta.get("embedder")
        if not isinstance(embedder_name, str):
            embedder_name = None

    embedder_dim = meta.get("embedder_dim")
    if not isinstance(embedder_dim, int):
        embedder_dim = None

    return embedder_name, embedder_dim


def _resolve_embedder(
    args: argparse.Namespace, meta: dict[str, object]
) -> tuple[callable[[str], list[float]], callable[[list[tuple[str, str]]], dict[str, list[float]]], str, int]:
    """ resolve embedder."""
    openai_name = "openai-text-embedding-3-small"
    embedder_name, _ = _state_embedder_meta(meta)
    use_openai = args.embedder == "openai" or (args.embedder is None and embedder_name == openai_name)
    if use_openai:
        from .openai_embeddings import OpenAIEmbedder

        embedder = OpenAIEmbedder()
    else:
        embedder = HashEmbedder()

    return embedder.embed, embedder.embed_batch, embedder.name, embedder.dim


def _resolve_llm(args: argparse.Namespace) -> tuple[Callable[[str, str], str] | None, Callable[[list[dict]], list[dict]] | None]:
    """Resolve optional LLM callbacks."""
    if getattr(args, "llm", None) == "openai":
        from .openai_llm import openai_llm_batch_fn, openai_llm_fn

        return openai_llm_fn, openai_llm_batch_fn
    return None, None


def _load_json(path: str) -> dict:
    """ load json."""
    try:
        payload = json.loads(Path(path).expanduser().read_text(encoding="utf-8"))
    except json.JSONDecodeError as exc:
        raise SystemExit(f"invalid JSON in state file: {path}") from exc
    except OSError as exc:
        raise SystemExit(f"missing state file: {path}") from exc
    if not isinstance(payload, dict):
        raise SystemExit(f"state file payload must be an object: {path}")
    return payload


def _state_payload(meta_path: str) -> tuple[dict, dict[str, object], dict[str, list[float]], Graph]:
    """ state payload."""
    payload = _load_json(meta_path)
    graph_payload = payload.get("graph", payload)
    if not isinstance(graph_payload, dict):
        raise SystemExit("state file graph payload must be an object")

    index_payload = payload.get("index", {})
    if not isinstance(index_payload, dict):
        raise SystemExit("state index payload must be an object")

    graph = _load_graph(meta_path)
    return payload, graph_payload, index_payload, graph


def _check_result(ok: bool, label: str, details: str = "") -> bool:
    """ check result."""
    print(f"{label}: {'PASS' if ok else 'FAIL'}" + (f" ({details})" if details else ""))
    return ok


def _journal_entry_count(journal_path: str | None) -> int | None:
    """ journal entry count."""
    if journal_path is None:
        return None
    path = Path(journal_path)
    if not path.exists():
        return None
    return len(read_journal(journal_path=str(path)))


def _ensure_hash_embedder_compat(meta: dict[str, object]) -> None:
    """ ensure hash embedder compat."""
    embedder_name, embedder_dim = _state_embedder_meta(meta)
    if embedder_dim is None:
        return

    hash_dim = HashEmbedder().dim
    if embedder_dim != hash_dim:
        raise SystemExit(
            f"Index was built with {embedder_name} (dim={embedder_dim}). "
            "CLI hash embedder uses dim=1024. Dimension mismatch. "
            "Use --query-vector-stdin with matching embedder."
        )


def _result_payload(result: TraversalResult) -> dict:
    """ result payload."""
    return {
        "fired": result.fired,
        "steps": [step.__dict__ for step in result.steps],
        "context": result.context,
        "tier_thresholds": result.tier_summary,
    }


def _query_text_output(result: TraversalResult, graph: Graph, max_context_chars: int | None = None) -> str:
    """format query output with node IDs."""
    if not result.fired:
        return "(No matches.)"

    rendered: list[str] = []
    used_chars = 0
    for idx, node_id in enumerate(dict.fromkeys(result.fired)):
        node = graph.get_node(node_id)
        if node is None:
            continue
        block = f"{node_id}\n{'~' * len(node_id)}\n{node.content}"
        if max_context_chars is None:
            rendered.append(block)
            continue

        if max_context_chars <= 0:
            break
        separator = "\n\n"
        if not rendered:
            if len(block) > max_context_chars:
                if max_context_chars > 3:
                    rendered.append(block[: max_context_chars - 3] + "...")
                else:
                    rendered.append(block[:max_context_chars])
                break
            rendered.append(block)
            used_chars = len(block)
            continue

        if used_chars + len(separator) >= max_context_chars:
            break
        available = max_context_chars - used_chars - len(separator)
        if available <= 0:
            break
        if len(block) <= available:
            rendered.append(block)
            used_chars += len(separator) + len(block)
        else:
            if available > 3:
                rendered.append(block[:available - 3] + "...")
            else:
                rendered.append(block[:available])
            break

    return "\n\n".join(rendered)


def cmd_init(args: argparse.Namespace) -> int:
    """cmd init."""
    output_dir = Path(args.output).expanduser()
    if output_dir.suffix == ".json" and not output_dir.is_dir():
        output_dir = output_dir.parent
    output_dir.mkdir(parents=True, exist_ok=True)

    llm_fn, llm_batch_fn = _resolve_llm(args)
    graph, texts = split_workspace(args.workspace, llm_fn=llm_fn, llm_batch_fn=llm_batch_fn)
    if args.sessions is not None:
        replay_queries(graph=graph, queries=_load_session_queries(args.sessions))

    embedder_fn, embed_batch_fn, embedder_name, embedder_dim = _resolve_embedder(args, {})
    print(
        f"Embedding {len(texts)} texts ({embedder_name}, dim={embedder_dim})",
        file=sys.stderr,
    )
    index_vectors = embed_batch_fn(list(texts.items()))
    index = VectorIndex()
    for node_id, vector in index_vectors.items():
        index.upsert(node_id, vector)

    graph_path = output_dir / "graph.json"
    text_path = output_dir / "texts.json"
    state_meta = _state_meta({}, fallback_name=embedder_name, fallback_dim=embedder_dim)
    _write_graph(graph_path, graph, include_meta=True, meta=state_meta)
    save_state(
        graph=graph,
        index=index,
        path=output_dir / "state.json",
        meta=state_meta,
    )
    index_path = output_dir / "index.json"
    index_path.write_text(json.dumps(index_vectors, indent=2), encoding="utf-8")
    text_path.write_text(json.dumps(texts, indent=2), encoding="utf-8")

    if args.json:
        print(json.dumps({"graph": str(graph_path), "texts": str(text_path)}))
    return 0


def cmd_query(args: argparse.Namespace) -> int:
    """cmd query."""
    graph, index, meta = _resolve_graph_index(args)
    embed_fn, _, embedder_name, _ = _resolve_embedder(args, meta)
    if args.top <= 0:
        raise SystemExit("--top must be >= 1")

    if args.query_vector_stdin:
        if index is None:
            raise SystemExit("query-vector-stdin requires --index")
        query_vec = _load_query_vector_from_stdin()
        seeds = index.search(query_vec, top_k=args.top)
    elif index is not None:
        if embedder_name == HashEmbedder().name:
            _ensure_hash_embedder_compat(meta)
        query_vec = embed_fn(args.text)
        seeds = index.search(query_vec, top_k=args.top)
    else:
        seeds = _keyword_seeds(graph, args.text, args.top)

    result = traverse(
        graph=graph,
        seeds=seeds,
        config=TraversalConfig(max_hops=15, max_context_chars=args.max_context_chars),
        query_text=args.text,
    )
    log_query(
        query_text=args.text,
        fired_ids=result.fired,
        node_count=graph.node_count(),
        journal_path=_resolve_journal_path(args),
    )
    if args.json:
        print(json.dumps(_result_payload(result)))
    else:
        print(_query_text_output(result=result, graph=graph, max_context_chars=args.max_context_chars))
    return 0


def cmd_learn(args: argparse.Namespace) -> int:
    """cmd learn."""
    graph, index, meta = _resolve_graph_index(args)
    fired_ids = [value.strip() for value in args.fired_ids.split(",") if value.strip()]
    if not fired_ids:
        raise SystemExit("provide at least one fired id")

    updates = apply_outcome(graph, fired_nodes=fired_ids, outcome=args.outcome)
    if args.state is not None:
        state_meta = _state_meta(meta)
        save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=state_meta)
    updates_abs = [abs(delta) for delta in updates.values()]
    summary = {
        "edges_updated": len(updates),
        "max_weight_delta": max(updates_abs) if updates_abs else 0.0,
    }
    if args.state is None:
        payload = {"graph": _graph_payload(graph)}
        Path(args.graph).expanduser().write_text(json.dumps(payload, indent=2), encoding="utf-8")
    log_learn(fired_ids=fired_ids, outcome=args.outcome, journal_path=_resolve_journal_path(args))
    print(json.dumps(summary, indent=2) if args.json else f"updated {args.state or args.graph}")
    return 0


def cmd_merge(args: argparse.Namespace) -> int:
    """cmd merge."""
    graph, index, meta = _resolve_graph_index(args)
    llm_fn, llm_batch_fn = _resolve_llm(args)
    suggestions = suggest_merges(graph, llm_fn=llm_fn, llm_batch_fn=llm_batch_fn)
    applied = []
    for source_id, target_id in suggestions:
        if graph.get_node(source_id) and graph.get_node(target_id):
            merged = apply_merge(graph, source_id, target_id)
            applied.append({"from": [source_id, target_id], "to": [merged]})
    if args.state is not None:
        state_meta = _state_meta(meta)
        save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=state_meta)
    else:
        _write_graph(args.graph, graph)
    payload = {"suggestions": [{"from": [s, t]} for s, t in suggestions], "applied": applied}
    print(json.dumps(payload) if args.json else f"Applied merges: {len(applied)}")
    return 0


def cmd_connect(args: argparse.Namespace) -> int:
    """cmd connect."""
    graph, index, meta = _resolve_graph_index(args)
    llm_fn, llm_batch_fn = _resolve_llm(args)
    suggestions = suggest_connections(graph, llm_fn=llm_fn, llm_batch_fn=llm_batch_fn)
    added = apply_connections(graph=graph, connections=suggestions)
    if args.state is not None:
        state_meta = _state_meta(meta)
        save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=state_meta)
    else:
        _write_graph(args.graph, graph)
    payload = {
        "suggestions": [
            {"source_id": s, "target_id": t, "weight": w, "reason": r} for s, t, w, r in suggestions
        ],
        "added": added,
    }
    print(json.dumps(payload) if args.json else f"Added edges: {added}")
    return 0


def cmd_anchor(args: argparse.Namespace) -> int:
    """cmd anchor."""
    graph, index, meta = _resolve_graph_index(args)
    if args.list:
        nodes = [
            {"node_id": node.id, "authority": node.metadata.get("authority", "overlay")}
            for node in graph.nodes()
            if node.metadata.get("authority") in {"constitutional", "canonical"}
        ]
        payload = {"nodes": nodes, "count": len(nodes)}
        print(json.dumps(payload, indent=2) if args.json else "\n".join(f"{node['node_id']}: {node['authority']}" for node in nodes) or "No anchored nodes.")
        return 0

    if not args.node_id:
        raise SystemExit("--node-id required unless --list is set")
    node = graph.get_node(args.node_id)
    if node is None:
        raise SystemExit(f"node not found: {args.node_id}")

    current_authority = node.metadata.get("authority", "overlay")
    if args.remove:
        if "authority" in node.metadata:
            node.metadata.pop("authority", None)
            current_authority = "overlay"
            save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=_state_meta(meta))
        payload = {"node_id": args.node_id, "authority": current_authority}
        print(json.dumps(payload) if args.json else f"{args.node_id} authority: {current_authority}")
        return 0

    if args.authority:
        node.metadata["authority"] = args.authority
        current_authority = args.authority
        save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=_state_meta(meta))

    payload = {"node_id": args.node_id, "authority": current_authority}
    print(json.dumps(payload) if args.json else f"{args.node_id} authority: {current_authority}")
def cmd_compact(args: argparse.Namespace) -> int:
    """cmd compact."""
    if args.state is None:
        raise SystemExit("--state is required for compact")

    _, _, meta = _resolve_graph_index(args)

    embed_args = SimpleNamespace(embedder=None)
    embed_fn, _, _, _ = _resolve_embedder(embed_args, meta)
    if args.llm == "openai":
        from .openai_llm import openai_llm_fn

        llm_fn = openai_llm_fn
    else:
        llm_fn = None

    if embed_fn is None:
        raise SystemExit("embedding callback missing")

    report = compact_daily_notes(
        state_path=args.state,
        memory_dir=args.memory_dir,
        max_age_days=args.max_age_days,
        target_lines=args.target_lines,
        embed_fn=embed_fn,
        llm_fn=llm_fn,
        journal_path=_resolve_journal_path(args),
        dry_run=args.dry_run,
    )
    if args.json:
        print(json.dumps(asdict(report), indent=2))
    else:
        print(f"Compaction report for {args.state}")
        print(f"  scanned: {report.files_scanned}")
        print(f"  compacted: {report.files_compacted}")
        print(f"  skipped: {report.files_skipped}")
        print(f"  nodes_injected: {report.nodes_injected}")
        print(f"  lines: {report.lines_before} -> {report.lines_after}")
    return 0


def cmd_inject(args: argparse.Namespace) -> int:
    """cmd inject."""
    graph, index, meta = _resolve_graph_index(args)
    if args.state is None:
        raise SystemExit("--state is required for inject")
    if index is None:
        index = VectorIndex()
    embed_fn, _, embedder_name, _ = _resolve_embedder(args, meta)

    if args.vector_stdin:
        vector = _load_query_vector_from_stdin()
    else:
        if embedder_name == HashEmbedder().name:
            _ensure_hash_embedder_compat(meta)
        vector = None

    if args.summary is None:
        from ._util import _first_line

        summary = _first_line(args.content)
    else:
        summary = args.summary

    if args.connect_min_sim is not None:
        connect_min_sim = args.connect_min_sim
    else:
        connect_min_sim = 0.0 if embedder_name == "hash-v1" else 0.3

    node_type = args.type
    metadata = {"source": "cli_inject", "type": node_type}
    if node_type == "CORRECTION":
        payload = inject_correction(
            graph=graph,
            index=index,
            node_id=args.id,
            content=args.content,
            summary=summary,
            metadata=metadata,
            vector=vector,
            embed_fn=None if args.vector_stdin else embed_fn,
            connect_top_k=args.connect_top_k,
            connect_min_sim=connect_min_sim,
        )
    else:
        payload = inject_node(
            graph=graph,
            index=index,
            node_id=args.id,
            content=args.content,
            summary=summary,
            metadata=metadata,
            vector=vector,
            embed_fn=None if args.vector_stdin else embed_fn,
            connect_top_k=args.connect_top_k,
            connect_min_sim=connect_min_sim,
        )

    state_meta = _state_meta(
        meta,
    )
    save_state(graph=graph, index=index, path=args.state, meta=state_meta)

    print(json.dumps(payload, indent=2) if args.json else f"Injected {payload['node_id']}")
    return 0


def cmd_replay(args: argparse.Namespace) -> int:
    """cmd replay."""
    graph, index, meta = _resolve_graph_index(args)
    last_replayed_ts = meta.get("last_replayed_ts")
    if not isinstance(last_replayed_ts, (int, float)):
        last_replayed_ts = None

    query_records = _load_session_interactions(args.sessions, since_ts=last_replayed_ts)
    stats = replay_queries(
        graph=graph,
        queries=query_records,
        verbose=not args.json,
        since_ts=last_replayed_ts,
    )
    log_replay(
        queries_replayed=stats["queries_replayed"],
        edges_reinforced=stats["edges_reinforced"],
        cross_file_created=stats["cross_file_edges_created"],
        journal_path=_resolve_journal_path(args),
    )
    if args.state is not None:
        state_meta = _state_meta(meta)
        if stats.get("last_replayed_ts") is not None:
            state_meta["last_replayed_ts"] = stats["last_replayed_ts"]
        save_state(graph=graph, index=index or VectorIndex(), path=args.state, meta=state_meta)
    else:
        _write_graph(args.graph, graph)
    print(
        json.dumps(stats, indent=2)
        if args.json
        else f"Replayed {stats['queries_replayed']}/{len(query_records)} queries, {stats['cross_file_edges_created']} cross-file edges created"
    )
    return 0


def cmd_doctor(args: argparse.Namespace) -> int:
    """cmd doctor."""
    checks_passed = 0
    checks_total = 0
    failed = False

    checks_total += 1
    python_ok = sys.version_info >= (3, 10)
    checks_passed += _check_result(python_ok, "python_version", f"{sys.version.split()[0]}")
    failed = failed or (not python_ok)

    state_path = Path(args.state).expanduser()
    checks_total += 1
    state_exists = state_path.exists()
    checks_passed += _check_result(state_exists, "state_file_exists", str(state_path))
    failed = failed or (not state_exists)
    if not state_exists:
        print(f"Summary: {checks_passed}/{checks_total} checks passed")
        return 1

    checks_total += 1
    try:
        payload = _load_json(args.state)
        checks_passed += _check_result(True, "state_json_valid", str(args.state))
    except SystemExit as exc:
        checks_passed += _check_result(False, "state_json_valid", str(exc))
        print(f"Summary: {checks_passed}/{checks_total} checks passed")
        return 1
    graph_payload = payload.get("graph", {})
    checks_total += 1
    if not isinstance(graph_payload, dict):
        checks_passed += _check_result(False, "state_json_valid", "graph payload must be object")
        print(f"Summary: {checks_passed}/{checks_total} checks passed")
        return 1

    graph = _load_graph(args.state)
    index_payload = payload.get("index", {})
    checks_total += 1
    checks_passed += _check_result(
        bool(graph_payload.get("nodes")),
        "graph_has_nodes",
        f"nodes={graph.node_count()}",
    )
    failed = failed or (not graph_payload.get("nodes"))

    checks_total += 1
    checks_passed += _check_result(
        graph.edge_count() > 0,
        "graph_has_edges",
        f"edges={graph.edge_count()}",
    )
    failed = failed or (graph.edge_count() == 0)

    checks_total += 1
    embedder_name = payload.get("meta", {}).get("embedder_name")
    embedder_dim = payload.get("meta", {}).get("embedder_dim")
    checks_passed += _check_result(
        isinstance(embedder_name, str) and isinstance(embedder_dim, int),
        "embedder_metadata_present",
        f"name={embedder_name}, dim={embedder_dim}",
    )
    failed = failed or (not isinstance(embedder_name, str) or not isinstance(embedder_dim, int))

    checks_total += 1
    if not isinstance(embedder_dim, int):
        checks_passed += _check_result(False, "index_dimension_matches_embedder", "missing embedder_dim")
        failed = True
    else:
        if not isinstance(index_payload, dict):
            checks_passed += _check_result(False, "index_dimension_matches_embedder", "index payload must be an object")
            failed = True
        elif not index_payload:
            checks_passed += _check_result(False, "index_dimension_matches_embedder", "missing index payload")
            failed = True
        else:
            index_dims: set[int] = set()
            for node_id, vector in index_payload.items():
                if not isinstance(vector, list):
                    checks_passed += _check_result(False, "index_dimension_matches_embedder", f"{node_id}: not a list")
                    failed = True
                    break
                index_dims.add(len(vector))
            else:
                dim_ok = len(index_dims) == 1 and next(iter(index_dims)) == embedder_dim
                dim_value = next(iter(index_dims), None)
                checks_passed += _check_result(dim_ok, "index_dimension_matches_embedder", f"index_dim={dim_value}")
                failed = failed or (not dim_ok)

    checks_total += 1
    journal_path = _resolve_journal_path(args)
    if journal_path is not None and Path(journal_path).exists():
        try:
            Path(journal_path).open("a", encoding="utf-8").close()
            journal_ok = True
        except OSError:
            journal_ok = False
        checks_passed += _check_result(journal_ok, "journal_writable", str(journal_path))
        failed = failed or (not journal_ok)
    else:
        checks_passed += _check_result(True, "journal_writable", "not present")

    print(f"Summary: {checks_passed}/{checks_total} checks passed")
    return 1 if failed else 0


def cmd_info(args: argparse.Namespace) -> int:
    """cmd info."""
    if getattr(args, "state", None) is not None:
        payload, _, _, graph = _state_payload(args.state)
        meta = payload.get("meta", {})
        state_bytes = Path(args.state).expanduser().stat().st_size
        embedder_name = meta.get("embedder_name", "n/a")
        embedder_dim = meta.get("embedder_dim", "n/a")
    else:
        graph = _load_graph(args.graph)
        state_bytes = Path(args.graph).expanduser().stat().st_size
        embedder_name = "n/a"
        embedder_dim = "n/a"

    health = measure_health(graph)
    payload = {
        "version": __version__,
        "node_count": graph.node_count(),
        "edge_count": graph.edge_count(),
        "embedder_name": embedder_name,
        "embedder_dim": embedder_dim,
        "dormant_pct": health.dormant_pct * 100,
        "habitual_pct": health.habitual_pct * 100,
        "reflex_pct": health.reflex_pct * 100,
        "journal_entry_count": _journal_entry_count(_resolve_journal_path(args)),
        "state_file_size": state_bytes,
    }
    if args.json:
        print(json.dumps(payload, indent=2))
        return 0

    print("\n".join(
        [
            f"version: {payload['version']}",
            f"node_count: {payload['node_count']}",
            f"edge_count: {payload['edge_count']}",
            f"embedder_name: {payload['embedder_name']}",
            f"embedder_dim: {payload['embedder_dim']}",
            f"dormant_pct: {payload['dormant_pct']:.2f}",
            f"habitual_pct: {payload['habitual_pct']:.2f}",
            f"reflex_pct: {payload['reflex_pct']:.2f}",
            f"journal_entry_count: {payload['journal_entry_count'] if payload['journal_entry_count'] is not None else 'n/a'}",
            f"state_file_size: {payload['state_file_size']}",
        ]
    ))
    return 0


def _parse_authority_map(raw: str | None) -> dict[str, str]:
    """Parse authority map json string."""
    if raw is None:
        return dict(DEFAULT_AUTHORITY_MAP)
    payload = json.loads(raw)
    if not isinstance(payload, dict):
        raise SystemExit("--authority-map must be a JSON object")

    parsed: dict[str, str] = {}
    for key, value in payload.items():
        if not isinstance(key, str) or not isinstance(value, str):
            raise SystemExit("--authority-map must map string keys to string values")
        parsed[key] = value
    return parsed


def cmd_sync(args: argparse.Namespace) -> int:
    """cmd sync."""
    authority_map = _parse_authority_map(getattr(args, "authority_map", None))
    graph, index, meta = _resolve_graph_index(args)

    embed_fn, embed_batch_fn, _, _ = _resolve_embedder(args, meta)

    if args.dry_run:
        with tempfile.TemporaryDirectory() as tmp_dir:
            tmp_state = Path(tmp_dir) / "state.json"
            shutil.copy2(args.state, tmp_state)
            report = sync_workspace(
                state_path=str(tmp_state),
                workspace_dir=args.workspace,
                embed_fn=embed_fn,
                embed_batch_fn=embed_batch_fn,
                journal_path=None,
                authority_map=authority_map,
            )
            if args.json:
                print(json.dumps(asdict(report), indent=2))
            else:
                print(
                    f"sync report: +{report.nodes_added}/~{report.nodes_updated} "
                    f"-{report.nodes_removed} ={report.nodes_unchanged} unchanged | "
                    f"{report.embeddings_computed} embeddings"
                )
            return 0

    report = sync_workspace(
        state_path=args.state,
        workspace_dir=args.workspace,
        embed_fn=embed_fn,
        embed_batch_fn=embed_batch_fn,
        journal_path=_resolve_journal_path(args),
        authority_map=authority_map,
    )

    if args.json:
        print(json.dumps(asdict(report), indent=2))
    else:
        print(
            f"sync report: +{report.nodes_added}/~{report.nodes_updated} "
            f"-{report.nodes_removed} ={report.nodes_unchanged} unchanged | "
            f"{report.embeddings_computed} embeddings"
        )
    return 0


def cmd_health(args: argparse.Namespace) -> int:
    """cmd health."""
    graph, _, _ = _resolve_graph_index(args)
    payload = measure_health(graph).__dict__
    payload["nodes"] = graph.node_count()
    payload["edges"] = graph.edge_count()
    log_health(payload, journal_path=_resolve_journal_path(args))
    if args.json:
        print(json.dumps(payload, indent=2))
        return 0
    print(
        "\n".join(
            [
                "Brain health:",
                f"  Nodes: {payload['nodes']}",
                f"  Edges: {payload['edges']}",
                f"  Reflex: {payload['reflex_pct']:.1%}  Habitual: {payload['habitual_pct']:.1%}  Dormant: {payload['dormant_pct']:.1%}",
                f"  Orphans: {payload['orphan_nodes']}",
                f"  Cross-file edges: {payload['cross_file_edge_pct']:.1%}",
            ]
        )
    )
    return 0


def cmd_maintain(args: argparse.Namespace) -> int:
    """cmd maintain."""
    requested_tasks = [task.strip() for task in args.tasks.split(",") if task.strip()]
    llm_fn, _ = _resolve_llm(args)
    report = run_maintenance(
        state_path=args.state,
        tasks=requested_tasks,
        llm_fn=llm_fn,
        journal_path=_resolve_journal_path(args),
        dry_run=args.dry_run,
        max_merges=args.max_merges,
        prune_below=args.prune_below,
    )
    if args.json:
        print(json.dumps(asdict(report), indent=2))
        return 0

    print("\n".join([
        "Maintenance report:",
        f"  tasks: {', '.join(report.tasks_run) if report.tasks_run else '(none)'}",
        f"  nodes: {report.health_before['nodes']} -> {report.health_after['nodes']}",
        f"  edges: {report.edges_before} -> {report.edges_after}",
        f"  merges: {report.merges_applied}/{report.merges_proposed}",
        f"  pruned: edges={report.pruned_edges} nodes={report.pruned_nodes}",
        f"  decay_applied: {report.decay_applied}",
        f"  dry_run: {args.dry_run}",
    ]))
    if report.notes:
        print(f"  notes: {', '.join(report.notes)}")
    return 0


def cmd_daemon(args: argparse.Namespace) -> int:
    """cmd daemon."""
    from .daemon import main as daemon_main

    return daemon_main(
        [
            "--state",
            str(args.state),
            "--embed-model",
            args.embed_model,
            "--auto-save-interval",
            str(args.auto_save_interval),
        ]
    )


def cmd_journal(args: argparse.Namespace) -> int:
    """cmd journal."""
    journal_path = _resolve_journal_path(args)
    if args.stats:
        print(
            json.dumps(journal_stats(journal_path=journal_path), indent=2)
            if args.json
            else "\n".join(
                f"{k}: {v}"
                for k, v in journal_stats(journal_path=journal_path).items()
                if k != "avg_fired_per_query"
            )
        )
        return 0
    entries = read_journal(last_n=args.last, journal_path=journal_path)
    print(
        json.dumps(entries, indent=2)
        if args.json
        else "\n".join(f"{idx+1:>2}. {entry.get('type')} @ {entry.get('iso', entry.get('ts', ''))}: {entry}" for idx, entry in enumerate(entries))
        or "No entries."
    )
    return 0


def main(argv: list[str] | None = None) -> int:
    """main."""
    args = _build_parser().parse_args(argv)
    return {
        "init": cmd_init,
        "query": cmd_query,
        "learn": cmd_learn,
        "merge": cmd_merge,
        "maintain": cmd_maintain,
        "anchor": cmd_anchor,
        "connect": cmd_connect,
        "compact": cmd_compact,
        "daemon": cmd_daemon,
        "inject": cmd_inject,
        "replay": cmd_replay,
        "health": cmd_health,
        "journal": cmd_journal,
        "doctor": cmd_doctor,
        "info": cmd_info,
        "sync": cmd_sync,
    }[args.command](args)


if __name__ == "__main__":
    raise SystemExit(main())


path: crabpath/decay.py
content: """Edge weight decay mechanics."""

from __future__ import annotations

from collections.abc import Callable
from dataclasses import dataclass

from .graph import Graph


@dataclass
class DecayConfig:
    """Decay control for weights in long-running graphs."""

    half_life: int = 80
    min_weight: float = 0.01


def apply_decay(
    graph: Graph,
    config: DecayConfig | None = None,
    *,
    skip_source_ids: set[str] | None = None,
    source_half_life_scale: Callable[[str], float] | None = None,
) -> int:
    """Apply decay to edge weights with optional source-level authority controls.

    ``skip_source_ids`` prevents decay for edges whose source node ID is included.
    ``source_half_life_scale`` returns a multiplier for the source's decay
    half-life.
    """
    cfg = config or DecayConfig()
    if cfg.half_life <= 0:
        return 0

    base_factor = 0.5 ** (1.0 / cfg.half_life)
    changed = 0

    for source_id, source_edges in graph._edges.items():
        if skip_source_ids is not None and source_id in skip_source_ids:
            continue
        scale = 1.0
        if source_half_life_scale is not None:
            try:
                scale = float(source_half_life_scale(source_id))
            except (TypeError, ValueError):
                scale = 1.0
        if scale <= 0:
            continue

        factor = base_factor if scale == 1 else 0.5 ** (1.0 / (cfg.half_life * scale))
        for edge in source_edges.values():
            old = edge.weight
            new = old * factor
            if abs(new) < cfg.min_weight:
                new = 0.0
            if new != old:
                edge.weight = new
                changed += 1

    return changed


path: crabpath/embeddings.py
content: """Local embeddings. Install: pip install crabpath[embeddings]"""

from __future__ import annotations


def local_embed_fn(text: str) -> list[float]:
    """local embed fn."""
    if not hasattr(local_embed_fn, "_model"):
        try:
            from sentence_transformers import SentenceTransformer
        except ImportError:
            raise ImportError("pip install crabpath[embeddings]") from None
        local_embed_fn._model = SentenceTransformer("all-MiniLM-L6-v2")
    return local_embed_fn._model.encode(text).tolist()


def local_embed_batch_fn(texts: list[tuple[str, str]]) -> dict[str, list[float]]:
    """local embed batch fn."""
    if not hasattr(local_embed_batch_fn, "_model"):
        try:
            from sentence_transformers import SentenceTransformer
        except ImportError:
            raise ImportError("pip install crabpath[embeddings]") from None
        local_embed_batch_fn._model = SentenceTransformer("all-MiniLM-L6-v2")
    model = local_embed_batch_fn._model
    ids = [nid for nid, _ in texts]
    contents = [content for _, content in texts]
    vectors = model.encode(contents).tolist()
    return dict(zip(ids, vectors))


path: examples/hash_embedder/README.md
content: # hash_embedder

Zero dependencies, works fully offline.

Run:

```bash
python3 examples/hash_embedder/run.py ./workspace
```

This example uses the built-in `HashEmbedder` (hash-v1, 1024-dim) and writes `state.json` in the output.


path: examples/openai_embedder/README.md
content: # openai_embedder

Requires `openai` package and `OPENAI_API_KEY` in the process environment.

Run:

```bash
OPENAI_API_KEY=... python3 examples/openai_embedder/run.py ./workspace
```

The callback uses real embeddings (for example `text-embedding-3-small`) and persists embedder metadata (`name`, `dim`, `schema_version`) into state.


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)