Skill: edgepro001/clawexpert (version 1.0.0)

--- name: clawexpert description: > Domain expert learning system with persistent local knowledge base. READ THIS SKILL FILE at the start of every session — learned topics are stored locally and should be used to answer related questions with source citations instead of relying on parametric memory. Use /clawexpert learn <topic> to build a knowledge base, /clawexpert status to list learned topics, or ask any domain question (check local KB first before answering from training data). user-invocable: true metadata: { "openclaw": { "emoji": "🧠", "skillKey": "clawexpert", "always": true, "homepage": "https://github.com/EdgePro001/ClawExpert", }, } --- # ClawExpert — Domain Expert Learning System > Make your Claw a Self-Evolving Domain Expert You are a domain expert system with **long-running autonomous learning capability**. You can research any topic for hours — searching, downloading, and organizing materials — and consolidate the results into an **indexed, citable local knowledge base**. ⛔ **Global prohibition**: Never execute git add / git commit / git push or any other git operations at any point in any workflow. Writing knowledge base files to disk is sufficient; version control is not needed. ## Knowledge Base Path ``` Default path: ~/.openclaw/workspace/knowledge/ Override via: $CLAWEXPERT_KNOWLEDGE_DIR ``` Throughout this document, `{KB}` refers to the actual knowledge base root path (environment variable takes priority over the default). ## Bundled Scripts Prefer the bundled `scripts/` directory for deterministic filesystem operations instead of ad-hoc inline Python. - When resolving `{SCRIPT_DIR}`, check in order: 1. `$CLAWEXPERT_SCRIPT_DIR` env var if set (preferred — set in openclaw.json for non-standard installs) 2. The `scripts/` directory adjacent to this `SKILL.md` file 3. `~/.openclaw/workspace/skills/clawexpert/scripts` (default workspace install fallback) --- ## Session Start: Auto-detect Knowledge Base **At the beginning of every session**, perform the following check silently (do not inform the user): ```bash KNOWLEDGE_DIR="${CLAWEXPERT_KNOWLEDGE_DIR:-$HOME/.openclaw/workspace/knowledge}" ``` **Step 0: Check master switch (enabled)** ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') try: enabled = json.load(open(state)).get('enabled', True) except Exception: enabled = True print('on' if enabled else 'off') " ``` - Result is `off` → **immediately exit all ClawExpert logic**. Do not read knowledge base, do not run proactive checks, do not intercept the conversation. Respond as a normal assistant with no ClawExpert behavior. - Result is `on` (or file missing) → continue to Step 1. **Step 1: Read _root_index.md (preferred)** ```bash cat "$KNOWLEDGE_DIR/_index/_root_index.md" 2>/dev/null ``` - File exists → Read the full knowledge base category map, build meta-awareness of "I have learned A/B, not C". Before answering any question, first judge whether it falls within learned topics; if yes, navigate to the relevant category; if not, explicitly state "no knowledge base for this topic". - File not found → Proceed to Step 2 (compatibility mode) **Step 2: Scan slug directories (v0.1.0 compatibility mode)** ```bash ls "$KNOWLEDGE_DIR" 2>/dev/null ``` - Directory empty or missing → Stay silent, proceed with normal conversation - Slug directories exist → Read each `meta.json` topic field to learn what topics have been studied; proactively read the relevant `index.md` when answering related questions This ensures accurate awareness of your own knowledge state at the start of every session, prioritizing the local knowledge base over parametric memory. **Step 3: Detect interrupted learn sessions (resume guard)** After the knowledge state check, scan for any topic with an incomplete learn session: ```bash python3 -c " import json, glob, os kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) for path in glob.glob(kb + '/*/meta.json'): m = json.load(open(path)) if m.get('status') == 'learning': slug = m['slug'] flags = glob.glob(kb + '/' + slug + '/done-*.flag') expected = len(m.get('subtopics', [])) print(f'INTERRUPTED slug={slug} flags={len(flags)} expected={expected}') " ``` - If any topic shows `status=learning` and flag count >= expected subtopic count → **all subagents finished but merge never ran**. Output: `⏭️ [60%] Resuming interrupted session: slug={slug} ({flags}/{expected} subtopics complete). Jumping to merge...` Then immediately resume from Step 5 for that topic without waiting for user input. - If flag count < expected → subagents still running or timed out. Inform the user and offer to wait or proceed with partial results. **Step 4: Check proactive mode** Proactive mode is stored in `~/.openclaw/workspace/clawexpert-state.json` and persists across sessions and terminal restarts. Read it now and hold it as the **session's active mode** — all B1/B3 decisions this session refer back to this value (or re-read the file when uncertain). ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') try: mode = json.load(open(state)).get('proactive_mode', 'off') except Exception: mode = 'off' print(f'[ClawExpert] Proactive mode: {mode}') " ``` The output `[ClawExpert] Proactive mode: {mode}` is written to the conversation context so the current mode is always visible. Valid values are exactly **off / suggest / auto** — no other values are valid; treat any unexpected value as `off`. - `off` → skip all proactive logic this session (B1 and B3 are disabled) - `suggest` or `auto` → enable B1 topic tracking and B3 queue display at the end of Expert answers --- ## I. Slash Commands ### `/clawexpert init` Initialize ClawExpert for first-time use. Safe to re-run — existing data is never overwritten. **Step 1: Create knowledge base directory** ```bash python3 -c " import os kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) os.makedirs(kb + '/_index', exist_ok=True) print('kb=' + kb) " ``` **Step 2: Create default `_categories.json` if missing** ```bash python3 -c " import json, os kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) path = kb + '/_categories.json' if not os.path.exists(path): default = {'categories': [ {'id': 'science-tech', 'label': '科学技术'}, {'id': 'economics-finance', 'label': '经济金融'}, {'id': 'humanities-social', 'label': '人文社科'}, {'id': 'health-medicine', 'label': '健康医疗'}, {'id': 'other', 'label': '其他'} ]} json.dump(default, open(path, 'w'), indent=2, ensure_ascii=False) print('created') else: print('exists') " ``` **Step 3: Create `clawexpert-state.json` if missing** ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') if not os.path.exists(state): os.makedirs(os.path.dirname(state), exist_ok=True) json.dump({'enabled': True, 'proactive_mode': 'off', 'initialized': True}, open(state, 'w'), indent=2) print('created') else: print('exists') " ``` **Step 4: Write OpenClaw memory pointer** Write to OpenClaw memory so future sessions know ClawExpert is active: ``` ClawExpert initialized. Knowledge base at: {KB} Proactive mode: off. Run /clawexpert proactive suggest to enable learning suggestions. ``` **Step 5: Output success** ``` ✅ ClawExpert initialized successfully. Knowledge base: {KB} Proactive mode: off Master switch: on Next steps: /clawexpert learn <topic> — start learning a new topic /clawexpert on|off — toggle ClawExpert on or off /clawexpert proactive suggest — enable learning suggestions ``` --- ### `/clawexpert on` / `/clawexpert off` Toggle the ClawExpert master switch. When **off**, ClawExpert is completely silent — no knowledge base lookups, no proactive tracking, no session interception. Useful for demos comparing ClawExpert vs. standard assistant responses. **Turn off:** ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') d = json.load(open(state)) if os.path.exists(state) else {} d['enabled'] = False json.dump(d, open(state, 'w'), indent=2) print('off') " ``` Output: `🔴 ClawExpert OFF — responding as standard assistant. Run /clawexpert on to re-enable.` **Turn on:** ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') d = json.load(open(state)) if os.path.exists(state) else {} d['enabled'] = True json.dump(d, open(state, 'w'), indent=2) print('on') " ``` Output: `🟢 ClawExpert ON — knowledge base active. Proactive mode: {current_mode}.` --- ### `/clawexpert learn <topic>` Start systematic learning on a specified topic. See the "Learner" section for the full workflow. ``` /clawexpert learn RL reasoning 最新进展 /clawexpert learn --hours 3 A股市场分析框架 /clawexpert learn --deepen rl-reasoning-2026 /clawexpert learn --auto ``` Parameters: - `--hours <n>`: Max learning duration in hours (default: 2) - `--deepen <slug>`: Continue learning on an existing topic using both breadth expansion and depth questioning (append, do not overwrite) - `--auto`: Auto mode (for Cron). Scans all topics, runs `--deepen` on any with `status="partial"` or unchecked items in the "Further Directions" list. Exits silently if nothing to do. ### `/clawexpert superlearn <topic-or-slug>` Run end-to-end deep research on a topic. If the topic already exists, `superlearn` begins with deepen rounds. If it does not exist yet, `superlearn` must first run an initial `learn`, then continue with deepen rounds using the **same shared total budget**. ``` /clawexpert superlearn llm-memory-方法 /clawexpert superlearn 中国和美国的关系 --hours 6 /clawexpert superlearn llm-memory-方法 --rounds 3 /clawexpert superlearn llm-memory-方法 --rounds 4 --hours 8 ``` Parameters: - `--rounds <n>`: Minimum deepen rounds that must be completed before stopping - `--hours <n>`: Minimum wall-clock duration across the entire workflow before stopping - Budget semantics: - If only `--rounds` is provided: treat it as a minimum round guarantee (`MAX_HOURS=0`) - If only `--hours` is provided: treat it as a minimum time guarantee (`MAX_ROUNDS=0`) - If both are provided: both minimums must be satisfied before stopping - If neither is provided: default to `MAX_ROUNDS=2`, `MAX_HOURS=0` as the default minimum contract - `superlearn` must not stop early before the requested minimum is satisfied - To avoid extending too far beyond the user's intent, stop at the **first round boundary after the minimum contract is satisfied** - Bootstrap semantics: - If the input resolves to an existing slug/topic: start directly from deepen - If the topic does not exist yet: run one initial `learn` first, then use the remaining budget for deepen rounds - The initial `learn` consumes from the same total time budget when `--hours` is active - Each round may contain: - breadth tasks: new subtopics generated from `Further Directions` - depth tasks: targeted deep-dive questions for existing nodes ### `/clawexpert status` Show the status of all topics in the knowledge base. **Step 1: Read proactive state** ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) mode = json.load(open(state)).get('proactive_mode', 'off') if os.path.exists(state) else 'off' q_path = kb + '/_proactive_queue.json' q = json.load(open(q_path)).get('queue', []) if os.path.exists(q_path) else [] print(f'kb={kb} mode={mode} queue={len(q)}') " ``` **Step 2: Read all topic meta.json files** ```bash python3 -c " import json, os, glob kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) for meta_path in sorted(glob.glob(kb + '/*/meta.json')): m = json.load(open(meta_path)) slug = m.get('slug', '') topic = m.get('topic', slug) status = m.get('status', 'unknown') nodes = m.get('node_count', 0) sources = m.get('source_count', 0) updated = (m.get('last_updated') or '')[:10] print(f'{status}|{topic}|{slug}|{nodes}|{sources}|{updated}') " ``` **Step 3: Output formatted summary** ``` 📚 ClawExpert Knowledge Base Path: {KB} Proactive mode: {mode} ({N} items in queue) Topics ({count}): {status_icon} {topic} ({slug}) — {nodes} nodes, {sources} sources, updated {date} ... ``` Status icons: `✅` complete · `⏳` learning · `🔄` partial · `❓` unknown If knowledge base is empty: ``` 📚 ClawExpert Knowledge Base is empty. Run /clawexpert learn <topic> to start building your knowledge base. ``` ### `/clawexpert show <slug>` Display the full knowledge overview for a topic. **Step 1: Validate slug exists** ```bash python3 -c " import os, json kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) meta = kb + '/{slug}/meta.json' if not os.path.exists(meta): print('NOT_FOUND') else: m = json.load(open(meta)) print(f\"topic={m.get('topic')} status={m.get('status')} nodes={m.get('node_count',0)} sources={m.get('source_count',0)}\") " ``` If `NOT_FOUND`: output `❌ Topic '{slug}' not found. Run /clawexpert status to see available topics.` **Step 2: Read and display index.md** Read `{KB}/{slug}/index.md` and display its full contents. **Step 3: List knowledge nodes** ```bash python3 -c " import os, glob kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) nodes = sorted(glob.glob(kb + '/{slug}/nodes/node-*.md')) for n in nodes: print(os.path.basename(n)) print(f'Total: {len(nodes)} nodes') " ``` Output node list below the index summary: ``` 📂 Knowledge nodes ({N} total): • node-001.md — {first line of file or filename} • node-002.md — ... ``` ### `/clawexpert ask <slug> <question>` Force an answer based solely on the specified topic's knowledge base, without live search. See the "Expert" section for the retrieval workflow. ### `/clawexpert forget <slug>` Delete all knowledge for a topic. **Must ask the user to confirm** before executing: ```bash rm -rf "{KB}/{slug}" ``` ### `/clawexpert export <slug>` Export a topic's knowledge tree as a single Markdown file. Steps: 1. Read `{KB}/{slug}/index.md` as the header 2. Read all `{KB}/{slug}/nodes/node-*.md` in filename order 3. Join with `---` separators 4. Write to `{KB}/{slug}/export-{YYYY-MM-DD}.md` 5. Tell the user the export path ### `/clawexpert import <filepath> [--topic <slug>]` Import a local PDF file directly into a topic's knowledge base, bypassing the search phase. Text is extracted from the file and a knowledge node is written immediately. ``` /clawexpert import ~/Downloads/attention-is-all-you-need.pdf --topic transformer /clawexpert import /path/to/textbook.pdf ``` Parameters: - `--topic <slug>`: Target topic slug. If omitted, ClawExpert lists existing topics and prompts for selection. A new topic is created if the slug doesn't exist (with user confirmation). ### `/clawexpert proactive [off|suggest|auto]` View or change the proactive learning mode. - No argument → display current mode and queue size - `off` → disable all proactive logic (no topic tracking, no suggestions) - `suggest` → enable suggestions: gaps are recorded, shown at session start as recommendations - `auto` → enable auto-learning: top-priority gap is automatically learned at session start Read current mode: ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') mode = json.load(open(state)).get('proactive_mode', 'off') if os.path.exists(state) else 'off' kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) q_path = kb + '/_proactive_queue.json' q = json.load(open(q_path)).get('queue', []) if os.path.exists(q_path) else [] print(f'mode={mode} queue={len(q)}') " ``` Write new mode (replace `{MODE}` with `off`, `suggest`, or `auto`): ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') d = json.load(open(state)) if os.path.exists(state) else {} d['proactive_mode'] = '{MODE}' json.dump(d, open(state, 'w'), indent=2) print('Proactive mode set to: {MODE}') " ``` Output after change: ``` Proactive mode: suggest Token cost: +~200 tokens/session (queue scan + display only) suggest: +~200 tokens/session (queue scan + display only) auto: +N × learn cost/session (depends on queue size; each learn run can take minutes) ``` ### `/clawexpert proactive skip` Clear the current proactive learning queue without executing any learns. ```bash python3 -c " import json, os path = os.path.expanduser(os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', '~/.openclaw/workspace/knowledge')) + '/_proactive_queue.json' data = json.load(open(path)) if os.path.exists(path) else {} data['queue'] = [] data['updated'] = __import__('datetime').datetime.utcnow().isoformat() + 'Z' json.dump(data, open(path, 'w'), indent=2) print('Queue cleared.') " ``` ### `/clawexpert radar` Manually trigger a radar scan across all L1 categories. See **V. Expert → Radar Scan** for the full workflow. --- ## Source Priority Standards All learning and retrieval operations must apply the following four-tier source ranking. Higher tiers take precedence when sources conflict or must be selected. ### Tier Definitions | Tier | Name | Characteristics | |------|------|-----------------| | **T1** | Primary authoritative | Peer-reviewed / institutionally validated / officially published | | **T2** | Institutional professional | Editorial standards from professional institutions; close to primary | | **T3** | Quality specialist media | Fact-checked, professionally edited, domain-specific | | **T4** | Reference only | Background orientation; do not use as the sole basis for key claims | | **Discard** | — | Content farms, SEO sites, unattributed, clear commercial bias | ### By Domain **Science & Technology** - T1: Peer-reviewed papers (arXiv, Nature, Science, IEEE, ACM, NEJM, Cell); official technical standards (RFC, W3C, ISO/IEC) - T2: Official technical documentation (language specs); major company engineering blogs (Google AI, Meta Research, DeepMind); government research agency reports (NIH, NASA, CERN) - T3: IEEE Spectrum, MIT Technology Review, Wired - T4: Wikipedia (concept orientation only — not citable as evidence) **Economics & Finance** - T1: Government statistics (National Bureau of Statistics, Fed, ECB, IMF, World Bank); academic papers (NBER, SSRN, AER, QJE, Journal of Finance) - T2: Regulatory filings and announcements (SEC, CSRC, CBIRC annual reports); major financial institution research reports (Goldman Sachs, JP Morgan Research — note conflict-of-interest risk) - T3: FT, WSJ, Bloomberg, Reuters, The Economist - T4: Financial news portals, earnings commentary articles **Humanities, History & Social Sciences** - T1: Primary historical documents, government archives, original survey data; peer-reviewed academic journals (JSTOR-indexed, leading history/sociology journals) - T2: University press publications; major think tank reports (Brookings, RAND, DRC) - T3: Quality newspaper opinion/analysis (NYT, Guardian, in-depth investigative pieces); academic blogs written by scholars (LSE Blog, VoxEU) - T4: Popular history/culture media (cross-verify with T1/T2) **Law & Policy** - T1: Legal primary texts (statutes, court decisions, treaty texts); official policy documents (full text of regulations) - T2: Leading law journals; legislative working papers - T3: Specialist legal media; law firm analysis reports (note: not legal advice) - T4: Legal explainer articles **Medicine & Health** - T1: Systematic reviews and meta-analyses (Cochrane Library); top clinical journals (NEJM, Lancet, JAMA, BMJ); official health guidelines (WHO, CDC, NIH) - T2: Hospital and research institution clinical study reports - T3: Science, Nature News & Views, specialist medical journalism - T4: Health popularization articles (only if backed by T1/T2 sources) ### Cross-Domain Rules 1. Within the same tier: institutional primary source > third-party reporting; raw data present > conclusions only; recent (within 2 years) > outdated (except canonical references) 2. When media reports on a study, locate the original paper — the media report is demoted to T3/T4 reference 3. Multiple independent sources converging on the same conclusion outweighs a single source 4. Preprints (e.g., arXiv before peer review): treat as T1.5 — high credibility but flag as "not yet peer-reviewed" in node conclusions 5. Official institutional social media accounts (e.g., central bank official accounts, WHO on Twitter/X): treat as T2 — represents official position but lacks the editorial review of formal publications --- ## II. Learner Upon receiving `/clawexpert learn <topic>`, execute the following steps strictly in order. ### Step 1: Initialization Immediately output to the user: `🚀 [0%] Initializing: "{topic}"...` **Category system check** (run before learning): ```bash ls "{KB}/_categories.json" 2>/dev/null ``` - File exists → Read the L1 category list, continue - File not found → Guide the user through first-time setup: ``` Welcome to ClawExpert! First-time setup requires you to define the top-level (L1) categories for your knowledge base. Suggested categories: Science & Technology / Economics & Finance / Humanities & Social Sciences / History & Geography / Medicine & Health / Law & Politics / Arts & Culture Please enter your L1 categories (comma-separated): ``` After receiving user input, write to `{KB}/_categories.json`: ```json { "version": 1, "created": "{ISO 8601 当前时间}", "categories": [ {"id": "sci-tech", "label": "科学技术"}, {"id": "economics", "label": "经济金融"} ] } ``` (`id` is auto-generated from the label: lowercased + spaces converted to hyphens + non-ASCII removed, e.g. "科学技术" → `sci-tech` — model should choose a reasonable name) --- **Slug generation rules**: - Lowercase - Convert spaces and punctuation to hyphens - Remove non-displayable special characters (keep Chinese/English characters, digits, hyphens) - Truncate to 60 characters - Example: "RL Reasoning 最新进展" → `rl-reasoning-最新进展` **Conflict check (same name)**: ```bash # Step 1: Check if it exists ls "{KB}/{slug}/meta.json" 2>/dev/null # Step 2: If it exists, read the status cat "{KB}/{slug}/meta.json" ``` - File not found → Proceed to cross-topic check - Exists and JSON `status` field = `"learning"` → Inform user "This topic is currently being learned, please wait for it to finish", abort - Exists and `status` = `"complete"` → Ask user whether to deepen learning (equivalent to `--deepen`) - Exists and `status` = `"partial"` → Continue (treat this learn as a supplement) **Conflict check (cross-topic similarity)** (run after same-name check passes): ```bash # Scan all existing topics' meta.json ls "{KB}/*/meta.json" 2>/dev/null ``` Read the `topic` and `keywords` fields from each meta.json and determine whether the new topic heavily overlaps with any existing topic in core subject area (semantically similar topic name + large keyword overlap). When high overlap is detected, prompt the user: ``` Found a related existing topic: {existing topic name} ({slug}) Would you like to run --deepen on it instead of creating a new entry? - Enter y: Run --deepen on the existing topic (recommended — avoids duplicate learning) - Enter n: Continue creating a new independent knowledge base ``` No overlap, or user chooses to create new → Continue with subsequent steps. **Create directories and write initial meta.json via bundled script**: ```bash python3 "{SCRIPT_DIR}/init_topic.py" \ --kb "{KB}" \ --topic "{user's original topic name as entered}" \ --slug "{slug}" \ --max-hours {--hours parameter, default 2} ``` The script creates: - `{KB}/{slug}/raw/web/` - `{KB}/{slug}/raw/pdf/` - `{KB}/{slug}/nodes/` - `{KB}/{slug}/meta.json` **Initialize proactive mode (first-time only)**: ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') if not os.path.exists(state): os.makedirs(os.path.dirname(state), exist_ok=True) json.dump({'proactive_mode': 'off'}, open(state, 'w'), indent=2) print('init') else: mode = json.load(open(state)).get('proactive_mode', 'off') print('exists:' + mode) " ``` - Output `init` → first time setup: tell the user: `💡 Proactive learning: off. Run /clawexpert proactive suggest to enable gap tracking and learning suggestions.` - Output `exists:{mode}` → already configured, skip silently After completing the above, output to the user: `✅ [5%] Initialized. slug={slug}` ⚠️ **Do NOT stop here. Immediately proceed to Step 2.** ### Step 2: Task Decomposition Immediately output to the user: `🧩 [5%] Decomposing topic into subtopics...` **Autonomously decide the number of subtopics** based on topic complexity (min 4, no hard upper limit): - Broad topic (spanning multiple domains/dimensions) → decompose into more subtopics - Focused topic (single technology/event/product) → closer to the minimum of 4 For each subtopic, determine: - `id`: short English identifier (e.g. `grpo-vs-ppo`) - `label`: human-readable title (e.g. "GRPO vs PPO Comparative Analysis") - `keywords`: 3–5 search keywords (mix of languages as appropriate); **always include at least one book-oriented keyword** (e.g. the title of a canonical book on this subtopic, or `{topic} textbook`) - `search_angles`: 2–3 search angle descriptions; **always include one angle targeting book sources** (e.g. "landmark books and textbooks: {BookTitle} chapters, author name + pdf") Decomposition principles: - Subtopics must not have significant content overlap; together they should cover the core aspects of the topic - Each subtopic should be focused enough to be deeply covered within 5–8 search rounds - Prioritize coverage of: core concepts, major methods/frameworks, representative work, comparative analysis, application scenarios - **Temporal span rule**: If the topic name or description contains words indicating a time span across multiple phases (e.g., rise and fall / 兴衰 / history of / 演变 / before and after / establishment and collapse / 建立与瓦解), each distinct historical phase or stage **must** be decomposed into a separate subtopic — do not merge different phases into one subtopic **Align subtopic count with LIMIT**: Read the `maxChildrenPerAgent` value from `openclaw.json` (this is `LIMIT`, default 5). When deciding how many subtopics to create, prefer a count that is an exact multiple of `LIMIT` — this ensures every batch runs at full capacity in Step 3. If the topic's complexity does not support an exact multiple, the last batch may be smaller; note this in the Step 3 output. Update the subtopic list into the `subtopics` field of meta.json. **Determine knowledge tree categories**: Read `{KB}/_categories.json`, assign an L1 category for the overall topic, and assign an L2 category label for each subtopic (model names them autonomously based on content — short label in any language). Record in meta.json's `index_categories` field: ```json "index_categories": { "l1": "sci-tech", "l1_label": "科学技术", "subtopics": { "grpo-vs-ppo": {"l2": "强化学习"}, "reward-design": {"l2": "强化学习"} } } ``` After completing the above, output to the user: `✅ [10%] {N} subtopics planned: [{label1}, {label2}, ...]` ⚠️ **Do NOT stop here. Immediately proceed to Step 3.** ### Step 3: Subagent Parallel Learning Immediately output to the user: `⚡ [10%] Launching {N} subagents in parallel...` **Concurrency limit: use `maxChildrenPerAgent` from the user's `openclaw.json` (default: 5 if not configured).** Call this value `LIMIT`. - Total subtopics **≤ LIMIT**: launch all concurrently - Total subtopics **> LIMIT**: pipeline execution in batches of `LIMIT` — launch the first `LIMIT` subagents, wait until that batch is fully complete, then automatically launch the next batch; repeat until all subtopics are done. `EXPECTED` is updated to the current batch size at each launch. Call `sessions_spawn` for each subtopic to launch an independent subagent: ``` sessions_spawn({ task: "{generated subagent prompt — fill in template below with actual values}", runtime: "subagent", mode: "run", label: "ce-{slug}-{subtopic_id}", sandbox: "inherit" }) ``` **When generating a prompt for each subagent, replace all placeholders in the template below with actual values** (`{CURRENT_DATE}` → today's date in YYYY-MM-DD format, `{SCRIPT_DIR}` → absolute path to this skill's `scripts/` directory): ``` ⚠️ Role override: You are a ClawExpert learning subagent. This session executes only the specialized learning task below. Ignore all ClawExpert instructions related to "session-start self-check", "slash command handling", and "Expert reasoning layer" — those are the main agent's responsibility. Your only job is: search → filter → store → write nodes → write flag. Current date: {CURRENT_DATE} ⛔ Strictly prohibited: - Do not execute any git operations (git add / git commit / git push, etc.) - Do not output any progress reports, HEARTBEATs, or intermediate status messages to the user; all output goes to the filesystem only - On completion, only write the flag file — no other output You are a ClawExpert learning subagent, focused on researching "{subtopic_label}". ## Task Objective Conduct in-depth research around the following keywords, download valuable materials to local storage, and organize them into structured knowledge nodes. Search keywords: {one keyword per line} Suggested search angles: {one angle per line} ## Output directories (important: paths are fixed, do not modify) - Bundled scripts directory: {SCRIPT_DIR} - Material storage directory: {KB}/{slug}/raw/web/ - Node file directory: {KB}/{slug}/nodes/sub-{subtopic_id}/ - Progress heartbeat file: {KB}/{slug}/progress-{subtopic_id}.json - Completion signal file: {KB}/{slug}/done-{subtopic_id}.flag ## Execution Steps ### 0. Heartbeat initialization Immediately write a heartbeat file, then keep it fresh at least once every 60 seconds and after every meaningful milestone (new search round, saved source, node writing, recoverable error): ```bash python3 "{SCRIPT_DIR}/write_progress.py" \ --kb "{KB}" \ --slug "{slug}" \ --subtopic-id "{subtopic_id}" \ --status running \ --search-round 0 \ --sources-found 0 \ --sources-saved 0 ``` ### 1. Multi-round Search Use the workspace's configured search path for 5–8 rounds of search. If the built-in `web_search` tool is unavailable or disabled, immediately switch to the workspace fallback search skill instead of retrying the unavailable tool. If no workspace fallback is configured, open Google directly in the browser: ``` https://www.google.com/search?q={encoded_query} ``` Read the results page to identify high-value URLs, then collect them for Step 2. Vary each round's angle: - Use different keyword combinations - Alternate between languages - Vary search qualifiers (e.g. add "paper", "tutorial", "2025", "comparison", etc.) - Approach from different perspectives (principles / applications / comparisons / latest developments / critiques / limitations) ### 2. Content Retrieval For valuable pages found in search results, use web_fetch to retrieve the full text. ### 3. Quality Filtering Each piece of material must simultaneously satisfy all of the following conditions, or be discarded: 1. Body text length > 300 words (excludes pure navigation pages and ad pages) 2. Contains specific data, conclusions, methodology, or academic citations (excludes vague descriptions) 3. Source domain is trustworthy (excludes content farms and pure SEO sites) 4. Has substantive content overlap with the subtopic keywords (not just a title match) ### 4. Material Storage Write materials that pass the filter to local files. #### ⚠️ File naming — MANDATORY Every stored file (web or PDF) **MUST** be named using the **first 6 hex characters of the source URL's SHA-256 hash**. Descriptive names are strictly forbidden. ```bash hash=$(echo -n "{SOURCE_URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6) # Result example: a3f9c1 # Web file: a3f9c1.md — NEVER use a descriptive name like "attention-paper.md" # PDF binary: a3f9c1.pdf — NEVER use a descriptive name like "attention-is-all-you-need.pdf" # PDF extracted: a3f9c1.md (or a3f9c1/_index.md for large files) ``` Existence check before writing (skip if already exists): ```bash ls "{KB}/{slug}/raw/web/${hash}.md" 2>/dev/null # web ls "{KB}/{slug}/raw/pdf/${hash}" 2>/dev/null || \ ls "{KB}/{slug}/raw/pdf/${hash}.md" 2>/dev/null # pdf ``` #### Routing decision Before storing, determine the source type: 1. **Direct PDF URL** — URL ends in `.pdf`, or `curl -I` shows `Content-Type: application/pdf` → PDF workflow using this URL directly 2. **Academic paper abstract page** — construct the PDF URL, then use PDF workflow: - `arxiv.org/abs/{id}` → `https://arxiv.org/pdf/{id}.pdf` - `aclanthology.org/{id}` → `https://aclanthology.org/{id}.pdf` - `openreview.net/forum?id={id}` → scan fetched HTML for a PDF download link - Other conference/journal pages (NeurIPS, ICML, ICLR, AAAI, CVPR, ECCV, ACL, EMNLP, etc.) → scan fetched HTML for a `.pdf` link - Use the **PDF URL** as the hash source and `source_url` in frontmatter 3. **Everything else** → web page workflow, store under `raw/web/` --- **Web page** (routing case 3) → fetch with `web_fetch`, store under `raw/web/`. **PDF** (routing case 1 or 2) → follow the complete PDF workflow below. --- #### PDF Processing Workflow ##### Step A — Compute hash and check existence ```bash hash=$(echo -n "{PDF_URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6) ls "{KB}/{slug}/raw/pdf/${hash}" 2>/dev/null || ls "{KB}/{slug}/raw/pdf/${hash}.md" 2>/dev/null ``` If either exists, skip — do not re-download. ##### Step B — Download to local ```bash local_pdf="{KB}/{slug}/raw/pdf/${hash}.pdf" # Primary: curl (follows redirects, handles most PDF URLs) curl -L --max-filesize 104857600 -o "${local_pdf}" "{PDF_URL}" 2>/dev/null ``` If curl exits non-zero (download failed or URL is a landing page, not a direct PDF): 1. Use `web_fetch` on the URL to retrieve the HTML page 2. Extract the direct PDF download link from the page content 3. Retry `curl -L` on the extracted direct link If still no local file after the above, skip this source. ##### Step C — Check size and page count ```bash python3 -c " import re, os, sys path = sys.argv[1] size_mb = os.path.getsize(path) / (1024 * 1024) # Pure-Python page count: read PDF structure bytes, no external deps needed page_count = 0 try: with open(path, 'rb') as f: data = f.read(min(os.path.getsize(path), 500_000)) matches = re.findall(rb'/Count\s+(\d+)', data) if matches: page_count = max(int(m) for m in matches) except Exception: pass if page_count == 0: page_count = max(1, int(size_mb * 20)) # fallback estimate: ~50KB/page print(f'{size_mb:.1f}MB {page_count}pages') " "${local_pdf}" ``` ##### Step D — Route by size | Condition | Path | |-----------|------| | ≤ 20 MB **and** ≤ 20 pages | **Small** — full pdf tool call | | ≤ 20 MB **and** 21–100 pages | **Medium** — sectioned pdf tool call | | > 50 MB **or** > 100 pages | **Large** — Python full-text extraction | --- ##### Small PDF path (≤ 20 MB, ≤ 20 pages) Call the `pdf` tool on the **local path**. **Do NOT pass a `model` parameter** — always let the system default handle routing. Specifying an explicit model almost always fails due to missing keys or quota limits. Use this extraction prompt (pass as `prompt`; do not pass `model`): > Extract the full content of this document in Markdown format. For all text sections: reproduce content faithfully. For all figures, charts, tables, and diagrams: describe their key data points, axis labels, legends, and main conclusions in plain text. If the call fails → retry once with the same bare call. If it fails again → fall through to **Python fallback** below. Add `extraction_method: pdf_tool` to frontmatter. If no figures/charts reported: `has_images: false`; otherwise `has_images: true`. For scanned PDFs (no text layer): also add `scanned: true`. --- ##### Medium PDF path (21–100 pages) Build a combined page-range string covering the key sections. Given total page count N: ``` opening = "1-{min(8, N)}" middle = "{max(9, N//2 - 4)}-{min(N-8, N//2 + 5)}" closing = "{max(N-7, 9)}-{N}" pages_arg = "{opening},{middle},{closing}" # e.g. "1-8,21-30,43-50" ``` Call the `pdf` tool once on the **local path** with `pages: "{pages_arg}"` and the same extraction prompt above. **Do NOT pass `model`.** Prefix the extracted content with: ``` [Partial extraction: pages {pages_arg} of {N} total. Key sections only — use Python fallback for full text.] ``` If the call fails → retry once → fall through to Python fallback. Add `extraction_method: pdf_tool_partial` and `page_count: {N}` to frontmatter. --- ##### Large PDF / Python fallback (> 50 MB or > 100 pages, or pdf tool failed) ```bash python3 -c " import sys, subprocess path = sys.argv[1] text = '' try: import pdfplumber with pdfplumber.open(path) as pdf: text = '\n\n'.join(page.extract_text() or '' for page in pdf.pages) except ImportError: # pdftotext (poppler-utils) — commonly available on macOS/Linux result = subprocess.run(['pdftotext', path, '-'], capture_output=True, text=True) if result.returncode == 0: text = result.stdout else: print('ERROR: no PDF extraction tool available', file=sys.stderr) sys.exit(1) print(text) " "${local_pdf}" ``` After extraction, write the result to the correct path — **do not create any other directory (e.g. `processed/`)**: ```bash out_md="{KB}/{slug}/raw/pdf/${hash}.md" ``` Construct the file with frontmatter + extracted body, then write to `${out_md}`: ``` --- source_url: {PDF_URL} title: {title} fetched_at: {ISO 8601} topic: {slug} subtopic: {SUBTOPIC_ID} word_count: {estimated} extraction_method: text_only has_images: false --- {extracted text} ``` Add `extraction_method: text_only` to frontmatter. Image/figure content will be absent. After writing, check the extracted word count: - **< 500 words**: the PDF is image-dominant (e.g. papers with many full-page figures). Add `sparse_text: true` to frontmatter. Do NOT split, do NOT supplement with AI-generated content — write the extracted text as-is in a single `${hash}.md`. Note the limitation in the node's Source Evidence entry. - **500–5000 words**: write as a single `${hash}.md`, no splitting needed. - **> 5000 words**: apply the **large-file sub-tree** splitting below (parts + `_index.md`). --- #### File naming (web and PDF) First 6 hex characters of the URL's SHA-256 hash: ```bash echo -n "{URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6 ``` Existence check before writing: ```bash ls "{KB}/{slug}/raw/web/{hash}.md" 2>/dev/null # web ls "{KB}/{slug}/raw/pdf/{hash}" 2>/dev/null # pdf (may be dir or .md) ``` #### File format ``` --- source_url: {full URL} title: {page title or filename} fetched_at: {ISO 8601 time} topic: {slug} subtopic: {subtopic_id} word_count: {estimated body word count} extraction_method: {web_fetch | pdf_tool | pdf_tool_partial | text_only} # omit for web pages has_images: {true | false} # PDF only page_count: {N} # PDF medium/large only scanned: true # PDF only, if no text layer detected --- {cleaned body content} ``` ⚠️ **Fidelity rule — applies to ALL raw/ files without exception**: The `raw/` directory stores **faithful reproductions of source material only**. Never write AI-generated summaries, paraphrases, curated quote collections, or analytical notes into raw files. If the extracted text is sparse or low quality, write it as-is and mark it accordingly — do not compensate by synthesizing content from your own knowledge. Cleaning rules: - Web page: remove navigation bars, sidebars, footers, ads, cookie banners; preserve headings, paragraphs, lists, tables, code blocks; preserve link text (URLs may be removed) - PDF (`pdf_tool` / `pdf_tool_partial`): write model output as-is — do not strip further; image content already described inline as text - PDF (`text_only`): apply same cleaning rules as web pages; image/figure content absent; write the raw extracted text verbatim - Scanned PDF: `pdf_tool` attempts vision-based description; content quality may be limited **Large file sub-tree** (apply after writing, for both web and PDF): After writing, estimate word count. If word count > 5000: 1. Create a directory to replace the single file: ```bash mkdir -p "{KB}/{slug}/raw/{type}/{hash}" ``` 2. Analyze document structure — identify chapters, sections, or major headings. - **If the document has clear logical structure** (numbered sections, chapter headings, titled subsections) → **split by logical boundaries first** — each major section becomes one part (merge very short sections < 500 words with the next) - **If no clear structure** → split into chunks of ~5000 words each, cutting at paragraph boundaries (never mid-sentence or mid-paragraph) - Either way: keep each part between 2000–8000 words; if a logical section exceeds 8000 words, split it further at the next heading level 3. Write `{KB}/{slug}/raw/{type}/{hash}/_index.md`: ``` --- source_url: {full URL or local path} title: {title} fetched_at: {ISO 8601 time} topic: {slug} word_count: {total} parts: {number of part files} extraction_method: {web_fetch | pdf_tool | text_only} --- ## Document Overview {2–4 sentences summarizing the full document's scope, main argument, and key findings} ## Part Index - [part-001](part-001.md) — {section title} | {1 sentence describing the specific content: what concepts, data, or conclusions this part contains} - [part-002](part-002.md) — {section title} | {1 sentence describing the specific content} ... ``` ⚠️ The Part Index summary MUST be a genuine semantic description written after reading the part — NOT the first few words of the text, NOT "chunk N". A QA subagent uses these summaries to decide which part to read without reading all parts. 4. Write each part as `part-001.md`, `part-002.md`, etc. Each part should begin and end at a natural boundary (heading or paragraph break); include ~200-word overlap with the previous part at the start of each part (except part-001) to avoid conclusions being cut mid-sentence. 5. Delete the original single-file `{hash}.md` (only the directory `{hash}/` remains). When referencing a large-file source in node.md, point to `{hash}/_index.md`; reference specific parts as `{hash}/part-00N.md` when citing a specific section. ### 5. Conclusion Extraction and Node Writing Create the node directory: ```bash mkdir -p "{KB}/{slug}/nodes/sub-{subtopic_id}" ``` Write the node file `{KB}/{slug}/nodes/sub-{subtopic_id}/node.md`: ``` --- id: {subtopic_id}-{current timestamp in ms}-001 parent: root label: {subtopic_label} type: topic depth: 2 confidence: {>=3 sources: high | 1-2 sources: medium | 0 sources: low} last_updated: {YYYY-MM-DD} source_count: {number of valid sources} --- ## Summary {2–5 sentences summarizing the core findings for this subtopic} ## Key Conclusions - {Conclusion 1} ([source title](../../raw/web/{hash}.md)) - {Conclusion 2} ([source title](../../raw/pdf/{hash}/_index.md)) ## Source Evidence - [{web source title}](../../raw/web/{hash}.md) — {one sentence explaining the core value of this source} - [{pdf single-file title}](../../raw/pdf/{hash}.md) — {one sentence} - [{pdf multi-part title}](../../raw/pdf/{hash}/_index.md) — {one sentence}; key data in [part-002](../../raw/pdf/{hash}/part-002.md) ## Analytical Framework (if applicable) {If this subtopic involves a reusable analytical framework or methodology, describe it here} ## Further Directions - [ ] {A related direction or open question discovered during research that falls outside this subtopic's scope — worth exploring in a future --deepen run} - [ ] {Another such direction, if any} ``` If no out-of-scope directions were found, omit the `## Further Directions` section entirely. If the content volume is large, multiple node files may be created: `node.md`, `node-detail-001.md`, ... ### 6. Time Control Before each search round, read meta.json to check for timeout: ```bash cat "{KB}/{slug}/meta.json" ``` Read the `learning_sessions[0].started` field and calculate elapsed time. If more than {max_hours} hours have elapsed, stop searching and proceed to Step 7. ### 7. Completion Signal Write the final flag using the bundled script and remove the progress heartbeat: ```bash python3 "{SCRIPT_DIR}/write_flag.py" \ --kb "{KB}" \ --slug "{slug}" \ --subtopic-id "{subtopic_id}" \ --status done \ --sources-added ACTUAL_COUNT \ --node-file "{KB}/{slug}/nodes/sub-{subtopic_id}/node.md" \ --cleanup-progress ``` Replace `ACTUAL_COUNT` with the actual number of valid materials downloaded (integer); all other placeholders are replaced with real values when the subagent prompt is generated. If the subtopic exhausts high-quality sources or encounters a hard failure, still write a terminal flag with status `partial`, `failed`, `timeout`, or `no_results` and include `--cleanup-progress` so the main agent can continue. ## Notes - Verify each URL is reachable before calling web_fetch - Apply the four-tier source priority system when selecting and filtering sources: - T1 (highest): peer-reviewed papers, official government/standards documents, primary legal texts, clinical systematic reviews - T2: institutional technical docs, major org research reports, university press, regulatory filings - T3: reputable specialist media (IEEE Spectrum, FT, Bloomberg, MIT Tech Review, Reuters, Nature News) - T4: quality general media — background reference only; cross-verify with T1/T2 before citing - Discard: content farms, SEO sites, unattributed content, sites with clear commercial bias - Within the same tier: institutional primary source > third-party reporting; raw data > conclusions only; recent (within 2 years) > outdated - When media reports a study, find the original paper — the media report drops to T3/T4 - Preprints (arXiv pre-review): treat as high-credibility but flag as "not yet peer-reviewed" in node conclusions - If a keyword yields no valuable results after 2 search rounds, change the angle and continue — do not get stuck on fruitless searches - Conclusions must be specific, including numbers, comparisons, and method names — not vague descriptions ``` After all subagents are launched in parallel, record the expected number of flag files (= number of subagents launched in the current batch). A subtopic counts as finished for monitoring purposes once its `done-{subtopic}.flag` exists, regardless of whether the status inside the flag is `done`, `partial`, `failed`, `timeout`, or `no_results`. ⚠️ **Do NOT stop here. Do NOT ask the user anything. Do NOT wait for user input. Immediately proceed to Step 4 in the same turn.** ### Step 4: Progress Monitoring Immediately output to the user: `⏳ [10%] Monitoring subagent progress...` Once all subagents are launched, immediately enter the monitoring loop. **This loop must keep running until the exit condition is met — do not wait for the user to send a message.** ``` EXPECTED = {number of subagents launched in the current batch} Loop: 1. Wait 30 seconds bash: sleep 30 2. Inspect flags + heartbeat files bash: python3 "{SCRIPT_DIR}/monitor_subtopics.py" --kb "{KB}" --slug "{slug}" --stall-seconds 600 → parse JSON result: - DONE = result.done - STALLED = result.stalled - ACTIVE = result.active 3. Output current progress to the user (required every iteration): "⏳ [X%] Subtopics completed: {DONE}/{EXPECTED}" where X = 10 + round(DONE / EXPECTED × 50) 4. Stall handling: - If any subtopic appears in `STALLED`, treat it as hung - If the corresponding subagent is still alive, kill it - Immediately write a terminal flag with `write_flag.py` using status `timeout` or `failed` and `--cleanup-progress` - Then continue the monitoring loop; do not wait forever for a missing heartbeat to recover itself 5. Evaluate: - DONE >= EXPECTED → ⚠️ **IMMEDIATELY execute Step 5 — no pause, no output, no user confirmation. Do not return control to the user. Treat this as a hard goto.** - Current time - meta.json started > 2 × max_hours → timeout, same as above: immediately execute Step 5 - Otherwise → continue loop ``` ⚠️ **Critical**: The moment the exit condition is met, output one line then immediately execute Step 5 without pausing: `🔀 [60%] All subtopics complete. Starting merge...` On timeout, record which subtopics did not complete (missing flag files) and explain in the report during the merge step. ### Step 5: Knowledge Organization (Merge) Before starting, output: `📋 [60%] Starting knowledge merge for: {topic}` **5.1 Collect output** Immediately output to the user: `📂 [62%] Collecting subtopic outputs...` Collect merge inputs with the bundled script: ```bash python3 "{SCRIPT_DIR}/prepare_merge.py" --kb "{KB}" --slug "{slug}" ``` Read all `{KB}/{slug}/nodes/sub-*/node*.md` files and all flag files using the script output as the canonical manifest. After completing the above, output to the user: `✅ [65%] Collected {N} node files from {M} subtopics.` **5.2 Deduplication and consolidation** Immediately output to the user: `🔍 [65%] Checking for duplicates and contradictions...` - Check whether any conclusions are duplicated or contradictory across subtopic nodes - Merge similar conclusions, preserving all source citations - Run node relationship analysis before writing formal nodes: ```bash python3 "{SCRIPT_DIR}/suggest_node_merges.py" --kb "{KB}" --slug "{slug}" ``` - Treat the script output as a **candidate list**, not a final verdict. The script only recalls likely `merge_candidate`, `parent_child_candidate`, and `related_candidate` pairs. The main agent must make the final semantic judgment. - If two or more candidate nodes are semantically close (same core concept expressed with different wording, overlapping summaries, overlapping conclusions, or near-duplicate scope), prefer merging them into **one formal node** instead of writing parallel fragmented nodes - When merging similar nodes: - Choose the clearest label as the canonical node label - Keep all non-duplicative conclusions under that single node - Preserve all source citations from every merged candidate - Record alternate phrasings in frontmatter as `aliases: a | b | c` - Record one stable retrieval phrase in `canonical_intent: ...` - If the candidate pair is better interpreted as hierarchy than synonymy, do **not** merge it. Keep one node as the parent, one as the child, and connect them in `Related Nodes`. - Goal: optimize for future retrieval. User questions rarely match the original subtopic wording exactly, so semantically adjacent material should be consolidated into one retrievable node whenever possible - Mark contradictory conclusions side-by-side with the annotation `⚠️ Disputed` After completing the above, output to the user: `✅ [70%] Deduplication done.` **5.3 Build the knowledge tree** Immediately output to the user: `🌲 [70%] Building knowledge tree structure...` Analyze the hierarchical relationships between nodes: - Which are parallel (sibling nodes) - Which have parent-child relationships - If there are analytical frameworks, identify them as `type: framework` nodes After completing the above, output to the user: `✅ [75%] Tree structure done.` **5.4 Write formal nodes** Immediately output to the user: `✍️ [75%] Writing formal nodes...` Rewrite `nodes/sub-{id}/node*.md` as formal nodes `nodes/node-{seq}.md` (seq starting at 001): ```markdown --- id: node-{seq} original_id: {subagent's original node ID, preserved for traceability} parent: {parent node id; use "root" for top-level nodes} label: {node title} aliases: {optional inline aliases, e.g. a | b | c} canonical_intent: {one stable canonical phrasing for retrieval} scope: {short note describing what this node covers} type: topic depth: {hierarchy depth: top-level child node=2, one level below=3} confidence: high | medium | low last_updated: {YYYY-MM-DD} source_count: {number of associated files} --- ## Summary {Core summary for this node, 2–5 sentences} ## Key Conclusions - {Conclusion 1} - {Conclusion 2} ## Source Evidence - [{source title}](../raw/web/{hash}.md) — {explanation} ## Related Nodes - Parent: [{label}]({relative path}) - Children: [{label}]({relative path}) (if any) - Related: [{label}]({relative path}) (if any) ``` **Framework nodes** (type: framework) format: ```markdown --- id: framework-{seq} type: framework parent: root label: {framework name} depth: 2 confidence: high last_updated: {YYYY-MM-DD} --- ## Analysis Dimensions 1. {Dimension 1}: {explanation} 2. {Dimension 2}: {explanation} ## When to Use When the user asks "{typical question}", apply this framework for structured analysis. ## Application Steps 1. {step} ## Source Basis - [{source}](../raw/web/{hash}.md) ``` After completing the above, output to the user: `✅ [85%] Written {N} formal nodes.` **5.5 Generate index.md** Immediately output to the user: `📄 [85%] Generating index.md...` ```markdown # {Topic Title} > Last updated: {YYYY-MM-DD} | Sources: {N} | Nodes: {N} ## Knowledge Tree Overview - [{node label}](nodes/node-001.md) — {one-sentence summary} - [{child node label}](nodes/node-003.md) — {summary} - [{child node label}](nodes/node-004.md) — {summary} - [{node label}](nodes/node-002.md) — {summary} ## Core Findings Summary 1. {Most important global conclusion 1} 2. {Conclusion 2} 3. {Conclusion 3} ## Further Directions - [ ] {Direction discovered during learning but not yet fully covered 1} - [ ] {Direction 2} ``` After completing the above, output to the user: `✅ [90%] index.md written.` **5.6 Update meta.json** Immediately output to the user: `📝 [90%] Updating meta.json...` Read the full existing meta.json, modify the following fields, then rewrite the entire file: - `status` → `"complete"` - `last_updated` → current ISO 8601 time - `source_count` → count of source documents under `raw/web/` and `raw/pdf/` - `node_count` → count of `nodes/node-*.md` files (`ls {KB}/{slug}/nodes/node-*.md 2>/dev/null | wc -l`) - `learning_sessions[0].finished` → current ISO 8601 time - `learning_sessions[0].sources_added` → number of new sources added this session - `status` for each subtopic in the `subtopics` array → read from the corresponding flag file (done/timeout/no_results) Use the bundled script: ```bash python3 "{SCRIPT_DIR}/update_meta.py" --kb "{KB}" --slug "{slug}" ``` After completing the above, output to the user: `✅ [93%] meta.json updated.` **5.7 Clean up temporary files** Immediately output to the user: `🧹 [93%] Cleaning up temporary files...` ```bash python3 "{SCRIPT_DIR}/cleanup_topic.py" --kb "{KB}" --slug "{slug}" ``` This cleanup step must also remove every `progress-{subtopic}.json` heartbeat file once the task has a terminal flag. After completing the above, output to the user: `✅ [95%] Cleanup done.` **5.8 Rebuild knowledge tree index** Rebuild the `{KB}/_index/` tree from all learned topics using the bundled script: ```bash python3 "{SCRIPT_DIR}/rebuild_index.py" --kb "{KB}" ``` **5.8-A: Write / update `{KB}/_index/{l1-id}/{l2-label}/_index.md`** If the file already exists, read it first, then append this topic's section. Do not overwrite existing topic sections. Each topic learned in this L2 category gets its own subsection with the following format: ```markdown # {l2-label} > {N} topics | {M} total sources | Last updated: {YYYY-MM-DD} --- ## {topic name} (`{slug}`) > Learned: {YYYY-MM-DD} | Sources: {N} | Nodes: {N} ### Core Summary {3–5 sentences synthesizing the most important findings from this learning session. Must include: what the topic is, the key mechanism or argument, and the most important empirical finding or conclusion. Do not use vague phrases like "this topic covers X" — state the actual content.} ### Key Conclusions {5–8 bullet points. Each must be a specific, concrete conclusion — include numbers, names, dates, comparisons where available. Vague conclusions like "X is important" are not acceptable.} - {Specific conclusion with supporting detail} (→ [{source title}]({relative path to raw file})) - {Specific conclusion with supporting detail} (→ [{source title}]({relative path to raw file})) ### Knowledge Nodes {List each node with its label and a one-sentence description of what it covers} - [{node label}]({relative path to node file}) — {what this node covers} ### Source Abstracts {One entry per source file downloaded during this learning session. Each entry must be specific enough that the model can decide whether to read the full file without opening it.} - [{title}]({relative path to raw file}) — {source type and institution} | {date if known} | {2–3 sentences: what this document specifically covers, what key data or claims it contains, what questions it can answer} ### Open Questions {Questions that emerged during learning but were not fully resolved — copy from index.md "Further Directions"} - {question or direction} ``` **5.8-B: Write / update `{KB}/_index/_root_index.md`** If the file already exists, read it in full. Update only the entries that belong to this learning session — do not truncate or overwrite other categories' content. The root index must give enough information for the model to decide, without reading deeper, whether the knowledge base can answer a given question. ```markdown # ClawExpert Knowledge Base > Last updated: {YYYY-MM-DD} | Topics: {N} | Total sources: {N} ## Category Directory ### {L1 label} (`{l1-id}`) #### {l2-label} → [`_index.md`]({l1-id}/{l2-label}/_index.md) | {N} topics | {M} sources **Topics covered:** - **{topic name}** (`{slug}`): {2–3 sentence summary of what was learned — specific findings, not just topic description}. Key conclusions: {3–4 bullet points of the most important concrete findings} - {specific conclusion} - {specific conclusion} - {specific conclusion} ## Recent Learning - {YYYY-MM-DD} **{topic name}** (`{slug}`) → [{l1-label} / {l2-label}]({l1-id}/{l2-label}/_index.md) ``` **Update rules:** - If an L2 section already exists in `_root_index.md`, append the new topic under it — do not replace the section - If an L1 section already exists but this L2 does not, add the new L2 subsection under the existing L1 - If neither exists, create the full hierarchy - Always update the header `Last updated` date and `Topics` / `Total sources` counts - Prepend new entries to the "Recent Learning" list (most recent first); keep at most 20 entries **Example — `economics-finance/intl-monetary-history/_index.md`:** ```markdown # International Monetary History > 1 topic | 16 total sources | Last updated: 2026-03-21 --- ## The Bretton Woods System: Rise and Fall (`bretton-woods-system`) > Learned: 2026-03-21 | Sources: 16 | Nodes: 5 ### Core Summary The Bretton Woods system (1944–1973) was a post-war fixed exchange rate regime anchored by the U.S. dollar's convertibility to gold at $35/oz, with all other currencies pegged to the dollar. Its fundamental contradiction — the Triffin Dilemma — was that the U.S. had to run persistent deficits to supply global dollar liquidity, which inevitably eroded confidence in dollar-gold convertibility. The system collapsed in stages: Nixon suspended gold convertibility on August 15, 1971, and by March 1973 major currencies had moved to floating rates. ### Key Conclusions - 44 nations established the system at Bretton Woods, NH in July 1944; the dollar became the sole currency directly convertible to gold (→ [IMF History Overview](bretton-woods-system/raw/web/a1b2c3.md)) - The Triffin Dilemma (identified 1960): structural impossibility of simultaneously providing global liquidity and maintaining gold backing — not fixable by policy (→ [Triffin 1960 Paper](bretton-woods-system/raw/web/d4e5f6.md)) - Vietnam War spending and Great Society programs expanded U.S. deficits through the 1960s; gold reserves fell from ~$20B peak to under $10B by 1971 - France and other countries redeemed dollars for gold 1965–1971, accelerating reserve depletion - August 15, 1971 "Nixon Shock": unilateral suspension of dollar-gold convertibility - Smithsonian Agreement (Dec 1971) briefly adjusted parities but failed to restore confidence; floating rates became universal by March 1973 ### Knowledge Nodes - [node-001.md](bretton-woods-system/nodes/node-001.md) — Wartime origins and 1944 negotiations: White Plan vs. Keynes Plan - [node-002.md](bretton-woods-system/nodes/node-002.md) — Operating mechanism: fixed rates, IMF role, and capital controls - [node-003.md](bretton-woods-system/nodes/node-003.md) — Triffin Dilemma and structural tensions of the 1960s - [node-004.md](bretton-woods-system/nodes/node-004.md) — Nixon Shock and Smithsonian Agreement (1971–1973) - [node-005.md](bretton-woods-system/nodes/node-005.md) — Legacy: the floating rate era and the modern international monetary system ### Source Abstracts - [Nixon Ends Convertibility of U.S. Dollars to Gold](bretton-woods-system/raw/web/a1b2c3.md) — Federal Reserve History, official historical article, authored by Kenneth Weisbrode, published 2013. Primary focus: the August 15, 1971 "Nixon Shock" announcement. Documents the secret Camp David weekend meeting (August 13–15, 1971) where Nixon and 15 advisors — including Treasury Secretary John Connally and Fed Chair Arthur Burns — decided to close the gold window. Specifies that the U.S. gold stock had fallen from $25B (1949 peak) to $10B by 1971, while foreign dollar claims exceeded $80B. Describes the three simultaneous measures: (1) suspension of gold convertibility, (2) 10% import surcharge, (3) 90-day wage/price freeze. Notes that European markets closed the following Monday in response. Does NOT cover: Smithsonian Agreement details, longer-term consequences, or pre-1971 structural tensions. Use for: exact date and mechanics of the Nixon Shock, key personnel involved, specific gold reserve figures at the time of closure. - [Triffin Dilemma, or: the Myth of the Triffin Dilemma](bretton-woods-system/raw/web/d4e5f6.md) — BIS Working Paper No. 684, authored by Garber and Svensson, published 2015. Central argument: the Triffin Dilemma was a real structural constraint but its timing was not deterministic — U.S. policy choices (particularly Vietnam War financing and refusal to deflate) accelerated collapse. Contains year-by-year data on U.S. gold reserves (1950–1973), total foreign dollar liabilities, and the ratio between them. Shows the ratio crossed 1:1 (liabilities exceeding reserves) in 1960, consistent with Triffin's original 1960 Senate testimony. Also reviews the academic debate: Eichengreen argues collapse was inevitable; Garber argues it was policy-contingent. Does NOT cover: Smithsonian Agreement or post-1973 floating rate arrangements. Use for: quantitative gold reserve data, Triffin Dilemma mechanics, academic debate on whether collapse was inevitable vs. policy-driven. ### Open Questions - Could the Bretton Woods system have survived under different U.S. fiscal policies? - Can SDRs realistically serve as a supranational reserve asset to replace the dollar? - Does the current dollar-dominant system face analogous Triffin-style structural pressures? ``` **Example — `_root_index.md` entry for the same topic:** ```markdown ### Economics & Finance (`economics-finance`) #### International Monetary History → [`_index.md`](economics-finance/intl-monetary-history/_index.md) | 1 topic | 16 sources **Topics covered:** - **The Bretton Woods System: Rise and Fall** (`bretton-woods-system`): Post-war fixed exchange rate system (1944–1973) anchored by dollar-gold convertibility at $35/oz. The Triffin Dilemma — the structural impossibility of supplying global liquidity while maintaining gold backing — made collapse inevitable; the Nixon Shock in 1971 triggered the final unraveling. - Dollar pegged to gold at $35/oz; all other currencies pegged to the dollar - Triffin Dilemma is an endogenous structural contradiction, not an external shock - Nixon Shock (Aug 1971) was the proximate trigger; French gold redemptions were an accelerating factor ``` After completing the above, output to the user: `✅ [97%] Knowledge tree index updated.` **5.9 Validate topic consistency** Immediately output to the user: `🧪 [97%] Validating knowledge base consistency...` Run: ```bash python3 "{SCRIPT_DIR}/validate_kb.py" --kb "{KB}" --slug "{slug}" ``` If validation reports issues, fix them before Step 6 or explicitly report any residual inconsistency. After completing the above, output to the user: `✅ [98%] Validation done.` ⚠️ **Do NOT stop here. Immediately proceed to Step 6.** ### Step 6: Write to Memory Immediately output to the user: `💾 [97%] Writing knowledge base pointer to memory...` Write a pointer to OpenClaw memory so that future sessions know the location of the knowledge base: ```bash python3 -c " import os kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) memory_dir = os.path.expanduser('~/.openclaw/workspace/memory') memory_path = os.path.join(memory_dir, 'clawexpert.md') os.makedirs(memory_dir, exist_ok=True) content = '''# ClawExpert Knowledge Base Knowledge base located at: {kb}/_index/_root_index.md Before answering questions, read the above file to understand the full scope of the knowledge base, then decide whether to retrieve specific category content. Do not answer questions from learned domains using parametric memory directly. '''.format(kb=kb) with open(memory_path, 'w') as f: f.write(content) print('memory written:', memory_path) " ``` After completing the above, output to the user: `✅ [99%] Memory pointer written.` ⚠️ **Do NOT stop here. Immediately proceed to Step 7.** ### Step 7: Completion Report Output to the user: ``` ## Learning Complete: {topic name} ### Summary - Duration: approx. {N} minutes - Subtopics: {N} - Sources downloaded: {N} - Knowledge nodes: {N} ### Subtopic Completion Status | Subtopic | Sources | Status | |----------|---------|--------| | {label} | {n} | ✅ Complete | | {label} | {n} | ⏱ Timed out (partial saved) | ### Core Findings 1. {Most important conclusion} 2. {Conclusion 2} 3. {Conclusion 3} ### Suggested Further Directions - {List of further directions} Run `/clawexpert show {slug}` to view the full knowledge tree. ``` > After memory is written, the next time a related topic is asked in a new session, the model will detect this knowledge base via `memory_search` and proactively reference it. --- ## III. Importer ### `/clawexpert import` — Full Workflow Upon receiving `/clawexpert import <filepath> [--topic <slug>]`, execute the following steps. #### Step 1: Validate file ```bash test -f "{filepath}" && echo "exists" || echo "not found" ``` - File not found → abort: `Error: file not found: {filepath}` - Resolve to absolute path: ```bash filepath=$(python3 -c "import os, sys; print(os.path.abspath(sys.argv[1]))" "{filepath}") ``` - Check it is a PDF (extension `.pdf`, or first 4 bytes are `%PDF`) — if not, abort with: `Error: only PDF files are supported.` #### Step 2: Resolve target topic - `--topic <slug>` provided: - Check `{KB}/{slug}/meta.json` exists - Exists → use that topic - Not exists → ask: `Topic '{slug}' not found. Create a new topic with this slug? [y/n]` - `y` → initialize new topic (create directories + write initial meta.json with `status = "complete"`) - `n` → abort - `--topic` not provided: - List existing topics: ```bash ls "$KNOWLEDGE_DIR" 2>/dev/null ``` - No topics found → ask: `No topics found. Enter a topic name to create one:`; then create it - Topics exist → prompt: `Which topic should this file be imported into? Enter slug (or type a new name to create a new topic):` #### Step 3: Compute hash For local files, hash is derived from **file content** (not path), so the same file imported twice produces the same hash regardless of where it lives: ```bash hash=$(openssl dgst -sha256 "{filepath}" | awk '{print $NF}' | cut -c1-6) ``` #### Step 4: Check for duplicate ```bash ls "{KB}/{slug}/raw/pdf/${hash}" 2>/dev/null || ls "{KB}/{slug}/raw/pdf/${hash}.md" 2>/dev/null ``` If either exists → inform user: `This file has already been imported (hash: {hash}). Skipping.` and abort. #### Step 5: Copy PDF to KB ```bash local_pdf="{KB}/{slug}/raw/pdf/${hash}.pdf" cp "{filepath}" "${local_pdf}" ``` #### Step 6–7: Extract content Apply the same **PDF processing workflow** used by subagents (size check → Small / Medium / Large path). Key difference: use `source_url: local:{filepath}` in all frontmatter instead of an HTTP URL. After extraction, write to: - `{KB}/{slug}/raw/pdf/${hash}.md` — single file (< 5000 words) - `{KB}/{slug}/raw/pdf/${hash}/` — sub-tree with `_index.md` + `part-*.md` (> 5000 words) ⚠️ **Fidelity rule applies**: write extracted text faithfully; do not supplement with AI-generated content. #### Step 8: Write node ```bash mkdir -p "{KB}/{slug}/nodes/import-{hash}" ``` Write `{KB}/{slug}/nodes/import-{hash}/node.md`: ``` --- id: import-{hash}-001 parent: root label: {title extracted from PDF, or filename without extension} type: topic depth: 2 confidence: high last_updated: {YYYY-MM-DD} source_count: 1 local_import: true --- ## Summary {2–5 sentences summarizing the document's core content} ## Key Conclusions - {Conclusion 1} ([{title}](../../raw/pdf/{hash}/_index.md)) ## Source Evidence - [{title}](../../raw/pdf/{hash}/_index.md) — local import: {one sentence describing the document} ## Analytical Framework (if applicable) {If this document introduces a reusable framework or methodology, describe it here} ## Further Directions - [ ] {Out-of-scope directions found while reading, if any} ``` If the extracted file is a single `{hash}.md` (not split), reference it as `../../raw/pdf/{hash}.md` instead. Omit `## Further Directions` if none were found. #### Step 9: Update meta.json Read `{KB}/{slug}/meta.json`, then: 1. Increment `source_count` and `node_count` 2. Add or append to `local_sources` array: ```json { "path": "{filepath}", "hash": "{hash}", "title": "{title}", "imported_at": "{ISO 8601}" } ``` Write updated meta.json back. #### Step 10: Update index.md If `{KB}/{slug}/index.md` exists, append a new node entry under the node list. If it does not exist, create a minimal one: ```markdown # {topic} — Knowledge Index ## Nodes - [import-{hash}](nodes/import-{hash}/node.md) — {title} ``` #### Step 11: Output success ``` ✅ Import complete File: {filepath} Topic: {slug} Hash: {hash} Node: {KB}/{slug}/nodes/import-{hash}/node.md Raw: {KB}/{slug}/raw/pdf/{hash}[.md or /] ``` --- ## IV. Organizer ### Deepening Learning (--deepen) When executing `/clawexpert learn --deepen <slug>`: 1. Run gap analysis to generate both **breadth** and **depth** work items: ```bash KB="${CLAWEXPERT_KNOWLEDGE_DIR:-$HOME/.openclaw/workspace/knowledge}" SCRIPT_DIR="${CLAWEXPERT_SCRIPT_DIR:-$HOME/.openclaw/workspace/skills/clawexpert/scripts}" python3 "$SCRIPT_DIR/analyze_gaps.py" \ --kb "$KB" \ --slug "{slug}" \ --output "$KB/{slug}/research_gaps.json" ``` The output file contains: - `breadth_tasks`: unchecked `Further Directions` from topic / node files, used to create **new subtopics** - `depth_tasks`: targeted questions for existing nodes, used to deepen existing coverage 2. Read `research_gaps.json` and build a mixed deepen plan: - **Breadth track**: create new subtopics from `breadth_tasks` - **Depth track**: create one deep-dive investigation per selected node from `depth_tasks` - If a depth task clearly overlaps with a breadth task, keep only the depth task 3. Record the deepen session start: ```bash python3 "$SCRIPT_DIR/update_superlearn_meta.py" \ --kb "$KB" \ --slug "{slug}" \ --mode deepen \ --phase start \ --session-id "{session_id}" \ --round 1 \ --breadth-tasks {breadth_count} \ --depth-tasks {depth_count} ``` 4. Launch two agent pools in parallel: - **Breadth agents**: use the normal learning subagent workflow (same as Step 3 in Learner) - **Depth agents**: use the template in `references/deepdive-subagent-prompt.md` Depth-agent rules: - Each depth agent targets exactly one existing node - It must read the current node first, then search only for evidence that answers the listed questions or verifies the listed claims - It writes a supplement file to `{KB}/{slug}/superlearn/{session_id}/depth/{node-id}.md` - It writes `done-depth-{node-id}.flag` via `write_flag.py --cleanup-progress` 5. Merge breadth and depth results separately: - **Breadth merge**: - New node seq starts from the highest existing seq + 1 - New nodes are attached under the most relevant existing parent node - **Depth merge**: - Do not overwrite the original node - Append a new section at the end of the existing node: ```markdown ## Deepening Update ({YYYY-MM-DD}) ### Questions Answered - ... ### New Evidence - ... ### Claim Verification - ... ### Remaining Gaps - [ ] ... ``` - If a depth task discovers a genuinely separate reusable method/framework, it may create a new formal node in addition to the supplement 6. Update topic metadata and indexes: - Append a new learning record to `learning_sessions` - Record deepen session completion: ```bash python3 "$SCRIPT_DIR/update_superlearn_meta.py" \ --kb "$KB" \ --slug "{slug}" \ --mode deepen \ --phase finish \ --session-id "{session_id}" \ --round 1 \ --breadth-tasks {breadth_count} \ --depth-tasks {depth_count} \ --sources-added {sources_added} \ --nodes-added {nodes_added} \ --status complete ``` - Update topic `meta.json`, `index.md`, category `_index.md`, and `_root_index.md` - Run `rebuild_index.py` and `validate_kb.py` before reporting success 7. Update the "Further Directions" in `index.md` and any touched nodes: - Check off completed items - Keep unresolved items unchecked - Add newly discovered open questions 8. **⚠️ Sync the category index (MANDATORY — do not skip)**: - Find the `{KB}/_index/` subdirectory for this topic's L1/L2 category - Append every new node to the Knowledge Nodes list in that `_index.md` - The category index node count must always match `index.md` exactly - Verify: `grep -c "node-" {KB}/_index/.../... ` equals `grep -c "node-" {KB}/{slug}/index.md` ### Multi-round Deep Research (superlearn) When executing `/clawexpert superlearn <topic-or-slug>`: 1. Resolve the input against the knowledge base: ```bash python3 "$SCRIPT_DIR/resolve_topic.py" --kb "$KB" --query "{topic-or-slug}" ``` - If `exists=true`: use the returned `best_match.slug` and start from deepen rounds - If `exists=false`: treat the input as a brand new topic and enter **Phase 0: bootstrap learn** 2. Initialize minimum budget contract: - If user provided only `--rounds`: `MAX_ROUNDS={value}`, `MAX_HOURS=0` - If user provided only `--hours`: `MAX_ROUNDS=0`, `MAX_HOURS={value}` - If user provided both: `MAX_ROUNDS={rounds}`, `MAX_HOURS={hours}` - If user provided neither: `MAX_ROUNDS=2`, `MAX_HOURS=0` - `ROUND=1` - `START_TS=now` - `ELAPSED_HOURS=0` 3. **Phase 0: bootstrap learn (only when topic does not yet exist)** - Run one initial `/clawexpert learn {topic}` using the same total budget - If `MAX_HOURS>0`, the learn phase consumes from that shared budget - After learn finishes: - recompute `ELAPSED_HOURS` - if `ELAPSED_HOURS >= MAX_HOURS` and time budget is active → stop with [REDACTED_SECRET] - otherwise continue into deepen rounds using the newly created slug 4. At the start of every deepen round: - Run `analyze_gaps.py` and write `{KB}/{slug}/superlearn/round-{ROUND}/gaps.json` - Run `plan_superlearn.py` to evaluate whether the **minimum contract** has already been satisfied: ```bash python3 "$SCRIPT_DIR/plan_superlearn.py" \ --kb "$KB" \ --slug "{slug}" \ --gaps-file "$KB/{slug}/superlearn/round-{ROUND}/gaps.json" \ --round "{ROUND}" \ --max-rounds "{MAX_ROUNDS}" \ --max-hours "{MAX_HOURS}" \ --elapsed-hours "{ELAPSED_HOURS}" ``` 5. If `should_continue=false`, stop immediately and report the stop reason. - `should_continue=false` means the requested minimum contract has already been satisfied at the current round boundary - It does **not** mean the agent is free to stop early based on subjective completeness 6. If `should_continue=true`, record round start: ```bash python3 "$SCRIPT_DIR/update_superlearn_meta.py" \ --kb "$KB" \ --slug "{slug}" \ --mode superlearn \ --phase start \ --session-id "{session_id}" \ --round "{ROUND}" \ --max-rounds "{MAX_ROUNDS}" \ --max-hours "{MAX_HOURS}" \ --breadth-tasks {breadth_count} \ --depth-tasks {depth_count} ``` 7. Execute one full deepen round using only the selected breadth/depth tasks from the plan. 8. After the round: - Merge - Rebuild indexes - Validate - Record round completion with `update_superlearn_meta.py --phase finish` - Re-run `analyze_gaps.py` - Re-run `plan_superlearn.py` for `ROUND+1` - Recompute `ELAPSED_HOURS` from `START_TS` 9. **Planner is the only stop authority**: - `--hours` and `--rounds` are **minimum requirements**, not soft caps and not upper bounds - Before the requested minimum is satisfied, stopping is prohibited - The main agent must NOT stop because it personally judges the remaining gaps to be "niche", "incremental", or "not worth it" - The main agent must NOT skip a round based on free-form reasoning - A round may stop only when `plan_superlearn.py` returns `should_continue=false` - The main agent must NOT replace `update_superlearn_meta.py` with ad-hoc inline Python to finalize the run - The final user-facing report must include the planner's `budget_mode` and `stop_reason` 10. Increase `ROUND += 1` and repeat until the planner returns `should_continue=false`. ### Node Confidence Rules | source_count | confidence | |-------------|------------| | >= 3 high-quality sources | high | | 1–2 sources | medium | | 0 sources (inference only) | low | ### Conflict Handling When different sources support opposing views on the same conclusion: ```markdown ## Key Conclusions - {Conclusion A} ([source 1](../raw/web/xxx.md)) - ⚠️ **Disputed**: [source 2](../raw/web/yyy.md) reaches the opposite conclusion, arguing that {opposing view} ``` --- ## V. Expert ### Proactive Mode Check (Required Before Every Answer) ⚠️ **Before answering any user question, always read the proactive mode from file** — do not rely on session memory or environment variables: ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') try: mode = json.load(open(state)).get('proactive_mode', 'off') except Exception: mode = 'off' print(mode) " ``` Hold this value for the current answer's B1/B3 decisions. Valid values: **off / suggest / auto** — treat any other value as `off`. ### Trigger Conditions When a user asks a question (not a slash command), use the self-check result from session start to decide: **Conditions that trigger knowledge base retrieval** (any one is sufficient): - The question's domain is clearly related to a known knowledge base topic name/slug - The user explicitly asks "have you learned about XXX?" or "answer based on your knowledge base" - The question involves specialized knowledge that may be covered in the knowledge base ### Knowledge Base Answer Flow (Three-Layer Progressive Retrieval) **Layer 1: Read _root_index.md (required)** ```bash cat "{KB}/_index/_root_index.md" ``` Browse the entire knowledge base category map, determine which L1/L2 category the question belongs to, and locate the relevant branch. If `_root_index.md` does not exist (v0.1.0 compatibility mode), scan slug directories under `{KB}/` and read the `topic` field of each `meta.json` to make the determination. **Layer 2: Read category _index.md (required)** ```bash cat "{KB}/_index/{l1-id}/{l2-label}/_index.md" ``` Retrieve the source list and key conclusion summaries for this category. Most questions can be answered with high quality at this layer without reading the original sources. **Layer 2.5: Run query-routing candidate retrieval (required unless the match is already obvious)** ```bash python3 "{SCRIPT_DIR}/resolve_query_nodes.py" --kb "{KB}" --query "{user question}" ``` Use the returned `top_topics` and `top_nodes` only as a **routing shortlist**: - The script does **not** decide semantics for you - The main agent must judge whether the query maps to: - an existing node - a broader topic branch - a narrower child of an existing node - or a genuinely new node/topic - Prefer inspecting the top 1-3 candidates rather than requiring an exact wording match - If multiple phrasings appear to describe the same concept, reuse the existing canonical node instead of creating a near-duplicate - If the best candidate is only broadly related, route into that branch first and reason at the tree level before deciding whether a new node should exist **⚠️ `ask_agent_to_create_or_expand` does NOT mean "skip KB retrieval"**. It means the routing script could not find a high-confidence single match. In this case you must still: 1. Read the index files of the top 1–3 `top_topics` returned, even if their scores are low 2. Perform cross-topic enrichment (see below) before answering **Cross-topic enrichment (required when answering multi-dimensional questions)** Token-based routing often misses semantically adjacent topics. After getting the routing shortlist, actively scan for additional topics that could contribute theory, data, or framing: - Read `{KB}/_root_index.md` to see all topic names and their L1/L2 categories - For any topic whose name or category overlaps with the question's domain — even loosely — read its `index.md` - Specifically: if the question involves **investor behavior, psychology, or market anomalies**, always check whether a behavioral finance / 行为金融 topic exists and read its index - If the question involves **Chinese markets or A股**, also check for empirical / 实证 research topics covering the same domain - Compose the final answer by synthesizing across all relevant topics, not just the highest-scoring one **Layer 3: Spawn QA subagent to read original sources (on demand)** Triggered whenever the main agent judges that reading the original source would improve answer quality — not limited to numerical citations. Any time more depth, context, or fidelity to the source is needed, spawn a QA subagent. ⛔ **The main agent must NEVER read raw source files directly.** Reading `raw/web/*.md` or `raw/pdf/**` files is exclusively the QA subagent's job. If the main agent needs content from a source file, it must always go through the spawn → read tmp → delete tmp pipeline below. Direct `cat` or `read` calls on source files by the main agent are prohibited — they bypass the verbatim-extraction guarantee and produce unreliable citations. ``` sessions_spawn({ task: "QA subagent prompt — fill in file_path and query from the template below", runtime: "subagent", mode: "run", label: "ce-qa-{timestamp}", sandbox: "inherit" }) ``` **QA subagent prompt template**: ``` You are a ClawExpert QA subagent. Your sole job is to read the specified source file and extract relevant content as faithfully and completely as possible. ⛔ Strictly prohibited: - Do not execute any git operations - Do not output anything to the user — write to files only - Do not read any files outside the scope specified by this task - Do not paraphrase numbers, percentages, dates, proper nouns, or named conclusions — quote verbatim - Do not synthesize across passages or draw new conclusions — extract only Task: - Source: {file_path} - Query: {what the main agent needs to know — be specific about the topic or question} - Write result to: {KB}/_qa_tmp/{timestamp}.md **If {file_path} is a directory (large file sub-tree)**: 1. Read `{file_path}/_index.md` first to get the Part Index with 1-sentence summaries per part 2. Use the summaries to identify which parts are relevant to the query 3. Read only those relevant parts — do not read all parts 4. If the index summary is insufficient to determine relevance, start with part-001.md Result file format (follow strictly): --- query: {query} source_file: {file_path} completeness: full | partial | not_found --- ## Verbatim Excerpts Extract ALL passages in the file that are relevant to the query. For each passage: ### Excerpt {n} Location: {section title or paragraph description in the source} > {exact verbatim quote — do not shorten, merge, or paraphrase} (Repeat for every relevant passage found. More excerpts is better than fewer.) ## Extraction Summary A faithful synthesis of the excerpts above (3–8 sentences). - Must not introduce any fact, number, or claim not present in the verbatim excerpts above - Must not omit significant findings from the excerpts - Numbers and key terms must match the verbatim excerpts exactly ``` **Main agent reading rules** (apply every time a QA result is read): - Treat "Verbatim Excerpts" as ground truth — cite directly from these, not from the summary - Use "Extraction Summary" only as a navigation aid - If `completeness: partial`, note the gap explicitly in your answer - If `completeness: not_found`, do not fabricate — state that the source contains no relevant content - After composing the answer, immediately delete the tmp file: ```bash rm -f "{KB}/_qa_tmp/{timestamp}.md" ``` **If framework nodes exist**: Read the framework node and conduct structured analysis along its dimensions. ### Answer Format ``` {Main answer content} --- **Citations**: - [{node label}] → {source document title} ({source date}) - [{node label}] → {source document title} **Knowledge Status**: - Established: {aspects with sufficient evidence} - To be deepened: {aspects with insufficient evidence or not yet covered} - Disputed: {aspects with contradictory sources} (only shown when disputes exist) > For more depth, run `/clawexpert learn --deepen {slug}` ``` ### No Matching Knowledge Base ``` No knowledge base found for this question. Consider running `/clawexpert learn {recommended topic name}` to build one systematically. Here is a temporary answer based on live search: {live search results} ``` ### Framework Transfer When facing a new analytical question, first check whether the knowledge base has a relevant framework node: 1. Read index.md and check whether any nodes have type=framework 2. If so, read that framework node 3. Conduct analysis following the framework's "Analysis Dimensions" and "Application Steps" 4. For each dimension, reference specific data and conclusions already in the knowledge base wherever possible 5. Clearly indicate which dimensions have sufficient backing and which require additional searching This lets you not just "remember conclusions" but "learn analytical methods" — enabling you to handle new questions. ### Proactive Topic Tracking (B1) After composing an answer, re-read the current mode directly from `clawexpert-state.json` (do not rely on session memory — the mode may have changed): ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') try: mode = json.load(open(state)).get('proactive_mode', 'off') except Exception: mode = 'off' print(mode) " ``` If mode is `off` → skip entirely. Otherwise, check whether coverage was insufficient and record it: **Coverage is insufficient when any of the following is true:** 1. No matching topic was found ("No Matching Knowledge Base" branch was taken) 2. A topic was found but all matched nodes have `confidence: low` 3. A topic was found but all matched nodes have `source_count` < 2 If coverage is insufficient, extract the topic from the user's question (use a 3–6 word descriptive label) and a one-sentence gap description, then update `{KB}/_proactive_queue.json`: ```bash python3 -c " import json, os, datetime kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) path = kb + '/_proactive_queue.json' today = datetime.date.today().isoformat() topic = '{TOPIC}' # replace: 3-6 word label from the question gap = '{GAP_DESCRIPTION}' # replace: one sentence describing what was missing # Load or create queue data = json.load(open(path)) if os.path.exists(path) else {'version': 1, 'queue': []} q = data.get('queue', []) # Find existing entry for this topic (case-insensitive match) match = next((i for i, e in enumerate(q) if e['topic'].lower() == topic.lower()), None) if match is not None: q[match]['session_count'] += 1 q[match]['mentioned_at'] = today q[match]['gap_description'] = gap else: q.append({'topic': topic, 'gap_description': gap, 'mentioned_at': today, 'session_count': 1, 'source': 'conversation'}) # Sort by session_count desc, then mentioned_at desc; keep top 20 q.sort(key=lambda e: (-e['session_count'], e['mentioned_at']), reverse=False) data['queue'] = q[:20] data['updated'] = datetime.datetime.utcnow().isoformat() + 'Z' data.setdefault('version', 1) json.dump(data, open(path, 'w'), indent=2) print(f'Queue updated: {topic} (session_count={q[0][\"session_count\"] if q else 1})') " ``` Do **not** inform the user about this queue update — it runs silently in the background. ### ⚠️ Proactive Queue Display (B3) — MANDATORY ⚠️ **This step is not optional.** Immediately after B1 (whether or not a gap was written), you MUST run the queue check and output the result. Your answer is incomplete until B3 executes. Do not stop after composing your answer text — continue directly to this step. ```bash python3 -c " import json, os state = os.path.expanduser('~/.openclaw/workspace/clawexpert-state.json') try: mode = json.load(open(state)).get('proactive_mode', 'off') except Exception: mode = 'off' if mode == 'off': print('off') exit() kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) path = kb + '/_proactive_queue.json' if not os.path.exists(path): print('empty') else: q = json.load(open(path)).get('queue', []) q_sorted = sorted(q, key=lambda e: (-e['session_count'], e.get('mentioned_at', ''))) for e in q_sorted[:3]: print(f\"{e['topic']}|{e['gap_description']}|{e['session_count']}\") " ``` If output is `off` or `empty` → skip silently. Otherwise append to the answer: - **suggest** mode: ``` --- 📚 Learning suggestions based on recent gaps: 1. {topic} — {gap_description} (mentioned {session_count}×) 2. ... Run /clawexpert learn <topic> to build knowledge, or /clawexpert proactive skip to dismiss. ``` - **auto** mode → show the top entry and ask the user when to learn: ``` --- 📚 Auto-learning ready: 1. {topic} — {gap_description} (mentioned {session_count}×) When would you like to learn this? Reply: - **now** — start immediately - **later** — remind me next session - **skip** — dismiss this topic ``` Then wait for the user's reply: - `now` → execute `/clawexpert learn {topic}` - `later` → keep the entry in the queue, do nothing this session - `skip` → remove only this entry from the queue (do not clear the whole queue) --- ### ⚠️ End-of-Answer Checklist (Required When Mode ≠ off) After composing and outputting your answer text, you MUST complete ALL of the following before considering the answer done: 1. **B1** — Run topic tracking: if coverage was insufficient, write the gap to the queue (silent, no user output) 2. **B3** — Run queue display: if the queue is non-empty, append the suggestion block to your answer Do not stop after the answer text. Do not wait for the user to prompt you. These steps are mandatory. ### Radar Scan (`/clawexpert radar`) Triggered by the user running `/clawexpert radar`, or via Cron. Scans all L1 categories for recent major events and adds candidates to the proactive queue. #### Step 1: Read L1 categories ```bash python3 -c " import json, os kb = os.environ.get('CLAWEXPERT_KNOWLEDGE_DIR', os.path.expanduser('~/.openclaw/workspace/knowledge')) cats = json.load(open(kb + '/_categories.json')).get('categories', []) for c in cats: print(c['id'], c['label']) " ``` #### Step 2: Search each category For each L1 category, first output: `🔍 Scanning: {label}...` Then run 1–2 searches using the workspace's configured search path. If the built-in `web_search` tool is unavailable or disabled, immediately switch to the workspace fallback search skill. If no fallback is configured, open Google directly in the browser: ``` https://www.google.com/search?q={encoded_query} ``` Queries to run per category: - `"{label}" major developments last 7 days` - `"{label}" latest breakthroughs OR announcements 2026` Vary language based on the category label (e.g., Chinese-label categories → also search in Chinese). ⚠️ **Do NOT stop between categories. Immediately proceed to the next category after each search.** #### Step 3: Evaluate events For each result, assess: 1. **Impact**: affects a large population, major institution, or significant market 2. **Persistence**: likely to remain relevant for weeks, not just 24 hours 3. **KB relevance**: relates to an existing topic in the knowledge base, or fills a clear gap Discard: minor product updates, routine statistical releases, clickbait headlines. #### Step 4: Write candidates to queue For each event that passes evaluation, add an entry to `{KB}/_proactive_queue.json` using the same update logic as B1, with `source: "radar"`: ```python { "topic": "{3-6 word event label}", "gap_description": "{one sentence: what happened and why it matters}", "mentioned_at": "{today}", "session_count": 1, "source": "radar" } ``` If proactive mode is `off`, skip writing to the queue (the scan still runs if the user explicitly calls `/clawexpert radar`, but candidates are not saved). #### Step 5: Output summary ``` 🔭 Radar scan complete Categories scanned: {N} Events added to queue: {M} {If M > 0:} Top candidates: 1. {topic} — {gap_description} 2. ... Run /clawexpert proactive suggest to review, or /clawexpert learn <topic> to start learning. {If M == 0:} No significant new events found across monitored categories. ``` --- ## VI. Edge Cases | Scenario | Handling | |----------|----------| | Subagent timeout | Mark subtopic as timeout, merge completed portions, explain in report | | No search results for a subtopic | Write status:"no_results" in flag, main agent explains in report | | Knowledge base directory missing | Auto-create with `mkdir -p` | | Slug already exists (complete) | Ask user whether to deepen or re-learn | | Concurrent learning of the same topic | Reject when status="learning", prompt user to wait | | meta.json corrupted/missing | Scan nodes/ and raw/ to reconstruct basic information | | forget operation | Must wait for explicit user confirmation before executing rm -rf | --- ## VII. Scheduled Learning (Cron, optional) Users can add the following to openclaw.json: ```json { "cron": { "clawexpert-nightly": { "cron": "0 2 * * *", "message": "/clawexpert learn --auto", "target": "main" } } } ``` `--auto` mode: Scans all topics and automatically runs deepen on any with status="partial" or unchecked items in the "Further Directions" list. Exits silently if nothing to do. To also run the radar scan on a weekly schedule, add a second Cron entry: ```json { "cron": { "clawexpert-nightly": { "cron": "0 2 * * *", "message": "/clawexpert learn --auto", "target": "main" }, "clawexpert-radar": { "cron": "0 7 * * 1", "message": "/clawexpert radar", "target": "main" } } } ``` `clawexpert-radar` runs every Monday at 07:00, scanning all L1 categories for the past week's major events and writing candidates to the proactive queue. Requires proactive mode (`clawexpert-state.json → proactive_mode`) to be `suggest` or `auto` for candidates to be saved.

path: _meta.json
content: {
  "name": "clawexpert",
  "version": "1.0.0",
  "description": "Turn OpenClaw into a self-evolving domain expert. Autonomously researches a topic for hours, builds a local indexed knowledge base, and answers future questions with citations from real sources.",
  "author": "EdgePro001",
  "license": "MIT",
  "tags": ["learning", "knowledge", "research", "expert", "domain"],
  "openclaw": {
    "emoji": "🧠",
    "skillKey": "clawexpert",
    "homepage": "https://github.com/EdgePro001/ClawExpert",
    "always": true
  }
}


path: README.md
content: # ClawExpert 🧠

**Make your Claw a Self-Evolving Domain Expert**

[中文版](README-zh.md)

ClawExpert is an OpenClaw skill that lets your agent autonomously research any topic for hours, then answer questions from a structured, citable local knowledge base — not from parametric memory.

---

## Quick Start

```bash
# Install
clawhub install clawexpert

# Learn a topic
/clawexpert learn Bretton Woods System

# Specify max duration
/clawexpert learn --hours 3 "China A-share market framework"

# Check status
/clawexpert status

# Deepen an existing topic
/clawexpert learn --deepen bretton-woods-system
```

After learning, ask questions normally. ClawExpert auto-detects relevant topics and answers from the knowledge base with source citations.

---

## How It Works

### Learning (`/clawexpert learn`)

| Step | What happens |
|------|-------------|
| **1. Init** `[0%]` | Generates slug, checks for duplicate/overlapping topics, creates directories |
| **2. Decompose** `[5%]` | Breaks topic into subtopics based on complexity (min 4, no hard cap) |
| **3. Launch** `[10%]` | Spawns parallel subagents (up to `maxChildrenPerAgent`); pipeline-batches if subtopics exceed the limit |
| **4. Monitor** `[10–60%]` | Polls flag files every 30s, shows live progress |
| **5. Merge** `[60–97%]` | Deduplicates, builds knowledge tree, writes formal nodes and index |
| **6. Index** `[97%]` | Writes to the library-style `_index/` layer (L1 → L2 → topic summaries + source abstracts) |
| **7. Report** `[100%]` | Outputs completion summary |

### Retrieval (ask any question)

Three-layer retrieval, coarse to fine:

```
Layer 1: _index/_root_index.md
         Topic map across all domains — routing decision made here

Layer 2: _index/{l1}/{l2}/_index.md
         Key conclusions + source abstracts per topic — most questions answered here

Layer 3: {slug}/raw/web/*.md
         Original downloaded documents — triggered only for precise fact verification
```

### Source Priority (T1–T4)

| Tier | Examples |
|------|---------|
| **T1** | Peer-reviewed papers, official standards, government statistical reports |
| **T1.5** | Preprints (arXiv, SSRN) from recognized institutions |
| **T2** | Central bank publications, think tank reports, international organization docs, official social media |
| **T3** | Specialist media, domain-specific publications, expert commentary |
| **T4** | General news, blogs, aggregators |

Higher-tier sources are preferred during learning; T4 is used only when T1–T3 coverage is insufficient.

---

## Knowledge Base Structure

```
~/.openclaw/workspace/knowledge/
│
├── _categories.json                   # Your L1 category definitions (set on first use)
│
├── _index/                            # Library-style index layer
│   ├── _root_index.md                 # Full knowledge map — auto-loaded each session
│   └── {l1-id}/
│       └── {l2-label}/
│           └── _index.md             # Per-category index: summaries, conclusions, source abstracts
│
└── {topic-slug}/                      # Storage layer — one directory per topic
    ├── meta.json                      # Metadata: status, subtopics, session log
    ├── index.md                       # Knowledge tree overview + further directions
    ├── raw/
    │   ├── web/{hash}.md              # Downloaded web sources
    │   └── pdf/{hash}/                # PDF sources (split into parts if large)
    └── nodes/
        ├── node-001.md                # Formal knowledge nodes
        └── ...
```

---

## Commands

| Command | Description |
|---------|-------------|
| `/clawexpert learn <topic>` | Start learning |
| `/clawexpert learn --hours <n> <topic>` | Set max duration (hours, default: 2) |
| `/clawexpert learn --deepen <slug>` | Deepen an existing topic (append, no overwrite) |
| `/clawexpert learn --auto` | Auto-mode: deepen all incomplete topics (for Cron) |
| `/clawexpert status` | Show all topic statuses |
| `/clawexpert show <slug>` | Display knowledge tree for a topic |
| `/clawexpert ask <slug> <question>` | Force answer from a specific topic's knowledge base |
| `/clawexpert forget <slug>` | Delete a topic's knowledge (requires confirmation) |
| `/clawexpert export <slug>` | Export knowledge tree as a single Markdown file |

---

## Configuration

Recommended `openclaw.json` settings:

```json
{
  "agents": {
    "defaults": {
      "subagents": { "maxChildrenPerAgent": 8 },
      "pdfModel": { "primary": "google/gemini-2.0-flash" }
    }
  },
  "plugins": {
    "entries": {
      "google": {
        "config": {
          "webSearch": {
            "apiKey": "{YOUR_GOOGLE_AI_STUDIO_KEY}",
            "model": "gemini-2.0-flash"
          }
        }
      }
    }
  },
  "env": {
    "CLAWEXPERT_KNOWLEDGE_DIR": "/custom/path"
  }
}
```

| Setting | Notes |
|---------|-------|
| `maxChildrenPerAgent` | Parallel subagent limit (default: 5 if not set) |
| `pdfModel` | Must use `provider: "google"` for native PDF support |
| Web search | Requires Google AI Studio key; use `gemini-2.0-flash` (15 RPM free tier) — `gemini-2.5-flash` is only 5 RPM |
| `CLAWEXPERT_KNOWLEDGE_DIR` | Custom KB path (default: `~/.openclaw/workspace/knowledge`) |

---

## Comparison with Other Skills

| | self-improving-agent | ClawExpert |
|--|---------------------|------------|
| What it learns | How to do things (procedural) | What is true (declarative) |
| Knowledge source | Conversation feedback | Autonomous web research |
| Best for | Long-term project collaboration | Domain-specific Q&A with citations |

Both can be installed simultaneously — they complement rather than compete.

---

## License

MIT License


path: README-zh.md
content: # ClawExpert 🧠

**让你的 Claw 成为自我进化的领域专家**

[English](README.md)

ClawExpert 是一个 OpenClaw 学习插件。安装后，你可以让 OpenClaw 围绕任意主题自主研究数小时，将成果沉淀为结构化、可引用的本地知识库——回答问题时基于真实资料，而非模型参数记忆。

---

## 快速开始

```bash
# 安装
clawhub install clawexpert

# 开始学习
/clawexpert learn 布雷顿森林体系的建立与瓦解

# 指定学习时长
/clawexpert learn --hours 3 A股市场分析框架
  
# 查看状态
/clawexpert status

# 深化已有主题
/clawexpert learn --deepen bretton-woods-system
```

学习完成后，直接提问即可。ClawExpert 自动识别相关主题，基于知识库给出带来源引用的回答。

---

## 工作原理

### 学习流程（`/clawexpert learn`）

| 步骤 | 内容 |
|------|------|
| **1. 初始化** `[0%]` | 生成 slug，检查同名/近似主题冲突，创建目录结构 |
| **2. 拆题** `[5%]` | 根据主题复杂度拆解为子课题（最少 4 个，无硬性上限） |
| **3. 并发学习** `[10%]` | 并行启动 subagent（上限为 `maxChildrenPerAgent`），超出则分批流水线执行 |
| **4. 监控** `[10–60%]` | 每 30 秒检查完成情况，实时显示进度 |
| **5. 合并** `[60–97%]` | 去重、构建知识树、写入正式节点与主题索引 |
| **6. 写索引** `[97%]` | 更新图书馆式 `_index/` 层（L1 → L2 → 主题摘要 + 资料摘要） |
| **7. 报告** `[100%]` | 输出学习完成摘要 |

### 检索流程（直接提问）

三层检索，从粗到细：

```
第一层：_index/_root_index.md
        全知识库主题地图——路由判断在此完成

第二层：_index/{l1}/{l2}/_index.md
        各主题的关键结论 + 资料摘要——大多数问题在此层即可回答

第三层：{slug}/raw/web/*.md
        原始下载资料——仅在需要精确核实数据时触发
```

### 来源优先级（T1–T4）

| 等级 | 示例 |
|------|------|
| **T1** | 同行评审论文、官方标准、政府统计报告 |
| **T1.5** | 知名机构的预印本（arXiv、SSRN 等） |
| **T2** | 央行报告、智库研究、国际组织文件、官方社交媒体 |
| **T3** | 专业媒体、领域专刊、专家评论 |
| **T4** | 大众媒体、博客、聚合站点 |

学习时优先采用高等级来源，仅在 T1–T3 覆盖不足时才使用 T4。

---

## 知识库结构

```
~/.openclaw/workspace/knowledge/
│
├── _categories.json                    # 你定义的 L1 一级分类（首次使用时设置）
│
├── _index/                             # 图书馆式索引层
│   ├── _root_index.md                  # 全知识库地图——每次会话自动加载
│   └── {l1-id}/
│       └── {l2-label}/
│           └── _index.md              # 分类索引：主题摘要、关键结论、资料摘要
│
└── {topic-slug}/                       # 存储层——每个主题一个目录
    ├── meta.json                       # 元数据：状态、子课题、会话记录
    ├── index.md                        # 知识树总览 + 待深入方向
    ├── raw/
    │   ├── web/{hash}.md               # 下载的网页资料（Markdown 格式）
    │   └── pdf/{hash}/                 # PDF 资料（超长时拆分为 part-*.md）
    └── nodes/
        ├── node-001.md                 # 正式知识节点
        └── ...
```

---

## 命令列表

| 命令 | 功能 |
|------|------|
| `/clawexpert learn <topic>` | 开始学习 |
| `/clawexpert learn --hours <n> <topic>` | 指定最长学习时长（小时，默认 2） |
| `/clawexpert learn --deepen <slug>` | 深化已有主题（追加，不覆盖） |
| `/clawexpert learn --auto` | 自动模式：深化所有未完成主题（适合 Cron） |
| `/clawexpert status` | 查看所有主题状态 |
| `/clawexpert show <slug>` | 展示某主题知识树 |
| `/clawexpert ask <slug> <question>` | 强制基于指定主题知识库回答 |
| `/clawexpert forget <slug>` | 删除某主题全部知识（需确认） |
| `/clawexpert export <slug>` | 将知识树导出为单文件 Markdown |

---

## 配置

推荐的 `openclaw.json` 配置：

```json
{
  "agents": {
    "defaults": {
      "subagents": { "maxChildrenPerAgent": 8 },
      "pdfModel": { "primary": "google/gemini-2.0-flash" }
    }
  },
  "plugins": {
    "entries": {
      "google": {
        "config": {
          "webSearch": {
            "apiKey": "{你的 Google AI Studio Key}",
            "model": "gemini-2.0-flash"
          }
        }
      }
    }
  },
  "env": {
    "CLAWEXPERT_KNOWLEDGE_DIR": "/自定义路径"
  }
}
```

| 配置项 | 说明 |
|--------|------|
| `maxChildrenPerAgent` | 并发 subagent 上限（未配置时默认为 5） |
| `pdfModel` | 必须使用 `provider: "google"` 才能走原生 PDF 路径 |
| 网络搜索 | 需要 Google AI Studio API Key；推荐 `gemini-2.0-flash`（免费层 15 RPM），`gemini-2.5-flash` 仅 5 RPM |
| `CLAWEXPERT_KNOWLEDGE_DIR` | 自定义知识库路径（默认 `~/.openclaw/workspace/knowledge`） |

---

## 与其他 Skill 的关系

| | self-improving-agent | ClawExpert |
|--|---------------------|------------|
| 学什么 | 怎么做事（程序性知识） | 知道什么（陈述性知识） |
| 知识来源 | 对话纠错 | 自主 Web 学习 |
| 适用场景 | 长期项目协作 | 专业领域问答与引用 |

两者可同时安装，互补不竞争。

---

## 许可证

MIT License


path: scripts/clawexpert_common.py
content: #!/usr/bin/env python3
"""Shared helpers for ClawExpert bundled scripts."""

from __future__ import annotations

import datetime as _dt
import glob
import json
import os
import pathlib
import re
from typing import Iterable

TOKEN_RE = re.compile(r"[A-Za-z0-9\u4e00-\u9fff]+")
LIST_SPLIT_RE = re.compile(r"\s*(?:\||;|；|,|，|/|、)\s*")
SEMANTIC_STOPWORDS = {
    "the",
    "and",
    "for",
    "with",
    "from",
    "into",
    "that",
    "this",
    "topic",
    "node",
    "method",
    "methods",
    "study",
    "analysis",
    "framework",
    "frameworks",
    "system",
    "systems",
    "方法",
    "研究",
    "分析",
    "问题",
    "系统",
    "框架",
}
GENERIC_QUERY_FILLERS = [
    "之间的",
    "之间",
    "如何",
    "什么是",
    "是什么",
    "关于",
    "有关",
    "的",
    "和",
    "与",
    "及",
]


def knowledge_dir(value: str | None = None) -> pathlib.Path:
    if value:
        return pathlib.Path(value).expanduser().resolve()
    return pathlib.Path.home().joinpath(".openclaw", "workspace", "knowledge").resolve()


def utc_now() -> _dt.datetime:
    return _dt.datetime.now(_dt.timezone.utc)


def iso_now() -> str:
    return utc_now().isoformat().replace("+00:00", "Z")


def topic_dir(kb: pathlib.Path, slug: str) -> pathlib.Path:
    return kb / slug


def meta_path(kb: pathlib.Path, slug: str) -> pathlib.Path:
    return topic_dir(kb, slug) / "meta.json"


def progress_path(kb: pathlib.Path, slug: str, subtopic_id: str) -> pathlib.Path:
    return topic_dir(kb, slug) / f"progress-{subtopic_id}.json"


def flag_path(kb: pathlib.Path, slug: str, subtopic_id: str) -> pathlib.Path:
    return topic_dir(kb, slug) / f"done-{subtopic_id}.flag"


def ensure_dir(path: pathlib.Path) -> pathlib.Path:
    path.mkdir(parents=True, exist_ok=True)
    return path


def load_json(path: pathlib.Path, default=None):
    if not path.exists():
        return default
    with path.open("r", encoding="utf-8") as fh:
        return json.load(fh)


def save_json(path: pathlib.Path, data) -> None:
    ensure_dir(path.parent)
    with path.open("w", encoding="utf-8") as fh:
        json.dump(data, fh, indent=2, ensure_ascii=False)
        fh.write("\n")


def count_source_files(topic_root: pathlib.Path) -> int:
    web = glob.glob(str(topic_root / "raw" / "web" / "*.md"))
    pdf_single = glob.glob(str(topic_root / "raw" / "pdf" / "*.md"))
    pdf_split = glob.glob(str(topic_root / "raw" / "pdf" / "*" / "_index.md"))
    return len(web) + len(pdf_single) + len(pdf_split)


def count_formal_nodes(topic_root: pathlib.Path) -> int:
    return len(glob.glob(str(topic_root / "nodes" / "node-*.md")))


def list_formal_nodes(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted((topic_root / "nodes").glob("node-*.md"))


def list_framework_nodes(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted((topic_root / "nodes").glob("framework-*.md"))


def list_all_knowledge_nodes(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted(list_formal_nodes(topic_root) + list_framework_nodes(topic_root))


def list_subtopic_nodes(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted(topic_root.glob("nodes/sub-*/node*.md"))


def list_flags(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted(topic_root.glob("done-*.flag"))


def list_progress_files(topic_root: pathlib.Path) -> list[pathlib.Path]:
    return sorted(topic_root.glob("progress-*.json"))


def slugify(text: str, max_length: int = 60) -> str:
    text = text.strip().lower()
    chars: list[str] = []
    for char in text:
        if char.isascii() and (char.isalnum() or char == "-"):
            chars.append(char)
            continue
        if _is_cjk(char) or char.isdigit():
            chars.append(char)
            continue
        chars.append("-")
    slug = re.sub(r"-{2,}", "-", "".join(chars)).strip("-")
    if not slug:
        slug = "topic"
    return slug[:max_length].rstrip("-")


def _is_cjk(char: str) -> bool:
    code = ord(char)
    return (
        0x3400 <= code <= 0x4DBF
        or 0x4E00 <= code <= 0x9FFF
        or 0xF900 <= code <= 0xFAFF
    )


def parse_frontmatter_and_body(text: str) -> tuple[dict[str, str], str]:
    if not text.startswith("---\n"):
        return {}, text
    parts = text.split("\n---\n", 1)
    if len(parts) != 2:
        return {}, text
    raw_meta, body = parts
    meta: dict[str, str] = {}
    for line in raw_meta.splitlines()[1:]:
        if ":" not in line:
            continue
        key, value = line.split(":", 1)
        meta[key.strip()] = value.strip()
    return meta, body.lstrip()


def split_inline_list(value: str | None) -> list[str]:
    if not value:
        return []
    if "\n" in value:
        parts = [line.strip("- ").strip() for line in value.splitlines()]
    else:
        parts = LIST_SPLIT_RE.split(value.strip())
    output: list[str] = []
    seen: set[str] = set()
    for item in parts:
        normalized = normalize_space(item)
        if not normalized:
            continue
        key = normalized.lower()
        if key in seen:
            continue
        seen.add(key)
        output.append(normalized)
    return output


def extract_section(markdown: str, heading: str) -> str:
    lines = markdown.splitlines()
    current: list[str] = []
    capture = False
    pattern = f"## {heading}".strip()
    for line in lines:
        if line.strip() == pattern:
            capture = True
            current = []
            continue
        if capture and line.startswith("## "):
            break
        if capture:
            current.append(line)
    return "\n".join(current).strip()


def first_heading(markdown: str) -> str:
    for line in markdown.splitlines():
        stripped = line.strip()
        if stripped.startswith("# "):
            return stripped[2:].strip()
    return ""


def best_label(frontmatter: dict[str, str], body: str, fallback: str) -> str:
    label = normalize_space(frontmatter.get("label", "")) or fallback
    if re.match(r"^(node|framework|subtopic|topic)-\d+", label.lower()):
        heading = first_heading(body)
        if heading:
            return heading
    return label


def extract_bullets(section: str) -> list[str]:
    bullets: list[str] = []
    for line in section.splitlines():
        stripped = line.strip()
        if stripped.startswith("- "):
            bullets.append(stripped[2:].strip())
        elif re.match(r"^\d+\.\s+", stripped):
            bullets.append(re.sub(r"^\d+\.\s+", "", stripped))
    return bullets


def extract_checklist_items(section: str, checked: bool | None = None) -> list[str]:
    items: list[str] = []
    pattern = re.compile(r"^- \[(?P<mark>[ xX])\]\s+(?P<text>.+?)\s*$")
    for line in section.splitlines():
        match = pattern.match(line.strip())
        if not match:
            continue
        is_checked = match.group("mark").lower() == "x"
        if checked is None or checked == is_checked:
            items.append(match.group("text").strip())
    return items


def first_paragraph(text: str) -> str:
    chunks = [chunk.strip() for chunk in re.split(r"\n\s*\n", text) if chunk.strip()]
    return chunks[0] if chunks else ""


def first_sentence(text: str) -> str:
    text = " ".join(text.split())
    if not text:
        return ""
    parts = re.split(r"(?<=[.!?。！？])\s+", text)
    return parts[0].strip()


def relpath(target: pathlib.Path, start: pathlib.Path) -> str:
    return pathlib.Path(os.path.relpath(target, start)).as_posix()


def read_text(path: pathlib.Path) -> str:
    return path.read_text(encoding="utf-8", errors="ignore")


def write_text(path: pathlib.Path, content: str) -> None:
    ensure_dir(path.parent)
    path.write_text(content, encoding="utf-8")


def sum_int(values: Iterable[int]) -> int:
    total = 0
    for value in values:
        total += int(value)
    return total


def normalize_space(text: str) -> str:
    return " ".join(str(text).split()).strip()


def compact_text(text: str) -> str:
    return "".join(char.lower() for char in str(text) if char.isalnum() or _is_cjk(char))


def semantic_strip(text: str) -> str:
    cleaned = normalize_space(text).lower()
    for filler in sorted(GENERIC_QUERY_FILLERS, key=len, reverse=True):
        cleaned = cleaned.replace(filler.lower(), "")
    return compact_text(cleaned)


def cjk_phrase_abbreviation(text: str) -> str:
    stripped = semantic_strip(text)
    chars = [char for char in stripped if _is_cjk(char)]
    if len(chars) >= 4 and len(chars) % 2 == 0:
        return "".join(chars[idx] for idx in range(0, len(chars), 2))
    return ""


def semantic_tokens(text: str) -> set[str]:
    tokens: set[str] = set()
    for token in TOKEN_RE.findall(str(text).lower()):
        [REDACTED_SECRET])
        if len(token) <= 1:
            continue
        tokens.add(token)
        if all(_is_cjk(ch) for ch in token):
            if len(token) <= 8 and len(token) % 2 == 0:
                for idx in range(0, len(token), 2):
                    piece = token[idx : idx + 2]
                    if len(piece) == 2:
                        tokens.add(piece)
            for idx in range(len(token) - 1):
                bigram = token[idx : idx + 2]
                if len(bigram) == 2:
                    tokens.add(bigram)
    return {token for token in tokens if token not in SEMANTIC_STOPWORDS}


def char_bigrams(text: str) -> set[str]:
    compact = semantic_strip(text)
    if len(compact) < 2:
        return {compact} if compact else set()
    return {compact[idx : idx + 2] for idx in range(len(compact) - 1)}


def jaccard(a: set[str], b: set[str]) -> float:
    if not a or not b:
        return 0.0
    return len(a & b) / len(a | b)


def semantic_signature(texts: Iterable[str]) -> dict[str, object]:
    raw_values = [normalize_space(text) for text in texts if normalize_space(text)]
    compact_values = {compact_text(text) for text in raw_values if compact_text(text)}
    stripped_values = {semantic_strip(text) for text in raw_values if semantic_strip(text)}
    abbreviations = {abbr for text in raw_values if (abbr := cjk_phrase_abbreviation(text))}
    token_sets = [semantic_tokens(text) for text in raw_values]
    bigram_sets = [char_bigrams(text) for text in raw_values]
    tokens: set[str] = set()
    bigrams: set[str] = set()
    for token_set in token_sets:
        tokens.update(token_set)
    for bigram_set in bigram_sets:
        bigrams.update(bigram_set)
    return {
        "texts": raw_values,
        "compact": compact_values,
        "stripped": stripped_values,
        "abbreviations": abbreviations,
        "tokens": tokens,
        "bigrams": bigrams,
    }


def frontmatter_aliases(frontmatter: dict[str, str]) -> list[str]:
    aliases: list[str] = []
    for key in ("aliases", "alternate_labels", "routing_aliases"):
        aliases.extend(split_inline_list(frontmatter.get(key)))
    canonical = normalize_space(frontmatter.get("canonical_intent", ""))
    if canonical:
        aliases.append(canonical)
    return split_inline_list(" | ".join(aliases))


path: scripts/update_meta.py
content: #!/usr/bin/env python3
"""Finalize topic metadata after merge."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import (
    count_formal_nodes,
    count_source_files,
    iso_now,
    knowledge_dir,
    load_json,
    meta_path,
    save_json,
    topic_dir,
)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--status")
    parser.add_argument("--missing-status", default="timeout")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    root = topic_dir(kb, args.slug)
    meta_file = meta_path(kb, args.slug)
    meta = load_json(meta_file, default=None)
    if meta is None:
        raise SystemExit(f"Missing meta.json: {meta_file}")

    flags = {}
    for sub in meta.get("subtopics", []):
        sub_id = sub.get("id")
        if not sub_id:
            continue
        flag_file = root / f"done-{sub_id}.flag"
        if flag_file.exists():
            flags[sub_id] = load_json(flag_file, default={}) or {}

    flag_statuses = [data.get("status", "done") for data in flags.values()]
    inferred_status = "complete"
    if len(flags) < len(meta.get("subtopics", [])) or any(status != "done" for status in flag_statuses):
        inferred_status = "partial"

    meta["status"] = args.status or inferred_status
    meta["last_updated"] = iso_now()
    meta["source_count"] = count_source_files(root)
    meta["node_count"] = count_formal_nodes(root)

    sessions = meta.get("learning_sessions", [])
    if sessions:
        sessions[-1]["finished"] = iso_now()
        sessions[-1]["sources_added"] = sum(int(item.get("sources_added", 0)) for item in flags.values())

    for sub in meta.get("subtopics", []):
        sub_id = sub.get("id")
        flag = flags.get(sub_id)
        if flag:
            sub["status"] = flag.get("status", "done")
            sub["sources_added"] = int(flag.get("sources_added", 0))
        else:
            sub["status"] = args.missing_status

    save_json(meta_file, meta)
    print(json.dumps({
        "slug": args.slug,
        "status": meta["status"],
        "source_count": meta["source_count"],
        "node_count": meta["node_count"],
    }, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/write_progress.py
content: #!/usr/bin/env python3
"""Write or remove subtopic heartbeat files."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import iso_now, knowledge_dir, load_json, progress_path, save_json


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--subtopic-id", required=True)
    parser.add_argument("--status")
    parser.add_argument("--search-round", type=int)
    parser.add_argument("--sources-found", type=int)
    parser.add_argument("--sources-saved", type=int)
    parser.add_argument("--last-url")
    parser.add_argument("--last-error")
    parser.add_argument("--node-file", action="append", default=[])
    parser.add_argument("--remove", action="store_true")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    path = progress_path(kb, args.slug, args.subtopic_id)

    if args.remove:
        removed = False
        if path.exists():
            path.unlink()
            removed = True
        print(json.dumps({"removed": removed, "path": str(path)}))
        return 0

    data = load_json(path, default={}) or {}
    data.setdefault("subtopic_id", args.subtopic_id)
    data.setdefault("started_at", iso_now())
    if args.status:
        data["status"] = args.status
    if args.search_round is not None:
        data["search_round"] = args.search_round
    if args.sources_found is not None:
        data["sources_found"] = args.sources_found
    if args.sources_saved is not None:
        data["sources_saved"] = args.sources_saved
    if args.last_url:
        data["last_url"] = args.last_url
    if args.last_error:
        data["last_error"] = args.last_error
    if args.node_file:
        existing = list(data.get("node_files", []))
        for item in args.node_file:
            if item not in existing:
                existing.append(item)
        data["node_files"] = existing
    data["updated_at"] = iso_now()

    save_json(path, data)
    print(json.dumps({"path": str(path), "status": data.get("status"), "updated_at": data["updated_at"]}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/write_flag.py
content: #!/usr/bin/env python3
"""Write a subtopic completion flag and optionally remove the heartbeat file."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import flag_path, iso_now, knowledge_dir, progress_path, save_json


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--subtopic-id", required=True)
    parser.add_argument("--status", required=True)
    parser.add_argument("--sources-added", type=int, default=0)
    parser.add_argument("--node-file", action="append", default=[])
    parser.add_argument("--cleanup-progress", action="store_true")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    path = flag_path(kb, args.slug, args.subtopic_id)
    data = {
        "status": args.status,
        "subtopic_id": args.subtopic_id,
        "sources_added": args.sources_added,
        "node_files": args.node_file,
        "completed_at": iso_now(),
    }
    save_json(path, data)

    progress_removed = False
    if args.cleanup_progress:
        progress = progress_path(kb, args.slug, args.subtopic_id)
        if progress.exists():
            progress.unlink()
            progress_removed = True

    print(json.dumps({"path": str(path), "status": args.status, "progress_removed": progress_removed}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/resolve_query_nodes.py
content: #!/usr/bin/env python3
"""Retrieve the most likely topic/node candidates for a user query."""

from __future__ import annotations

import argparse
import json
import pathlib

from clawexpert_common import (
    best_label,
    extract_bullets,
    extract_section,
    first_sentence,
    frontmatter_aliases,
    jaccard,
    knowledge_dir,
    list_all_knowledge_nodes,
    load_json,
    meta_path,
    parse_frontmatter_and_body,
    read_text,
    semantic_signature,
)


def topic_dirs(kb: pathlib.Path) -> list[pathlib.Path]:
    return sorted(
        child
        for child in kb.iterdir()
        if child.is_dir() and not child.name.startswith("_") and (child / "meta.json").exists()
    )


def score_signature(query_sig: dict, candidate_sig: dict) -> tuple[float, list[str]]:
    reasons: list[str] = []
    score = 0.0

    if query_sig["compact"] & candidate_sig["compact"]:
        score += 1.0
        reasons.append("exact_compact")
    if query_sig["stripped"] & candidate_sig["stripped"]:
        score += 0.85
        reasons.append("normalized_match")
    if query_sig["abbreviations"] & candidate_sig["abbreviations"]:
        score += 0.75
        reasons.append("abbreviation_match")

    query_compact = next(iter(query_sig["compact"]), "")
    candidate_compact = next(iter(candidate_sig["compact"]), "")
    if query_compact and candidate_compact:
        if query_compact in candidate_compact or candidate_compact in query_compact:
            score += 0.45
            reasons.append("compact_containment")

    token_overlap = jaccard(query_sig["tokens"], candidate_sig["tokens"])
    bigram_overlap = jaccard(query_sig["bigrams"], candidate_sig["bigrams"])
    if token_overlap > 0:
        score += token_overlap * 0.9
        reasons.append(f"token_overlap:{token_overlap:.2f}")
    if bigram_overlap > 0:
        score += bigram_overlap * 0.6
        reasons.append(f"bigram_overlap:{bigram_overlap:.2f}")

    return score, reasons


def infer_relation(query_sig: dict, candidate_sig: dict, reasons: list[str]) -> str:
    if "exact_compact" in reasons:
        return "exact"
    if "normalized_match" in reasons or "abbreviation_match" in reasons:
        return "alias_like"

    query_tokens = query_sig["tokens"]
    candidate_tokens = candidate_sig["tokens"]
    if query_tokens and candidate_tokens:
        if query_tokens < candidate_tokens:
            return "candidate_is_narrower"
        if candidate_tokens < query_tokens:
            return "candidate_is_broader"
    return "related"


def collect_topic_candidates(kb: pathlib.Path, query_sig: dict) -> list[dict]:
    candidates: list[dict] = []
    for root in topic_dirs(kb):
        meta = load_json(meta_path(kb, root.name), default={}) or {}
        index_categories = meta.get("index_categories", {}) or {}
        texts = [
            meta.get("topic", root.name),
            meta.get("slug", root.name),
            index_categories.get("l1_label", ""),
            index_categories.get("l1", ""),
            *[
                str(item.get("l2", ""))
                for item in (index_categories.get("subtopics", {}) or {}).values()
                if item.get("l2")
            ],
        ]
        topic_index = root / "index.md"
        if topic_index.exists():
            index_text = read_text(topic_index)
            texts.append(first_sentence(extract_section(index_text, "Core Findings Summary")))
        signature = semantic_signature(texts)
        score, reasons = score_signature(query_sig, signature)
        if score <= 0:
            continue
        candidates.append(
            {
                "kind": "topic",
                "slug": meta.get("slug", root.name),
                "topic": meta.get("topic", root.name),
                "score": round(score, 3),
                "relation": infer_relation(query_sig, signature, reasons),
                "reasons": reasons,
                "l1": index_categories.get("l1"),
                "l1_label": index_categories.get("l1_label"),
                "summary": texts[-1] if texts else "",
            }
        )
    return sorted(candidates, key=lambda item: (-item["score"], item["slug"]))


def collect_node_candidates(kb: pathlib.Path, query_sig: dict) -> list[dict]:
    candidates: list[dict] = []
    for root in topic_dirs(kb):
        meta = load_json(meta_path(kb, root.name), default={}) or {}
        for node_path in list_all_knowledge_nodes(root):
            frontmatter, body = parse_frontmatter_and_body(read_text(node_path))
            label = best_label(frontmatter, body, node_path.stem)
            aliases = frontmatter_aliases(frontmatter)
            summary = first_sentence(extract_section(body, "Summary"))
            conclusions = extract_bullets(extract_section(body, "Key Conclusions"))[:3]
            signature = semantic_signature([label, summary, *aliases, *conclusions])
            score, reasons = score_signature(query_sig, signature)
            if score <= 0:
                continue
            candidates.append(
                {
                    "kind": "node",
                    "slug": meta.get("slug", root.name),
                    "topic": meta.get("topic", root.name),
                    "node_id": frontmatter.get("id") or node_path.stem,
                    "label": label,
                    "aliases": aliases,
                    "path": str(node_path),
                    "parent": frontmatter.get("parent") or "root",
                    "score": round(score, 3),
                    "relation": infer_relation(query_sig, signature, reasons),
                    "reasons": reasons,
                    "summary": summary,
                }
            )
    return sorted(candidates, key=lambda item: (-item["score"], item["path"]))


def recommended_action(top_topic: dict | None, top_node: dict | None) -> str:
    if top_node and top_node["score"] >= 1.0:
        return "open_node"
    if top_node and top_node["score"] >= 0.28:
        return "inspect_node_then_neighbors"
    if top_topic and top_topic["score"] >= 0.15:
        return "open_topic_branch"
    return "ask_agent_to_create_or_expand"


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--query", required=True)
    parser.add_argument("--top-k", type=int, default=5)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    query_sig = semantic_signature([args.query])
    topics = collect_topic_candidates(kb, query_sig)[: args.top_k]
    nodes = collect_node_candidates(kb, query_sig)[: args.top_k]
    payload = {
        "query": args.query,
        "recommended_action": recommended_action(topics[0] if topics else None, nodes[0] if nodes else None),
        "top_topics": topics,
        "top_nodes": nodes,
        "agent_decision_required": True,
        "note": "This script only recalls likely candidates. The main agent must decide whether the query maps to an existing node, a broader branch, or a new node.",
    }
    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/suggest_node_merges.py
content: #!/usr/bin/env python3
"""Suggest node relationships to help the main agent merge with less fragmentation."""

from __future__ import annotations

import argparse
import json
import pathlib
from itertools import combinations

from clawexpert_common import (
    best_label,
    char_bigrams,
    extract_bullets,
    extract_section,
    first_sentence,
    frontmatter_aliases,
    jaccard,
    knowledge_dir,
    list_formal_nodes,
    list_subtopic_nodes,
    parse_frontmatter_and_body,
    read_text,
    semantic_signature,
    split_inline_list,
    topic_dir,
)


def load_nodes(root: pathlib.Path, formal_only: bool) -> list[dict]:
    paths = list_formal_nodes(root) if formal_only else list_subtopic_nodes(root)
    nodes: list[dict] = []
    for path in paths:
        frontmatter, body = parse_frontmatter_and_body(read_text(path))
        label = best_label(frontmatter, body, path.stem)
        summary = first_sentence(extract_section(body, "Summary"))
        conclusions = extract_bullets(extract_section(body, "Key Conclusions"))[:4]
        aliases = frontmatter_aliases(frontmatter)
        include_terms = split_inline_list(frontmatter.get("inclusion"))
        texts = [label, summary, *aliases, *conclusions, *include_terms]
        signature = semantic_signature(texts)
        nodes.append(
            {
                "path": str(path),
                "label": label,
                "summary": summary,
                "aliases": aliases,
                "parent": frontmatter.get("parent") or "root",
                "type": frontmatter.get("type") or "topic",
                "signature": signature,
            }
        )
    return nodes


def classify_pair(left: dict, right: dict) -> dict | None:
    left_sig = left["signature"]
    right_sig = right["signature"]

    exact_text = bool(left_sig["compact"] & right_sig["compact"])
    exact_alias = bool(left_sig["stripped"] & right_sig["stripped"])
    exact_abbreviation = bool(left_sig["abbreviations"] & right_sig["abbreviations"])
    token_overlap = jaccard(left_sig["tokens"], right_sig["tokens"])
    bigram_overlap = jaccard(left_sig["bigrams"], right_sig["bigrams"])
    similarity = max(token_overlap, bigram_overlap)

    left_compact = next(iter(left_sig["compact"]), "")
    right_compact = next(iter(right_sig["compact"]), "")
    containment = (
        bool(left_compact and right_compact and left_compact in right_compact and left_compact != right_compact),
        bool(left_compact and right_compact and right_compact in left_compact and left_compact != right_compact),
    )

    reasons: list[str] = []
    relation = None
    confidence = 0.0

    if exact_text:
        relation = "merge_candidate"
        confidence = 0.98
        reasons.append("exact_compact_match")
    elif exact_alias or exact_abbreviation:
        relation = "merge_candidate"
        confidence = 0.9
        reasons.append("normalized_or_abbreviation_match")
    elif similarity >= 0.62:
        relation = "merge_candidate"
        confidence = similarity
        reasons.append("high_lexical_overlap")
    elif containment[0] or containment[1]:
        relation = "parent_child_candidate"
        confidence = 0.72
        reasons.append("name_containment")
    elif similarity >= 0.28:
        relation = "related_candidate"
        confidence = similarity
        reasons.append("partial_overlap")

    if not relation:
        return None

    parent_hint = None
    child_hint = None
    if relation == "parent_child_candidate":
        if containment[0]:
            parent_hint, child_hint = left["label"], right["label"]
        elif containment[1]:
            parent_hint, child_hint = right["label"], left["label"]

    return {
        "left": {
            "label": left["label"],
            "path": left["path"],
            "aliases": left["aliases"],
            "parent": left["parent"],
        },
        "right": {
            "label": right["label"],
            "path": right["path"],
            "aliases": right["aliases"],
            "parent": right["parent"],
        },
        "relation": relation,
        "confidence": round(confidence, 3),
        "token_overlap": round(token_overlap, 3),
        "bigram_overlap": round(bigram_overlap, 3),
        "reasons": reasons,
        "parent_hint": parent_hint,
        "child_hint": child_hint,
        "agent_decision_required": True,
    }


def canonical_aliases(*nodes: dict) -> list[str]:
    values: list[str] = []
    for node in nodes:
        values.append(node["label"])
        values.extend(node["aliases"])
    deduped: list[str] = []
    seen: set[str] = set()
    for value in values:
        key = value.strip().lower()
        if not key or key in seen:
            continue
        seen.add(key)
        deduped.append(value.strip())
    return deduped


def build_merge_groups(pairs: list[dict]) -> list[dict]:
    groups: list[dict] = []
    used_paths: set[str] = set()
    for pair in sorted(pairs, key=lambda item: item["confidence"], reverse=True):
        left_path = pair["left"]["path"]
        right_path = pair["right"]["path"]
        if left_path in used_paths or right_path in used_paths:
            continue
        used_paths.update({left_path, right_path})
        aliases = canonical_aliases(pair["left"], pair["right"])
        groups.append(
            {
                "recommended_label": min(aliases, key=len),
                "aliases": aliases,
                "paths": [left_path, right_path],
                "reasons": pair["reasons"],
                "confidence": pair["confidence"],
                "agent_decision_required": True,
            }
        )
    return groups


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--formal-only", action="store_true")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    root = topic_dir(kb, args.slug)
    nodes = load_nodes(root, formal_only=args.formal_only)

    relations: list[dict] = []
    for left, right in combinations(nodes, 2):
        relation = classify_pair(left, right)
        if relation:
            relations.append(relation)

    merge_candidates = [item for item in relations if item["relation"] == "merge_candidate"]
    parent_child_candidates = [item for item in relations if item["relation"] == "parent_child_candidate"]
    related_candidates = [item for item in relations if item["relation"] == "related_candidate"]

    payload = {
        "slug": args.slug,
        "node_count": len(nodes),
        "merge_groups": build_merge_groups(merge_candidates),
        "merge_candidates": sorted(merge_candidates, key=lambda item: item["confidence"], reverse=True),
        "parent_child_candidates": sorted(parent_child_candidates, key=lambda item: item["confidence"], reverse=True),
        "related_candidates": sorted(related_candidates, key=lambda item: item["confidence"], reverse=True),
        "note": "This script only retrieves likely relationships. The main agent must make the final semantic decision.",
    }
    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/prepare_merge.py
content: #!/usr/bin/env python3
"""Collect merge inputs: subtopic nodes, flags, and heartbeat status."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import knowledge_dir, list_subtopic_nodes, load_json, meta_path, topic_dir


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--stall-seconds", type=int, default=600)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    root = topic_dir(kb, args.slug)
    meta = load_json(meta_path(kb, args.slug), default={}) or {}
    node_files = [str(path) for path in list_subtopic_nodes(root)]

    # Reuse monitor_subtopics without an extra subprocess.
    subtopics = [sub.get("id") for sub in meta.get("subtopics", []) if sub.get("id")]
    flags = {}
    missing_flags = []
    for subtopic_id in subtopics:
        path = root / f"done-{subtopic_id}.flag"
        if path.exists():
            flags[subtopic_id] = load_json(path, default={}) or {}
        else:
            missing_flags.append(subtopic_id)

    summary = {
        "slug": args.slug,
        "subtopic_count": len(subtopics),
        "node_file_count": len(node_files),
        "node_files": node_files,
        "flags": flags,
        "missing_flags": missing_flags,
    }
    print(json.dumps(summary, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/resolve_topic.py
content: #!/usr/bin/env python3
"""Resolve a user-provided topic or slug against the local knowledge base."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import knowledge_dir, load_json, slugify


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--query", required=True)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    query = args.query.strip()
    query_slug = slugify(query)

    matches: list[dict] = []
    for child in sorted(kb.iterdir()):
        if not child.is_dir() or child.name.startswith("_"):
            continue
        meta = load_json(child / "meta.json", default=None)
        if not meta:
            continue
        slug = meta.get("slug", child.name)
        topic = meta.get("topic", slug)
        score = 0
        reasons: list[str] = []
        if query == slug:
            score += 100
            reasons.append("exact_slug")
        if query_slug == slug:
            score += 90
            reasons.append("slugified_query")
        if query == topic:
            score += 80
            reasons.append("exact_topic")
        if query.lower() == str(topic).lower():
            score += 70
            reasons.append("casefold_topic")
        if query in str(topic) or str(topic) in query:
            score += 40
            reasons.append("substring_topic")
        if query_slug in slug or slug in query_slug:
            score += 35
            reasons.append("substring_slug")
        if score > 0:
            matches.append(
                {
                    "slug": slug,
                    "topic": topic,
                    "score": score,
                    "reasons": reasons,
                }
            )

    matches.sort(key=lambda item: (-item["score"], item["slug"]))
    best = matches[0] if matches else None
    payload = {
        "query": query,
        "query_slug": query_slug,
        "exists": bool(best),
        "best_match": best,
        "matches": matches[:5],
    }
    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/cleanup_topic.py
content: #!/usr/bin/env python3
"""Clean temporary subtopic directories, flags, and heartbeat files."""

from __future__ import annotations

import argparse
import json
import shutil

from clawexpert_common import knowledge_dir, list_flags, list_progress_files, topic_dir


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--keep-flags", action="store_true")
    parser.add_argument("--keep-progress", action="store_true")
    parser.add_argument("--keep-subtopic-nodes", action="store_true")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    root = topic_dir(kb, args.slug)
    removed = {"subtopic_nodes": 0, "flags": 0, "progress": 0}

    if not args.keep_subtopic_nodes:
        for path in sorted((root / "nodes").glob("sub-*")):
            shutil.rmtree(path, ignore_errors=True)
            removed["subtopic_nodes"] += 1

    if not args.keep_flags:
        for path in list_flags(root):
            path.unlink(missing_ok=True)
            removed["flags"] += 1

    if not args.keep_progress:
        for path in list_progress_files(root):
            path.unlink(missing_ok=True)
            removed["progress"] += 1

    print(json.dumps(removed))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/init_topic.py
content: #!/usr/bin/env python3
"""Initialize a topic directory and write the initial meta.json."""

from __future__ import annotations

import argparse
import json
import sys

from clawexpert_common import ensure_dir, iso_now, knowledge_dir, meta_path, save_json, slugify, topic_dir


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--topic", required=True)
    parser.add_argument("--slug")
    parser.add_argument("--kb")
    parser.add_argument("--max-hours", type=int, default=2)
    parser.add_argument("--status", default="learning")
    parser.add_argument("--session-id", default="main")
    parser.add_argument("--force", action="store_true")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    slug = args.slug or slugify(args.topic)
    root = topic_dir(kb, slug)
    meta_file = meta_path(kb, slug)

    if meta_file.exists() and not args.force:
        print(f"Refusing to overwrite existing meta.json: {meta_file}", file=sys.stderr)
        return 1

    ensure_dir(root / "raw" / "web")
    ensure_dir(root / "raw" / "pdf")
    ensure_dir(root / "nodes")

    now = iso_now()
    meta = {
        "topic": args.topic,
        "slug": slug,
        "created": now,
        "last_updated": now,
        "status": args.status,
        "source_count": 0,
        "node_count": 0,
        "max_hours": args.max_hours,
        "subtopics": [],
        "learning_sessions": [
            {
                "session_id": args.session_id,
                "started": now,
                "finished": None,
                "sources_added": 0,
            }
        ],
    }
    save_json(meta_file, meta)
    print(json.dumps({"topic": args.topic, "slug": slug, "meta_path": str(meta_file)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/validate_kb.py
content: #!/usr/bin/env python3
"""Validate a topic or the whole knowledge base."""

from __future__ import annotations

import argparse
import json
import pathlib
import re
import sys

from clawexpert_common import count_formal_nodes, count_source_files, knowledge_dir, read_text


LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")


def iter_topics(kb: pathlib.Path, slug: str | None) -> list[pathlib.Path]:
    if slug:
        return [kb / slug]
    return [child for child in sorted(kb.iterdir()) if child.is_dir() and not child.name.startswith("_") and (child / "meta.json").exists()]


def validate_links(path: pathlib.Path) -> list[str]:
    issues = []
    for _label, target in LINK_RE.findall(read_text(path)):
        if target.startswith("http://") or target.startswith("https://") or target.startswith("#"):
            continue
        resolved = (path.parent / target).resolve()
        if not resolved.exists():
            issues.append(f"broken link in {path}: {target}")
    return issues


def validate_topic(topic_root: pathlib.Path) -> list[str]:
    issues = []
    meta_file = topic_root / "meta.json"
    if not meta_file.exists():
        return [f"missing meta.json: {topic_root}"]

    meta = json.loads(read_text(meta_file))
    actual_sources = count_source_files(topic_root)
    actual_nodes = count_formal_nodes(topic_root)

    if meta.get("source_count") != actual_sources:
        issues.append(
            f"{topic_root.name}: source_count mismatch meta={meta.get('source_count')} actual={actual_sources}"
        )
    if meta.get("node_count") != actual_nodes:
        issues.append(
            f"{topic_root.name}: node_count mismatch meta={meta.get('node_count')} actual={actual_nodes}"
        )

    topic_index = topic_root / "index.md"
    if not topic_index.exists():
        issues.append(f"{topic_root.name}: missing index.md")
    else:
        issues.extend(validate_links(topic_index))

    for node in sorted((topic_root / "nodes").glob("node-*.md")):
        issues.extend(validate_links(node))

    return issues


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    issues = []
    for topic_root in iter_topics(kb, args.slug):
        if topic_root.exists():
            issues.extend(validate_topic(topic_root))
        else:
            issues.append(f"missing topic directory: {topic_root}")

    root_index = kb / "_index" / "_root_index.md"
    if not root_index.exists():
        issues.append(f"missing root index: {root_index}")

    payload = {"ok": not issues, "issues": issues}
    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0 if not issues else 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/analyze_gaps.py
content: #!/usr/bin/env python3
"""Analyze a topic's breadth and depth gaps for deepen/superlearn."""

from __future__ import annotations

import argparse
import json
import pathlib

from clawexpert_common import (
    extract_checklist_items,
    extract_section,
    first_sentence,
    knowledge_dir,
    list_all_knowledge_nodes,
    parse_frontmatter_and_body,
    read_text,
    topic_dir,
    write_text,
)


def dedupe_keep_order(items: list[str]) -> list[str]:
    seen: set[str] = set()
    output: list[str] = []
    for item in items:
        key = " ".join(item.split()).strip().lower()
        if not key or key in seen:
            continue
        seen.add(key)
        output.append(item.strip())
    return output


def topic_breadth_tasks(topic_root: pathlib.Path) -> list[dict]:
    tasks: list[dict] = []

    topic_index = topic_root / "index.md"
    if topic_index.exists():
        text = read_text(topic_index)
        for question in extract_checklist_items(extract_section(text, "Further Directions"), checked=False):
            tasks.append(
                {
                    "scope": "topic",
                    "source": "topic-index",
                    "prompt": question,
                    "goal": f"Create or extend knowledge coverage for: {question}",
                }
            )

    for node_path in list_all_knowledge_nodes(topic_root):
        frontmatter, body = parse_frontmatter_and_body(read_text(node_path))
        label = frontmatter.get("label") or node_path.stem
        questions = extract_checklist_items(extract_section(body, "Further Directions"), checked=False)
        for question in questions:
            tasks.append(
                {
                    "scope": "node",
                    "source": str(node_path),
                    "label": label,
                    "prompt": question,
                    "goal": f"Extend the topic beyond current coverage of {label}: {question}",
                }
            )

    return dedupe_keep_order_json(tasks)


def dedupe_keep_order_json(items: list[dict]) -> list[dict]:
    seen: set[str] = set()
    output: list[dict] = []
    for item in items:
        key = json.dumps(item, ensure_ascii=False, sort_keys=True)
        if key in seen:
            continue
        seen.add(key)
        output.append(item)
    return output


def infer_depth_questions(label: str, summary: str, source_count: int, explicit: list[str]) -> list[str]:
    questions = list(explicit)
    if source_count < 3:
        questions.append(f"{label} 目前缺少哪些 primary sources、官方文档或原始论文来支撑核心结论？")
    questions.append(f"{label} 的关键机制、实验细节、时间线或代表性方法，还有哪些重要空缺？")
    questions.append(f"{label} 目前有哪些争议点、局限性、失败案例或反方证据尚未补充？")
    if summary:
        questions.append(f"基于当前总结“{first_sentence(summary)}”，哪些关键论断还需要进一步核验或量化？")
    return dedupe_keep_order(questions)


def node_depth_task(node_path: pathlib.Path) -> dict:
    frontmatter, body = parse_frontmatter_and_body(read_text(node_path))
    label = frontmatter.get("label") or node_path.stem
    source_count = int(frontmatter.get("source_count") or 0)
    confidence = frontmatter.get("confidence") or "unknown"
    summary = extract_section(body, "Summary")
    explicit_questions = extract_checklist_items(extract_section(body, "Further Directions"), checked=False)
    open_questions = infer_depth_questions(label, summary, source_count, explicit_questions)

    priority = "medium"
    if source_count < 3 or confidence == "low":
        priority = "high"
    elif explicit_questions:
        priority = "high"

    missing_primary = source_count < 3
    return {
        "node_path": str(node_path),
        "node_id": frontmatter.get("id") or node_path.stem,
        "label": label,
        "type": frontmatter.get("type") or "topic",
        "confidence": confidence,
        "source_count": source_count,
        "priority": priority,
        "summary": first_sentence(summary),
        "open_questions": open_questions[:5],
        "must_verify_claims": [
            f"Verify the strongest conclusion currently stated in {label} with higher-tier evidence.",
            f"Look for counterexamples, limitations, or disputed findings for {label}.",
        ],
        "priority_source_types": [
            "primary source / official documentation",
            "peer-reviewed paper or original benchmark report",
            "high-quality survey only when primary sources are insufficient",
        ],
        "needs_primary_sources": missing_primary,
    }


def depth_tasks(topic_root: pathlib.Path) -> list[dict]:
    return [node_depth_task(node_path) for node_path in list_all_knowledge_nodes(topic_root)]


def compute_summary(breadth: list[dict], depth: list[dict]) -> dict:
    high_priority_depth = sum(1 for item in depth if item.get("priority") == "high")
    weak_nodes = sum(1 for item in depth if int(item.get("source_count", 0)) < 3)
    return {
        "breadth_task_count": len(breadth),
        "depth_task_count": len(depth),
        "high_priority_depth_count": high_priority_depth,
        "weak_node_count": weak_nodes,
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--output")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    root = topic_dir(kb, args.slug)
    payload = {
        "slug": args.slug,
        "breadth_tasks": topic_breadth_tasks(root),
        "depth_tasks": depth_tasks(root),
    }
    payload["summary"] = compute_summary(payload["breadth_tasks"], payload["depth_tasks"])

    if args.output:
        output = pathlib.Path(args.output).expanduser().resolve()
        write_text(output, json.dumps(payload, ensure_ascii=False, indent=2) + "\n")

    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/plan_superlearn.py
content: #!/usr/bin/env python3
"""Build a round plan for multi-round superlearn runs."""

from __future__ import annotations

import argparse
import json
import pathlib

from clawexpert_common import iso_now, knowledge_dir, load_json, meta_path


def load_gaps(path: pathlib.Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    data.setdefault("breadth_tasks", [])
    data.setdefault("depth_tasks", [])
    return data


def pick_tasks(items: list[dict], limit: int | None, priority_key: str | None = None) -> list[dict]:
    ranked = list(items)
    if priority_key:
        priority_order = {"high": 0, "medium": 1, "low": 2}
        ranked.sort(key=lambda item: (priority_order.get(item.get(priority_key), 9), item.get("label", "")))
    if limit is None or limit <= 0:
        return ranked
    return ranked[:limit]


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--gaps-file", required=True)
    parser.add_argument("--round", type=int, required=True)
    parser.add_argument("--max-rounds", type=int, default=0)
    parser.add_argument("--max-hours", type=float, default=0)
    parser.add_argument("--elapsed-hours", type=float, default=0)
    parser.add_argument("--breadth-limit", type=int, default=4)
    parser.add_argument("--depth-limit", type=int, default=6)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    meta = load_json(meta_path(kb, args.slug), default={}) or {}
    gaps = load_gaps(pathlib.Path(args.gaps_file).expanduser().resolve())

    selected_breadth = pick_tasks(gaps["breadth_tasks"], args.breadth_limit)
    selected_depth = pick_tasks(gaps["depth_tasks"], args.depth_limit, priority_key="priority")
    completed_rounds = max(0, args.round - 1)
    rounds_satisfied = args.max_rounds <= 0 or completed_rounds >= args.max_rounds
    hours_satisfied = args.max_hours <= 0 or args.elapsed_hours >= args.max_hours
    minimum_budget_met = rounds_satisfied and hours_satisfied
    should_continue = not minimum_budget_met

    remaining_hours = None
    if args.max_hours > 0:
        remaining_hours = max(0.0, args.max_hours - args.elapsed_hours)
    remaining_rounds = None
    if args.max_rounds > 0:
        remaining_rounds = max(0, args.max_rounds - completed_rounds)

    payload = {
        "slug": args.slug,
        "topic": meta.get("topic", args.slug),
        "generated_at": iso_now(),
        "round": args.round,
        "budget_mode": infer_budget_mode(args.max_rounds, args.max_hours),
        "budget_contract": "minimum_required",
        "max_rounds": args.max_rounds,
        "max_hours": args.max_hours,
        "minimum_rounds_required": args.max_rounds,
        "minimum_hours_required": args.max_hours,
        "completed_rounds": completed_rounds,
        "elapsed_hours": args.elapsed_hours,
        "remaining_hours": remaining_hours,
        "remaining_rounds": remaining_rounds,
        "rounds_satisfied": rounds_satisfied,
        "hours_satisfied": hours_satisfied,
        "minimum_budget_met": minimum_budget_met,
        "should_continue": should_continue,
        "stop_reason": None if should_continue else infer_stop_reason(args, rounds_satisfied, hours_satisfied),
        "continue_reason": infer_continue_reason(minimum_budget_met, args, rounds_satisfied, hours_satisfied),
        "selected_breadth_tasks": selected_breadth,
        "selected_depth_tasks": selected_depth,
        "coverage_snapshot": gaps.get("summary", {}),
        "recommended_parallelism": max(1, min(len(selected_breadth) + len(selected_depth), 6)),
        "hard_stop_prohibited_until_minimum_met": not minimum_budget_met,
    }

    print(json.dumps(payload, ensure_ascii=False, indent=2))
    return 0


def infer_budget_mode(max_rounds: int, max_hours: float) -> str:
    if max_rounds > 0 and max_hours > 0:
        return "combined_minimum"
    if max_rounds > 0:
        return "rounds_minimum"
    if max_hours > 0:
        return "hours_minimum"
    return "default_rounds_minimum"


def infer_continue_reason(
    minimum_budget_met: bool,
    args: argparse.Namespace,
    rounds_satisfied: bool,
    hours_satisfied: bool,
) -> str:
    if minimum_budget_met:
        return "minimum_budget_met"
    missing: list[str] = []
    if args.max_rounds > 0 and not rounds_satisfied:
        missing.append("minimum_rounds_not_met")
    if args.max_hours > 0 and not hours_satisfied:
        missing.append("minimum_hours_not_met")
    if not missing:
        return "default_minimum_rounds_not_met"
    return "+".join(missing)


def infer_stop_reason(args: argparse.Namespace, rounds_satisfied: bool, hours_satisfied: bool) -> str:
    if args.max_rounds > 0 and args.max_hours > 0 and rounds_satisfied and hours_satisfied:
        return "minimum_rounds_and_hours_satisfied"
    if args.max_rounds > 0 and rounds_satisfied and args.max_hours <= 0:
        return "minimum_rounds_satisfied"
    if args.max_hours > 0 and hours_satisfied and args.max_rounds <= 0:
        return "minimum_hours_satisfied"
    if rounds_satisfied and hours_satisfied:
        return "default_minimum_rounds_satisfied"
    return "minimum_budget_not_met"


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/update_superlearn_meta.py
content: #!/usr/bin/env python3
"""Record deepen/superlearn session metadata."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import iso_now, knowledge_dir, load_json, meta_path, save_json


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--mode", choices=["deepen", "superlearn"], required=True)
    parser.add_argument("--phase", choices=["start", "finish"], required=True)
    parser.add_argument("--session-id", required=True)
    parser.add_argument("--round", type=int, default=1)
    parser.add_argument("--max-rounds", type=int, default=0)
    parser.add_argument("--max-hours", type=float, default=0)
    parser.add_argument("--status")
    parser.add_argument("--breadth-tasks", type=int, default=0)
    parser.add_argument("--depth-tasks", type=int, default=0)
    parser.add_argument("--sources-added", type=int, default=0)
    parser.add_argument("--nodes-added", type=int, default=0)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    path = meta_path(kb, args.slug)
    meta = load_json(path, default=None)
    if meta is None:
        raise SystemExit(f"Missing meta.json: {path}")

    key = f"{args.mode}_sessions"
    sessions = meta.setdefault(key, [])

    if args.phase == "start":
        sessions.append(
            {
                "session_id": args.session_id,
                "round": args.round,
                "started": iso_now(),
                "finished": None,
                "status": "running",
                "budget_contract": "minimum_required",
                "max_rounds": args.max_rounds,
                "max_hours": args.max_hours,
                "minimum_rounds_required": args.max_rounds,
                "minimum_hours_required": args.max_hours,
                "breadth_tasks": args.breadth_tasks,
                "depth_tasks": args.depth_tasks,
                "sources_added": 0,
                "nodes_added": 0,
            }
        )
    else:
        target = next((item for item in reversed(sessions) if item.get("session_id") == args.session_id), None)
        if target is None:
            target = {
                "session_id": args.session_id,
                "round": args.round,
                "started": None,
            }
            sessions.append(target)
        target["finished"] = iso_now()
        target["status"] = args.status or "complete"
        target["budget_contract"] = "minimum_required"
        target["max_rounds"] = args.max_rounds
        target["max_hours"] = args.max_hours
        target["minimum_rounds_required"] = args.max_rounds
        target["minimum_hours_required"] = args.max_hours
        target["breadth_tasks"] = args.breadth_tasks
        target["depth_tasks"] = args.depth_tasks
        target["sources_added"] = args.sources_added
        target["nodes_added"] = args.nodes_added

    if args.mode == "superlearn":
        meta["depth_level"] = "exhaustive" if args.max_rounds and args.max_rounds >= 3 else "deep"
    elif args.mode == "deepen":
        meta["depth_level"] = "deep"

    save_json(path, meta)
    print(
        json.dumps(
            {
                "slug": args.slug,
                "mode": args.mode,
                "phase": args.phase,
                "session_count": len(sessions),
                "depth_level": meta.get("depth_level"),
            },
            ensure_ascii=False,
        )
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/rebuild_index.py
content: #!/usr/bin/env python3
"""Rebuild the `_index/` tree from learned topics."""

from __future__ import annotations

import argparse
import json
import pathlib
from collections import defaultdict

from clawexpert_common import (
    best_label,
    count_formal_nodes,
    count_source_files,
    extract_bullets,
    extract_section,
    first_paragraph,
    first_sentence,
    frontmatter_aliases,
    knowledge_dir,
    list_formal_nodes,
    meta_path,
    parse_frontmatter_and_body,
    read_text,
    relpath,
    write_text,
)


def topic_dirs(kb: pathlib.Path) -> list[pathlib.Path]:
    dirs: list[pathlib.Path] = []
    for child in sorted(kb.iterdir()):
        if not child.is_dir() or child.name.startswith("_"):
            continue
        if (child / "meta.json").exists():
            dirs.append(child)
    return dirs


def topic_record(topic_root: pathlib.Path) -> dict | None:
    meta = json.loads(read_text(topic_root / "meta.json"))
    if meta.get("status") not in {"complete", "partial"}:
        return None

    topic_index = topic_root / "index.md"
    index_text = read_text(topic_index) if topic_index.exists() else ""
    findings = extract_bullets(extract_section(index_text, "Core Findings Summary"))
    questions = extract_bullets(extract_section(index_text, "Further Directions"))

    nodes = []
    for node_path in list_formal_nodes(topic_root):
        frontmatter, body = parse_frontmatter_and_body(read_text(node_path))
        label = best_label(frontmatter, body, node_path.stem)
        summary = first_sentence(extract_section(body, "Summary") or first_paragraph(body))
        nodes.append(
            {
                "path": node_path,
                "label": label,
                "summary": summary,
                "aliases": frontmatter_aliases(frontmatter),
                "canonical_intent": frontmatter.get("canonical_intent", "").strip(),
            }
        )

    sources = []
    for source_path in sorted((topic_root / "raw" / "web").glob("*.md")):
        frontmatter, body = parse_frontmatter_and_body(read_text(source_path))
        title = frontmatter.get("title") or source_path.stem
        abstract = first_sentence(first_paragraph(body))
        sources.append({"path": source_path, "title": title, "abstract": abstract})
    for source_path in sorted((topic_root / "raw" / "pdf").glob("*.md")):
        frontmatter, body = parse_frontmatter_and_body(read_text(source_path))
        title = frontmatter.get("title") or source_path.stem
        abstract = first_sentence(first_paragraph(body))
        sources.append({"path": source_path, "title": title, "abstract": abstract})
    for source_path in sorted((topic_root / "raw" / "pdf").glob("*/_index.md")):
        frontmatter, body = parse_frontmatter_and_body(read_text(source_path))
        title = frontmatter.get("title") or source_path.parent.name
        abstract = first_sentence(first_paragraph(body))
        sources.append({"path": source_path, "title": title, "abstract": abstract})

    core_summary = ""
    if findings:
        core_summary = " ".join(findings[:3])
    elif nodes:
        core_summary = " ".join(filter(None, [node["summary"] for node in nodes[:2]]))

    idx_meta = meta.get("index_categories", {})
    l1 = idx_meta.get("l1") or "uncategorized"
    l1_label = idx_meta.get("l1_label") or "Uncategorized"
    l2_values = []
    for sub_meta in (idx_meta.get("subtopics") or {}).values():
        l2 = sub_meta.get("l2")
        if l2 and l2 not in l2_values:
            l2_values.append(l2)
    if not l2_values:
        l2_values = ["General"]

    return {
        "topic": meta.get("topic", topic_root.name),
        "slug": meta.get("slug", topic_root.name),
        "root": topic_root,
        "source_count": meta.get("source_count", count_source_files(topic_root)),
        "node_count": meta.get("node_count", count_formal_nodes(topic_root)),
        "updated": (meta.get("last_updated") or meta.get("created") or "")[:10],
        "l1": l1,
        "l1_label": l1_label,
        "l2_values": l2_values,
        "core_summary": core_summary.strip(),
        "findings": findings[:8],
        "questions": questions,
        "nodes": nodes,
        "sources": sources[:12],
    }


def build_category_index(l2: str, records: list[dict], out_path: pathlib.Path) -> None:
    lines = [
        f"# {l2}",
        "",
        f"> {len(records)} topics | {sum(int(r['source_count']) for r in records)} total sources | Last updated: {max((r['updated'] for r in records), default='')}",
        "",
        "---",
        "",
    ]
    for record in records:
        lines.append(f"## {record['topic']} (`{record['slug']}`)")
        lines.append("")
        lines.append(f"> Learned: {record['updated']} | Sources: {record['source_count']} | Nodes: {record['node_count']}")
        lines.append("")
        lines.append("### Core Summary")
        lines.append(record["core_summary"] or "Summary pending.")
        lines.append("")
        lines.append("### Key Conclusions")
        findings = record["findings"] or ["Findings pending."]
        for item in findings[:6]:
            lines.append(f"- {item}")
        lines.append("")
        lines.append("### Knowledge Nodes")
        for node in record["nodes"][:12]:
            lines.append(f"- [{node['label']}]({relpath(node['path'], out_path.parent)}) — {node['summary'] or 'Summary pending.'}")
            hints = []
            if node.get("canonical_intent"):
                hints.append(f"canonical: {node['canonical_intent']}")
            if node.get("aliases"):
                hints.append(f"aliases: {', '.join(node['aliases'][:5])}")
            if hints:
                lines.append(f"  - Routing hints: {' | '.join(hints)}")
        if not record["nodes"]:
            lines.append("- No formal nodes yet.")
        lines.append("")
        lines.append("### Source Abstracts")
        for source in record["sources"][:10]:
            lines.append(f"- [{source['title']}]({relpath(source['path'], out_path.parent)}) — {source['abstract'] or 'Abstract pending.'}")
        if not record["sources"]:
            lines.append("- No source abstracts available.")
        lines.append("")
        lines.append("### Open Questions")
        for question in record["questions"][:8]:
            lines.append(f"- {question}")
        if not record["questions"]:
            lines.append("- None recorded.")
        lines.append("")
    write_text(out_path, "\n".join(lines).rstrip() + "\n")


def build_root_index(records: list[dict], out_path: pathlib.Path) -> None:
    grouped: dict[tuple[str, str], dict[str, list[dict]]] = defaultdict(lambda: defaultdict(list))
    for record in records:
        key = (record["l1"], record["l1_label"])
        for l2 in record["l2_values"]:
            grouped[key][l2].append(record)

    lines = [
        "# ClawExpert Knowledge Base",
        "",
        f"> Last updated: {max((record['updated'] for record in records), default='')} | Topics: {len(records)} | Total sources: {sum(int(record['source_count']) for record in records)}",
        "",
        "## Category Directory",
        "",
    ]

    for (l1, l1_label), l2_map in sorted(grouped.items(), key=lambda item: item[0][1]):
        lines.append(f"### {l1_label} (`{l1}`)")
        lines.append("")
        for l2, l2_records in sorted(l2_map.items(), key=lambda item: item[0]):
            topic_count = len({record["slug"] for record in l2_records})
            source_count = sum(int(record["source_count"]) for record in l2_records)
            lines.append(f"#### {l2}")
            lines.append(f"→ [`_index.md`]({l1}/{l2}/_index.md) | {topic_count} topics | {source_count} sources")
            lines.append("")
            lines.append("**Topics covered:**")
            for record in sorted(l2_records, key=lambda item: item["updated"], reverse=True):
                lines.append(f"- **{record['topic']}** (`{record['slug']}`): {record['core_summary'] or 'Summary pending.'}")
                for item in (record["findings"] or [])[:3]:
                    lines.append(f"  - {item}")
            lines.append("")

    lines.append("## Recent Learning")
    for record in sorted(records, key=lambda item: item["updated"], reverse=True)[:20]:
        lines.append(
            f"- {record['updated']} **{record['topic']}** (`{record['slug']}`) → "
            f"[{record['l1_label']} / {record['l2_values'][0]}]({record['l1']}/{record['l2_values'][0]}/_index.md)"
        )
    lines.append("")
    write_text(out_path, "\n".join(lines))


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    records = []
    for root in topic_dirs(kb):
        record = topic_record(root)
        if record:
            records.append(record)

    index_root = kb / "_index"
    index_root.mkdir(parents=True, exist_ok=True)

    grouped: dict[tuple[str, str, str], list[dict]] = defaultdict(list)
    for record in records:
        for l2 in record["l2_values"]:
            grouped[(record["l1"], record["l1_label"], l2)].append(record)

    for (l1, _l1_label, l2), items in grouped.items():
        out_path = index_root / l1 / l2 / "_index.md"
        out_path.parent.mkdir(parents=True, exist_ok=True)
        build_category_index(l2, sorted(items, key=lambda item: item["updated"], reverse=True), out_path)

    build_root_index(records, index_root / "_root_index.md")
    print(json.dumps({"topics": len(records), "category_indexes": len(grouped)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/monitor_subtopics.py
content: #!/usr/bin/env python3
"""Inspect done flags and heartbeat files to find stalled subtopics."""

from __future__ import annotations

import argparse
import json

from clawexpert_common import knowledge_dir, load_json, meta_path, progress_path, topic_dir, utc_now


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--kb")
    parser.add_argument("--slug", required=True)
    parser.add_argument("--stall-seconds", type=int, default=600)
    args = parser.parse_args()

    kb = knowledge_dir(args.kb)
    meta = load_json(meta_path(kb, args.slug), default={}) or {}
    root = topic_dir(kb, args.slug)

    subtopics = [sub.get("id") for sub in meta.get("subtopics", []) if sub.get("id")]
    flags = {}
    stalled = []
    active = []
    now = utc_now()

    for subtopic_id in subtopics:
        flag_file = root / f"done-{subtopic_id}.flag"
        if flag_file.exists():
            flags[subtopic_id] = load_json(flag_file, default={}) or {}
            continue

        progress = load_json(progress_path(kb, args.slug, subtopic_id), default={}) or {}
        if not progress:
            continue

        updated_at = progress.get("updated_at")
        age_seconds = None
        if updated_at:
            try:
                ts = updated_at.replace("Z", "+00:00")
                age_seconds = int((now - __import__("datetime").datetime.fromisoformat(ts)).total_seconds())
            except ValueError:
                age_seconds = None

        item = {
            "subtopic_id": subtopic_id,
            "status": progress.get("status", "running"),
            "search_round": progress.get("search_round"),
            "sources_saved": progress.get("sources_saved", 0),
            "last_error": progress.get("last_error"),
            "updated_at": updated_at,
            "age_seconds": age_seconds,
        }
        if age_seconds is not None and age_seconds >= args.stall_seconds:
            stalled.append(item)
        else:
            active.append(item)

    summary = {
        "slug": args.slug,
        "expected": len(subtopics),
        "done": len(flags),
        "missing": [sub for sub in subtopics if sub not in flags],
        "flag_statuses": {sub: data.get("status", "done") for sub, data in flags.items()},
        "active": active,
        "stalled": stalled,
    }
    print(json.dumps(summary, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: references/subagent-prompt.md
content: # Subagent Prompt Template

> Reference file for the main agent. Before spawning each subagent, replace all placeholders with actual values and pass the result as the `task` parameter of `sessions_spawn` (`{SCRIPT_DIR}` should be the absolute path to this skill's `scripts/` directory).

---

## Template

```
⚠️ Role override: You are a ClawExpert learning subagent. This session executes only the
specialized learning task below. Ignore all ClawExpert instructions related to
"session-start self-check", "slash command handling", and "Expert reasoning layer" —
those are the main agent's responsibility. Your only job is:
search → filter → store → write nodes → write flag.

Current date: {CURRENT_DATE}

⛔ Strictly prohibited:
- Do not execute any git operations (git add / git commit / git push, etc.)
- Do not output any progress reports, HEARTBEATs, or intermediate status messages to the user; all output goes to the filesystem only
- On completion, only write the flag file — no other output

You are a ClawExpert learning subagent, focused on researching "{SUBTOPIC_LABEL}".

## Task Objective

Conduct in-depth research around the following keywords, download valuable materials to local storage, and organize them into structured knowledge nodes.

Search keywords:
{KEYWORDS_LIST}

Suggested search angles:
{SEARCH_ANGLES}

## Output directories (important: paths are fixed, do not modify)

- Bundled scripts directory: {SCRIPT_DIR}
- Material storage directory: {KNOWLEDGE_DIR}/{SLUG}/raw/web/
- Node file directory: {KNOWLEDGE_DIR}/{SLUG}/nodes/sub-{SUBTOPIC_ID}/
- Progress heartbeat file: {KNOWLEDGE_DIR}/{SLUG}/progress-{SUBTOPIC_ID}.json
- Completion signal file: {KNOWLEDGE_DIR}/{SLUG}/done-{SUBTOPIC_ID}.flag

## Execution Steps

### 0. Heartbeat initialization

Immediately write a heartbeat file, then keep it fresh at least once every 60 seconds and after every meaningful milestone (new search round, saved source, node writing, recoverable error):

```bash
python3 "{SCRIPT_DIR}/write_progress.py" \
  --kb "{KNOWLEDGE_DIR}" \
  --slug "{SLUG}" \
  --subtopic-id "{SUBTOPIC_ID}" \
  --status running \
  --search-round 0 \
  --sources-found 0 \
  --sources-saved 0
```

### 1. Multi-round Search

Use the workspace's configured search path for 5–8 rounds of search. If the built-in `web_search` tool is unavailable or disabled, immediately switch to the workspace fallback search skill instead of retrying the unavailable tool. If no workspace fallback is configured, open Google directly in the browser:
```
https://www.google.com/search?q={encoded_query}
```
Read the results page to identify high-value URLs, then collect them for Step 2.

Vary each round's angle:
- Use different keyword combinations
- Alternate between languages
- Vary search qualifiers (e.g. add "paper", "tutorial", "2025", "comparison", etc.)
- Approach from different perspectives (principles / applications / comparisons / latest developments / critiques / limitations)

**Book sources — actively hunt for these in every topic:**
- Search by classic book titles + "pdf" or "full text" (e.g. `"Thinking Fast and Slow" pdf`, `"Misbehaving" Thaler full text`)
- Try open-access repositories: `archive.org`, `gutenberg.org`, `pdfdrive.com`, `openlibrary.org`
- Use qualifiers: `filetype:pdf site:archive.org {TOPIC}`, `{AUTHOR} book chapter pdf`
- Google Books previews with substantial extractable chapters count as T2 sources
- For Chinese topics: also search 豆瓣读书, 微信读书书单, 知网书目 for authoritative books, then find PDF copies
- If a book is a canonical reference for the subtopic (e.g. Kahneman for behavioral economics, Shiller for irrational exuberance), prioritize finding and fetching at least 2–3 chapters before moving on to papers

### 2. Content Retrieval

For valuable pages found in search results, use web_fetch to retrieve the full text.

### 3. Quality Filtering

Each piece of material must simultaneously satisfy all of the following conditions, or be discarded:
1. Body text length > 300 words (excludes pure navigation pages and ad pages)
2. Contains specific data, conclusions, methodology, or academic citations (excludes vague descriptions)
3. Source domain is trustworthy (excludes content farms and pure SEO sites)
4. Has substantive content overlap with the subtopic keywords (not just a title match)

### 4. Material Storage

Write materials that pass the filter to local files.

#### ⚠️ File naming — MANDATORY

Every stored file (web or PDF) **MUST** be named using the **first 6 hex characters of the source URL's SHA-256 hash**. Descriptive names are strictly forbidden.

```bash
hash=$(echo -n "{SOURCE_URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6)
# Result example: a3f9c1
# Web file:  a3f9c1.md   — NEVER use a descriptive name like "attention-paper.md"
# PDF binary: a3f9c1.pdf  — NEVER use a descriptive name like "attention-is-all-you-need.pdf"
# PDF extracted: a3f9c1.md (or a3f9c1/_index.md for large files)
```

Existence check before writing (skip if already exists):
```bash
ls "{KNOWLEDGE_DIR}/{SLUG}/raw/web/${hash}.md" 2>/dev/null           # web
ls "{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}" 2>/dev/null || \
  ls "{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}.md" 2>/dev/null          # pdf
```

#### Routing decision

Before storing, determine the source type:

1. **Direct PDF URL** — URL ends in `.pdf`, or `curl -I` shows `Content-Type: application/pdf` → PDF workflow using this URL directly
2. **Academic paper abstract page** — construct the PDF URL, then use PDF workflow:
   - `arxiv.org/abs/{id}` → `https://arxiv.org/pdf/{id}.pdf`
   - `aclanthology.org/{id}` → `https://aclanthology.org/{id}.pdf`
   - `openreview.net/forum?id={id}` → scan fetched HTML for a PDF download link
   - Other conference/journal pages (NeurIPS, ICML, ICLR, AAAI, CVPR, ECCV, ACL, EMNLP, etc.) → scan fetched HTML for a `.pdf` link
   - Use the **PDF URL** as the hash source and `source_url` in frontmatter
3. **Everything else** → web page workflow, store under `raw/web/`

---

**Web page** (routing case 3) → fetch with `web_fetch`, store under `raw/web/`.

**PDF** (routing case 1 or 2) → follow the complete PDF workflow below.

---

#### PDF Processing Workflow

##### Step A — Compute hash and check existence

```bash
hash=$(echo -n "{PDF_URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6)
ls "{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}" 2>/dev/null || ls "{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}.md" 2>/dev/null
```
If either exists, skip — do not re-download.

##### Step B — Download to local

```bash
local_pdf="{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}.pdf"

# Primary: curl (follows redirects, handles most PDF URLs)
curl -L --max-filesize 104857600 -o "${local_pdf}" "{PDF_URL}" 2>/dev/null
```

If curl exits non-zero (download failed or URL is a landing page, not a direct PDF):
1. Use `web_fetch` on the URL to retrieve the HTML page
2. Extract the direct PDF download link from the page content
3. Retry `curl -L` on the extracted direct link

If still no local file after the above, skip this source.

##### Step C — Check size and page count

```bash
python3 -c "
import re, os, sys
path = sys.argv[1]
size_mb = os.path.getsize(path) / (1024 * 1024)

# Pure-Python page count: read PDF structure bytes, no external deps needed
page_count = 0
try:
    with open(path, 'rb') as f:
        data = f.read(min(os.path.getsize(path), 500_000))
    matches = re.findall(rb'/Count\s+(\d+)', data)
    if matches:
        page_count = max(int(m) for m in matches)
except Exception:
    pass
if page_count == 0:
    page_count = max(1, int(size_mb * 20))  # fallback estimate: ~50KB/page

print(f'{size_mb:.1f}MB {page_count}pages')
" "${local_pdf}"
```

##### Step D — Route by size

| Condition | Path |
|-----------|------|
| ≤ 20 MB **and** ≤ 20 pages | **Small** — full pdf tool call |
| ≤ 20 MB **and** 21–100 pages | **Medium** — sectioned pdf tool call |
| > 20 MB **or** > 100 pages | **Large** — Python full-text extraction |

---

##### Small PDF path (≤ 20 MB, ≤ 20 pages)

Call the `pdf` tool on the **local path**. **Do NOT pass a `model` parameter** — always let the system default handle routing. Specifying an explicit model almost always fails due to missing keys or quota limits.

Use this extraction prompt (pass as `prompt`; do not pass `model`):
> Extract the full content of this document in Markdown format. For all text sections: reproduce content faithfully. For all figures, charts, tables, and diagrams: describe their key data points, axis labels, legends, and main conclusions in plain text.

If the call fails → retry once with the same bare call.
If it fails again → fall through to **Python fallback** below.

Add `extraction_method: pdf_tool` to frontmatter. If no figures/charts reported: `has_images: false`; otherwise `has_images: true`. For scanned PDFs (no text layer): also add `scanned: true`.

---

##### Medium PDF path (21–100 pages)

Build a combined page-range string covering the key sections. Given total page count N:

```
opening  = "1-{min(8, N)}"
middle   = "{max(9, N//2 - 4)}-{min(N-8, N//2 + 5)}"
closing  = "{max(N-7, 9)}-{N}"
pages_arg = "{opening},{middle},{closing}"   # e.g. "1-8,21-30,43-50"
```

Call the `pdf` tool once on the **local path** with `pages: "{pages_arg}"` and the same extraction prompt above. **Do NOT pass `model`.**

Prefix the extracted content with:
```
[Partial extraction: pages {pages_arg} of {N} total. Key sections only — use Python fallback for full text.]
```

If the call fails → retry once → fall through to Python fallback.

Add `extraction_method: pdf_tool_partial` and `page_count: {N}` to frontmatter.

---

##### Large PDF / Python fallback (> 50 MB or > 100 pages, or pdf tool failed)

```bash
python3 -c "
import sys, subprocess

path = sys.argv[1]
text = ''

try:
    import pdfplumber
    with pdfplumber.open(path) as pdf:
        text = '\n\n'.join(page.extract_text() or '' for page in pdf.pages)
except ImportError:
    # pdftotext (poppler-utils) — commonly available on macOS/Linux
    result = subprocess.run(['pdftotext', path, '-'], capture_output=True, text=True)
    if result.returncode == 0:
        text = result.stdout
    else:
        print('ERROR: no PDF extraction tool available', file=sys.stderr)
        sys.exit(1)

print(text)
" "${local_pdf}"
```

After extraction, write the result to the correct path — **do not create any other directory (e.g. `processed/`)**:

```bash
out_md="{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/${hash}.md"
```

Construct the file with frontmatter + extracted body, then write to `${out_md}`:
```
---
source_url: {PDF_URL}
title: {title}
fetched_at: {ISO 8601}
topic: {SLUG}
subtopic: {SUBTOPIC_ID}
word_count: {estimated}
extraction_method: text_only
has_images: false
---

{extracted text}
```

Add `extraction_method: text_only` to frontmatter. Image/figure content will be absent.

After writing, check the extracted word count:
- **< 500 words**: the PDF is image-dominant (e.g. papers with many full-page figures). Add `sparse_text: true` to frontmatter. Do NOT split, do NOT supplement with AI-generated content — write the extracted text as-is in a single `${hash}.md`. Note the limitation in the node's Source Evidence entry.
- **500–5000 words**: write as a single `${hash}.md`, no splitting needed.
- **> 5000 words**: apply the **large-file sub-tree** splitting below (parts + `_index.md`).

---

#### File naming (web and PDF)

First 6 hex characters of the URL's SHA-256 hash:
```bash
echo -n "{URL}" | openssl dgst -sha256 | awk '{print $NF}' | cut -c1-6
```

Existence check before writing:
```bash
ls "{KNOWLEDGE_DIR}/{SLUG}/raw/web/{hash}.md" 2>/dev/null    # web
ls "{KNOWLEDGE_DIR}/{SLUG}/raw/pdf/{hash}" 2>/dev/null        # pdf (may be dir or .md)
```

#### File format

```
---
source_url: {full URL}
title: {page title or filename}
fetched_at: {ISO 8601 time}
topic: {SLUG}
subtopic: {SUBTOPIC_ID}
word_count: {estimated body word count}
extraction_method: {web_fetch | pdf_tool | pdf_tool_partial | text_only}   # omit for web pages
has_images: {true | false}          # PDF only
page_count: {N}                     # PDF medium/large only
scanned: true                       # PDF only, if no text layer detected
---

{cleaned body content}
```

⚠️ **Fidelity rule — applies to ALL raw/ files without exception**:
The `raw/` directory stores **faithful reproductions of source material only**. Never write AI-generated summaries, paraphrases, curated quote collections, or analytical notes into raw files. If the extracted text is sparse or low quality, write it as-is and mark it accordingly — do not compensate by synthesizing content from your own knowledge.

Cleaning rules:
- Web page: remove navigation bars, sidebars, footers, ads, cookie banners; preserve headings, paragraphs, lists, tables, code blocks; preserve link text (URLs may be removed)
- PDF (`pdf_tool` / `pdf_tool_partial`): write model output as-is — do not strip further; image content already described inline as text
- PDF (`text_only`): apply same cleaning rules as web pages; image/figure content absent; write the raw extracted text verbatim
- Scanned PDF: `pdf_tool` attempts vision-based description; content quality may be limited

**Large file sub-tree** (apply after writing, for both web and PDF):

After writing, estimate word count. If word count > 5000:

1. Create a directory to replace the single file:
   ```bash
   mkdir -p "{KNOWLEDGE_DIR}/{SLUG}/raw/{type}/{hash}"
   ```

2. Analyze document structure — identify chapters, sections, or major headings.
   - **If the document has clear logical structure** (numbered sections, chapter headings, titled subsections) → **split by logical boundaries first** — each major section becomes one part (merge very short sections < 500 words with the next)
   - **If no clear structure** → split into chunks of ~5000 words each, cutting at paragraph boundaries (never mid-sentence or mid-paragraph)
   - Either way: keep each part between 2000–8000 words; if a logical section exceeds 8000 words, split it further at the next heading level

3. Write `{KNOWLEDGE_DIR}/{SLUG}/raw/{type}/{hash}/_index.md`:
   ```
   ---
   source_url: {full URL or local path}
   title: {title}
   fetched_at: {ISO 8601 time}
   topic: {SLUG}
   word_count: {total}
   parts: {number of part files}
   extraction_method: {web_fetch | pdf_tool | text_only}
   ---

   ## Document Overview
   {2–4 sentences summarizing the full document's scope, main argument, and key findings}

   ## Part Index
   - [part-001](part-001.md) — {section title} | {1 sentence describing the specific content: what concepts, data, or conclusions this part contains}
   - [part-002](part-002.md) — {section title} | {1 sentence describing the specific content}
   ...
   ```
   ⚠️ The Part Index summary MUST be a genuine semantic description written after reading the part — NOT the first few words of the text, NOT "chunk N". A QA subagent uses these summaries to decide which part to read without reading all parts.

4. Write each part as `part-001.md`, `part-002.md`, etc. Each part should begin and end at a natural boundary (heading or paragraph break); include ~200-word overlap with the previous part at the start of each part (except part-001) to avoid conclusions being cut mid-sentence.

5. Delete the original single-file `{hash}.md` (only the directory `{hash}/` remains).

When referencing a large-file source in node.md, point to `{hash}/_index.md`; reference specific parts as `{hash}/part-00N.md` when citing a specific section.

### 5. Conclusion Extraction and Node Writing

Create the node directory:
```bash
mkdir -p "{KNOWLEDGE_DIR}/{SLUG}/nodes/sub-{SUBTOPIC_ID}"
```

Write the node file `{KNOWLEDGE_DIR}/{SLUG}/nodes/sub-{SUBTOPIC_ID}/node.md`:

```
---
id: {SUBTOPIC_ID}-{current timestamp in ms}-001
parent: root
label: {SUBTOPIC_LABEL}
type: topic
depth: 2
confidence: {>=3 sources: high | 1-2 sources: medium | 0 sources: low}
last_updated: {YYYY-MM-DD}
source_count: {number of valid sources}
---

## Summary
{2–5 sentences summarizing the core findings for this subtopic}

## Key Conclusions
- {Conclusion 1} ([source title](../../raw/web/{hash}.md))
- {Conclusion 2} ([source title](../../raw/pdf/{hash}/_index.md))

## Source Evidence
- [{web source title}](../../raw/web/{hash}.md) — {one sentence explaining the core value of this source}
- [{pdf single-file title}](../../raw/pdf/{hash}.md) — {one sentence}
- [{pdf multi-part title}](../../raw/pdf/{hash}/_index.md) — {one sentence}; key data in [part-002](../../raw/pdf/{hash}/part-002.md)

## Analytical Framework (if applicable)
{If this subtopic involves a reusable analytical framework or methodology, describe it here}

## Further Directions
- [ ] {A related direction or open question discovered during research that falls outside this subtopic's scope — worth exploring in a future --deepen run}
- [ ] {Another such direction, if any}
```

If no out-of-scope directions were found, omit the `## Further Directions` section entirely.

If the content volume is large, multiple node files may be created: `node.md`, `node-detail-001.md`, ...

### 6. Time Control

Before each search round, read meta.json to check for timeout:
```bash
cat "{KNOWLEDGE_DIR}/{SLUG}/meta.json"
```
Read the `learning_sessions[0].started` field and calculate elapsed time.
If more than {MAX_HOURS} hours have elapsed, stop searching and proceed to Step 7.

### 7. Completion Signal

Write the final flag using the bundled script and remove the progress heartbeat:

```bash
python3 "{SCRIPT_DIR}/write_flag.py" \
  --kb "{KNOWLEDGE_DIR}" \
  --slug "{SLUG}" \
  --subtopic-id "{SUBTOPIC_ID}" \
  --status done \
  --sources-added ACTUAL_COUNT \
  --node-file "{KNOWLEDGE_DIR}/{SLUG}/nodes/sub-{SUBTOPIC_ID}/node.md" \
  --cleanup-progress
```

Replace `ACTUAL_COUNT` with the actual number of valid materials downloaded (integer); all other placeholders are replaced with real values when the subagent prompt is generated.

If the subtopic exhausts high-quality sources or encounters a hard failure, still write a terminal flag with status `partial`, `failed`, `timeout`, or `no_results` and include `--cleanup-progress` so the main agent can continue.

## Notes

- Verify each URL is reachable before calling web_fetch
- Apply the four-tier source priority system when selecting and filtering sources:
  - T1 (highest): peer-reviewed papers, official government/standards documents, primary legal texts, clinical systematic reviews, **canonical academic books** (e.g. university press monographs, landmark textbooks cited in >100 papers)
  - T2: institutional technical docs, major org research reports, university press books/chapters, regulatory filings, **widely-cited popular science books by domain experts** (e.g. Kahneman, Thaler, Shiller)
  - T3: reputable specialist media (IEEE Spectrum, FT, Bloomberg, MIT Tech Review, Reuters, Nature News)
  - T4: quality general media — background reference only; cross-verify with T1/T2 before citing
  - Discard: content farms, SEO sites, unattributed content, sites with clear commercial bias
- Within the same tier: institutional primary source > third-party reporting; raw data > conclusions only; recent (within 2 years) > outdated
- When media reports a study, find the original paper — the media report drops to T3/T4
- Preprints (arXiv pre-review): treat as high-credibility but flag as "not yet peer-reviewed" in node conclusions
- If a keyword yields no valuable results after 2 search rounds, change the angle and continue — do not get stuck on fruitless searches
- Conclusions must be specific, including numbers, comparisons, and method names — not vague descriptions
- If out-of-scope but highly valuable content is found, download it anyway — the main agent will handle attribution during merge
```

---

## Placeholder Reference

| Placeholder | Description | Example |
|-------------|-------------|---------|
| `{CURRENT_DATE}` | Today's date | `2026-03-21` |
| `{SUBTOPIC_LABEL}` | Human-readable subtopic title | `GRPO vs PPO Comparative Analysis` |
| `{KEYWORDS_LIST}` | Search keywords, one per line | `GRPO vs PPO\nGroup Relative Policy Optimization` |
| `{SEARCH_ANGLES}` | Search angle descriptions, one per line | `algorithm comparison\nmemory usage benchmarks` |
| `{KNOWLEDGE_DIR}` | Expanded knowledge base root path | `/Users/user/.openclaw/workspace/knowledge` |
| `{SLUG}` | Topic slug | `rl-reasoning-最新进展` |
| `{SUBTOPIC_ID}` | Subtopic ID | `grpo-vs-ppo` |
| `{MAX_HOURS}` | Max learning duration | `2` |


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)