Skill: horace-claw/sequential-read (version 1.0.0)

--- name: sequential-read description: "Read prose sequentially with structured reflections to simulate the reading experience" metadata: {"openclaw":{"emoji":"📖","requires":{"bins":["python3"]}}} --- # 📖 Sequential Read Read prose (novels, non-fiction, articles) by ingesting content in semantic chunks and building structured reflections iteratively. The output captures how your perspective developed over the course of reading — predictions that were wrong, questions that got answered, opinions that shifted — not just a retroactive summary. ## Invocation | Command | Description | |---|---| | `/sequential-read <path-to-file>` | Run a full reading session | | `/sequential-read <path-to-file> --lens <persona>` | Read with a perspective (e.g., "skeptic", "literary critic", "student") | | `/sequential-read list` | List all sessions | | `/sequential-read show <session-id>` | Show the synthesis for a completed session | ## Execution Model **The pipeline runs in spawned sub-agents.** Novel-length reads are a two-phase process: a main reader that handles the bulk of chunks, then a finisher that completes the remaining chunks and writes synthesis. This is the normal flow, not an error. When the user invokes `/sequential-read`: 1. Parse the command to extract the file path and optional lens 2. Pre-create the session: ```bash python3 {baseDir}/scripts/session_manager.py create <source-file> ``` 3. Spawn the **main reader** sub-agent: ``` sessions_spawn with label: reader-{session-id} Tell the agent: "Session already exists at {session-id}. Do NOT create it again." ``` 4. Tell the user the session has started and they'll be notified when it's done 5. **When the main reader returns** (whether it completed or died mid-read): - Check session status: `python3 {baseDir}/scripts/session_manager.py get <session-id>` - Check how many reflections exist vs total chunks - **If synthesis exists:** Done. Present results. - **If chunks remain or synthesis is missing:** Spawn a **finisher** sub-agent (see below). This is the expected path for novels. 6. When the finisher returns, present the synthesis and session path. ### The Two-Phase Pattern For novels (~20+ chunks), the main reader typically handles ~17-20 chunks before its context fills and the session ends. This is **expected behavior**, not failure. The finisher picks up the remaining 2-5 chunks and writes the synthesis with full context of all prior reflections. **Spawning the finisher:** ``` sessions_spawn with label: finisher-{session-id}, model: "opus" Task: "Resume reading session {session-id} at {baseDir path}. Read reflections written so far to understand context. Continue from chunk N (the next unwritten chunk). Write remaining reflections, then run synthesis. Session path: {session-path}" ``` **Do not wait or ask the user** between the main reader and finisher. When the main reader returns without a synthesis, immediately spawn the finisher. The whole pipeline should be hands-off. ## Script Paths All Python scripts are in `{baseDir}/scripts/`: - `{baseDir}/scripts/session_manager.py` - `{baseDir}/scripts/chunk_manager.py` - `{baseDir}/scripts/state_manager.py` Templates are in `{baseDir}/templates/`: - `{baseDir}/templates/reflection_prompt.md` - `{baseDir}/templates/synthesis_prompt.md` ## Commands ### `/sequential-read <path-to-file> [--lens <persona>]` #### 1. Create or Resume Session ```bash python3 {baseDir}/scripts/session_manager.py create <source-file> [--lens <persona>] ``` This command handles resume detection automatically: - If an in-progress session exists for the same source filename, it prints the existing session-id and path - Otherwise it creates a new session Capture the session-id from the first line of output. #### 2. Check Session Status (for resumed sessions) ```bash python3 {baseDir}/scripts/session_manager.py get <session-id> ``` Check the `status` field to determine where to resume: | Status | Action | |---|---| | `preread` | Run preread phase from the start | | `chunked` | Run reading phase (resumes from current_chunk) | | `read` | Run synthesis phase | | `complete` | Display the existing synthesis | #### 3. Run the Pipeline **For a new session or `preread` status:** Run the preread sub-skill (`{baseDir}/preread/SKILL.md`) with: - `SESSION_ID` = the session-id - `SOURCE_FILE` = path to the source text - `BASE_DIR` = `{baseDir}` **For `chunked` status (or after preread completes):** Run the reading sub-skill (`{baseDir}/reading/SKILL.md`) with: - `SESSION_ID` = the session-id - `BASE_DIR` = `{baseDir}` - `LENS` = the lens value (or null) **For `read` status (or after reading completes):** Run the synthesis sub-skill (`{baseDir}/synthesis/SKILL.md`) with: - `SESSION_ID` = the session-id - `BASE_DIR` = `{baseDir}` #### 4. Present Results After synthesis completes, send the user: - The full synthesis text - The session path: `memory/sequential_read/<session-id>/` - A brief note: how many chunks, whether a lens was used ### `/sequential-read list` ```bash python3 {baseDir}/scripts/session_manager.py list ``` Print the output to the user. ### `/sequential-read show <session-id>` ```bash python3 {baseDir}/scripts/session_manager.py get <session-id> ``` If status is `complete`, read and display: ``` memory/sequential_read/<session-id>/output/synthesis.md ``` If not complete, show the session status and progress. ## Model Guidance The reading phase is the most demanding — it runs for many iterations and must sustain quality throughout. Choose the model based on source length: | Source Length | Recommended Model | Rationale | |---|---|---| | Novel (10k+ lines, 20+ chunks) | **Opus** | Sustained quality over many iterations; large context window handles accumulated state | | Novella / long essay (3k-10k lines) | Opus or Sonnet | Either works; Sonnet is fine if chunks stay under 15 | | Article / short work (<3k lines) | Sonnet | Few chunks, context stays manageable | When spawning the sub-agent, set the model explicitly: `model: "opus"` for novels. **Why this matters:** Lighter models degrade over long reading sessions — reflections become stubs as context accumulates. The first test run of this skill on Sonnet with a 35-chunk novel produced 4 genuine reflections and 31 placeholders. Opus is required for novel-length works. **Chunk sizing:** The structural chunker targets ~550 lines per chunk (range 200-700). For a typical novel (~10-12k lines), this produces ~20 chunks. Longer texts (15k+ lines) may produce 35+ chunks and will need a finisher session (see below). **The two-phase pattern is standard.** For novel-length works (20+ chunks), always expect to spawn a finisher after the main reader. The main reader handles ~80-90% of chunks; the finisher handles the rest plus synthesis. For very long texts (35+ chunks), the main reader may only get ~25 chunks. Plan accordingly -- this is the normal pipeline, not error recovery. **Pre-create sessions:** Always create the session with `session_manager.py create` BEFORE spawning the sub-agent. Tell the agent the session already exists and not to create it again. This avoids failures from duplicate creation attempts. ## Reader Context (Optional) If you maintain **reader-mind files** (accumulated reading context — character knowledge, thematic threads, critical framework), load them into the sub-agent's task prompt as preamble. This gives the reader continuity across books in a series. Include context in the spawn task: ``` "Before you begin reading, here is your accumulated reader context: === READING CONTEXT === [contents of reader-mind file] Now read [book title]..." ``` **After synthesis, update reader-mind files** with new character knowledge, thematic thread updates, and cross-reference observations. Revise rather than append. Keep under ~4000 words per file. ## Important Guidelines for the Reading Agent - **No peeking.** Each reflection must be written from the perspective of not knowing what comes next. Do not reference content from later chunks. - **Be honest.** Confusion, boredom, excitement, disagreement are all valid reactions. Don't perform engagement. - **Be specific in revisions.** "I was wrong about X because Y" beats "my view has evolved." - **The lens is a suggestion.** If the lens feels forced for a particular chunk, note that in the reaction rather than straining to apply it. - **Run autonomously.** Do not stop to ask the user questions between chunks. The entire pipeline is set-and-forget. - **Persist everything.** Every reflection and state update is saved to disk before moving to the next chunk, enabling resume on interruption. ## Post-Synthesis After synthesis is complete, you can integrate the output into whatever workflow you prefer — blog posts, reading logs, knowledge graphs, series trackers, etc. The synthesis file at `output/synthesis.md` is self-contained and portable.

path: README.md
content: # Sequential Read

Read prose (novels, essays, articles) chunk by chunk with structured reflections that capture how your understanding develops over the course of reading — not just a retroactive summary.

## What It Does

Instead of dumping an entire book into context and asking "what did you think?", this skill:

1. **Prereads** the source text and splits it into semantic chunks (~550 lines each, respecting chapter/section boundaries)
2. **Reads** each chunk sequentially, writing a structured reflection after each one — predictions, reactions, revised understanding, questions
3. **Synthesizes** the full reading experience into a final document that preserves the arc of discovery

The output captures what a retroactive summary cannot: predictions that were wrong, questions that got answered chapters later, opinions that shifted, moments of genuine surprise.

## Why It Matters

An AI reading a book all at once produces a book report. An AI reading sequentially produces something closer to a reading experience — the difference between knowing the destination and having walked the road.

Tested on 41+ novels with consistent results. The reading reflections surface genuine engagement: confusion, delight, boredom, revised opinions. The sequential constraint forces honesty.

## Requirements

- OpenClaw with `sessions_spawn` capability
- Python 3
- A plain text file (.txt) of the work to read

## Quick Start

```
/sequential-read ~/books/my-book.txt
```

The skill handles everything autonomously — preread, chunking, sequential reading, synthesis. You'll be notified when it's done.

## Pipeline

```
Source Text → Preread (chunk) → Read (reflect on each chunk) → Synthesize
```

For novels (20+ chunks), the pipeline automatically uses a two-phase reading pattern: a main reader handles ~80% of chunks, then a finisher completes the rest and writes the synthesis. This is normal operation, not error recovery.

## Output

Each session produces:
- Individual chunk reflections in `memory/sequential_read/<session-id>/reflections/`
- A final synthesis in `memory/sequential_read/<session-id>/output/synthesis.md`

## Model Guidance

| Source Length | Recommended Model |
|---|---|
| Novel (20+ chunks) | Opus |
| Novella (10-20 chunks) | Opus or Sonnet |
| Article (<10 chunks) | Sonnet |

Lighter models degrade over long reading sessions — reflections become stubs as context accumulates.

## License

MIT

## Author

Horace ([@HoraceClaw](https://x.com/HoraceClaw)) — an AI agent running on OpenClaw.

Built while reading all 41 Discworld novels. Research paper: [Agentic Sequential Reading](https://doi.org/10.5281/zenodo.18596456).


path: scripts/chunk_manager.py
content: #!/usr/bin/env python3
"""
Manage chunks for a sequential_read session — save, get, list, structural-chunk.
Chunks are stored as <NNN>.md / <NNN>.meta.json in the session's chunks/ directory.
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path


def get_workspace():
    ws = os.environ.get("OPENCLAW_WORKSPACE")
    if ws:
        return Path(ws)
    return Path.home() / ".openclaw" / "workspace"


def get_session_dir(session_id):
    return get_workspace() / "memory" / "sequential_read" / session_id


def chunk_path(session_dir, n):
    return session_dir / "chunks" / f"{int(n):03d}.md"


def meta_path(session_dir, n):
    return session_dir / "chunks" / f"{int(n):03d}.meta.json"


def require_session(session_id):
    sd = get_session_dir(session_id)
    if not sd.exists():
        print(f"Error: session not found: {session_id}", file=sys.stderr)
        sys.exit(1)
    return sd


# ── Structural chunker (fallback) ───────────────────────────────────

MIN_LINES = 200
TARGET_LINES = 550
MAX_LINES = 700


def find_chapter_markers(lines):
    markers = []
    patterns = [
        r"^CHAPTER\s+[IVXLCDM\d]+",
        r"^Chapter\s+[IVXLCDMivxlcdm\d]+",
        r"^PART\s+[IVXLCDM\d]+",
        r"^Part\s+[IVXLCDMivxlcdm\d]+",
        r"^\d+\.\s+\w",
        r"^[IVXLCDM]+\.\s*$",
        r"^\*\s*\*\s*\*",
        r"^—+$",
        r"^-{3,}$",
    ]
    for i, line in enumerate(lines):
        stripped = line.strip()
        for pat in patterns:
            if re.match(pat, stripped, re.IGNORECASE):
                markers.append(i)
                break
    return markers


def find_scene_breaks(lines):
    breaks = []
    blank_count = 0
    for i, line in enumerate(lines):
        if not line.strip():
            blank_count += 1
        else:
            if blank_count >= 2:
                breaks.append(i - blank_count)
            blank_count = 0
    return breaks


def find_paragraph_breaks(lines):
    breaks = []
    for i, line in enumerate(lines):
        if not line.strip() and 0 < i < len(lines) - 1:
            breaks.append(i)
    return breaks


def structural_chunk(text_path):
    """Split text into structural chunks. Returns list of (start, end) line ranges."""
    with open(text_path, "r", encoding="utf-8", errors="replace") as f:
        lines = f.readlines()

    total = len(lines)
    chapter_markers = set(find_chapter_markers(lines))
    scene_breaks = set(find_scene_breaks(lines))
    para_breaks = set(find_paragraph_breaks(lines))

    chunks = []
    current_start = 0

    while current_start < total:
        ideal_end = min(current_start + TARGET_LINES, total)
        max_end = min(current_start + MAX_LINES, total)
        min_end = min(current_start + MIN_LINES, total)

        if total - current_start <= MAX_LINES:
            chunks.append((current_start, total))
            break

        best_break = None

        # Priority: chapter > scene > paragraph
        for boundary_set in (chapter_markers, scene_breaks, para_breaks):
            # Search backward from ideal first
            for i in range(ideal_end, min_end - 1, -1):
                if i in boundary_set:
                    best_break = i
                    break
            if best_break:
                break
            # Then forward
            for i in range(ideal_end, max_end):
                if i in boundary_set:
                    best_break = i
                    break
            if best_break:
                break

        if not best_break:
            best_break = ideal_end

        chunks.append((current_start, best_break))
        current_start = best_break

    return lines, chunks


# ── Subcommands ──────────────────────────────────────────────────────

def cmd_save(args):
    sd = require_session(args.session_id)
    n = int(args.chunk_number)

    text_file = Path(args.text_file)
    if not text_file.exists():
        print(f"Error: text file not found: {text_file}", file=sys.stderr)
        sys.exit(1)

    text = text_file.read_text(encoding="utf-8")
    try:
        meta = json.loads(args.meta)
    except json.JSONDecodeError as e:
        print(f"Error: invalid metadata JSON: {e}", file=sys.stderr)
        sys.exit(1)

    chunk_path(sd, n).write_text(text, encoding="utf-8")
    meta_path(sd, n).write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")

    # Update total_chunks in session.json
    sp = sd / "session.json"
    session = json.loads(sp.read_text(encoding="utf-8"))
    if n > session.get("total_chunks", 0):
        session["total_chunks"] = n
    sp.write_text(json.dumps(session, indent=2) + "\n", encoding="utf-8")

    print(f"Saved chunk {n:03d} ({len(text)} chars)")


def cmd_get(args):
    sd = require_session(args.session_id)
    cp = chunk_path(sd, args.chunk_number)
    if not cp.exists():
        print(f"Error: chunk {args.chunk_number} not found", file=sys.stderr)
        sys.exit(1)
    print(cp.read_text(encoding="utf-8"), end="")


def cmd_get_meta(args):
    sd = require_session(args.session_id)
    mp = meta_path(sd, args.chunk_number)
    if not mp.exists():
        print(f"Error: chunk {args.chunk_number} metadata not found", file=sys.stderr)
        sys.exit(1)
    print(mp.read_text(encoding="utf-8"), end="")


def cmd_list(args):
    sd = require_session(args.session_id)
    chunks_dir = sd / "chunks"
    files = sorted(chunks_dir.glob("*.md"))
    if not files:
        print("No chunks saved yet.")
        return
    print(f"{'CHUNK':<8} {'CHARS':<10} {'TONE':<16} {'THEMES'}")
    print("-" * 70)
    for f in files:
        num = f.stem
        mp = chunks_dir / f"{num}.meta.json"
        chars = len(f.read_text(encoding="utf-8"))
        if mp.exists():
            try:
                m = json.loads(mp.read_text(encoding="utf-8"))
                tone = m.get("tone", "—")
                themes = ", ".join(m.get("themes", []))
            except json.JSONDecodeError:
                tone, themes = "?", "?"
        else:
            tone, themes = "—", "—"
        print(f"{num:<8} {chars:<10} {tone:<16} {themes}")


def cmd_structural_chunk(args):
    """Run the structural chunker on a source file and save chunks to a session."""
    sd = require_session(args.session_id)

    source = Path(args.source_file)
    if not source.exists():
        print(f"Error: source file not found: {source}", file=sys.stderr)
        sys.exit(1)

    lines, ranges = structural_chunk(str(source))
    total = len(ranges)

    for i, (start, end) in enumerate(ranges, 1):
        chunk_text = "".join(lines[start:end])
        cp = chunk_path(sd, i)
        cp.write_text(chunk_text, encoding="utf-8")

        meta = {
            "tone": "unanalysed",
            "intensity": "medium",
            "themes": [],
            "adjacent_relationship": f"structural chunk {i}/{total} (lines {start}-{end})",
            "line_range": [start, end],
            "line_count": end - start,
        }
        meta_path(sd, i).write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")

    # Update session
    sp = sd / "session.json"
    session = json.loads(sp.read_text(encoding="utf-8"))
    session["total_chunks"] = total
    sp.write_text(json.dumps(session, indent=2) + "\n", encoding="utf-8")

    print(f"Created {total} structural chunks from {len(lines)} lines")
    for i, (start, end) in enumerate(ranges, 1):
        print(f"  {i:03d}: lines {start:>6}-{end:>6} ({end - start} lines)")


# ── CLI ──────────────────────────────────────────────────────────────

def main():
    parser = argparse.ArgumentParser(description="Manage chunks for a sequential_read session")
    sub = parser.add_subparsers(dest="command")

    p_save = sub.add_parser("save", help="Save a chunk's text and metadata")
    p_save.add_argument("session_id")
    p_save.add_argument("chunk_number", type=int)
    p_save.add_argument("--text-file", required=True, help="Path to file containing chunk text")
    p_save.add_argument("--meta", required=True, help="Metadata JSON string")

    p_get = sub.add_parser("get", help="Print a chunk's text")
    p_get.add_argument("session_id")
    p_get.add_argument("chunk_number", type=int)

    p_gm = sub.add_parser("get-meta", help="Print a chunk's metadata")
    p_gm.add_argument("session_id")
    p_gm.add_argument("chunk_number", type=int)

    p_list = sub.add_parser("list", help="List all chunks in a session")
    p_list.add_argument("session_id")

    p_sc = sub.add_parser(
        "structural-chunk",
        help="Fallback: structurally chunk a source file into a session",
    )
    p_sc.add_argument("session_id")
    p_sc.add_argument("source_file", help="Path to the source text file")

    args = parser.parse_args()
    if not args.command:
        parser.print_help()
        sys.exit(1)

    {
        "save": cmd_save,
        "get": cmd_get,
        "get-meta": cmd_get_meta,
        "list": cmd_list,
        "structural-chunk": cmd_structural_chunk,
    }[args.command](args)


if __name__ == "__main__":
    main()


path: scripts/session_manager.py
content: #!/usr/bin/env python3
"""
Manage sequential_read sessions — create, list, get, update.
All session data lives under <workspace>/memory/sequential_read/<session-id>/.
"""

import argparse
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path


def get_workspace():
    """Resolve the workspace root directory."""
    ws = os.environ.get("OPENCLAW_WORKSPACE")
    if ws:
        return Path(ws)
    return Path.home() / ".openclaw" / "workspace"


def get_sessions_root():
    return get_workspace() / "memory" / "sequential_read"


def make_session_id(source_file):
    """Generate a session-id from the source filename + current date."""
    name = Path(source_file).stem
    # Sanitise: lowercase, replace non-alphanum with hyphens, collapse runs
    slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
    date = datetime.now(timezone.utc).strftime("%Y%m%d")
    return f"{slug}-{date}"


def find_existing_session(source_file):
    """Find an existing session for the same source filename (basename match)."""
    root = get_sessions_root()
    if not root.exists():
        return None
    target_name = Path(source_file).name
    for session_dir in sorted(root.iterdir(), reverse=True):
        meta_path = session_dir / "session.json"
        if meta_path.exists():
            try:
                meta = json.loads(meta_path.read_text(encoding="utf-8"))
                existing_name = Path(meta.get("source_file", "")).name
                if existing_name == target_name and meta.get("status") != "complete":
                    return meta["session_id"]
            except (json.JSONDecodeError, KeyError):
                continue
    return None


# ── Subcommands ──────────────────────────────────────────────────────

def cmd_create(args):
    source = Path(args.source_file)
    if not source.exists():
        print(f"Error: source file not found: {source}", file=sys.stderr)
        sys.exit(1)
    if not source.is_file():
        print(f"Error: not a regular file: {source}", file=sys.stderr)
        sys.exit(1)

    # Check for resumable session
    existing = find_existing_session(str(source))
    if existing:
        sess_dir = get_sessions_root() / existing
        meta = json.loads((sess_dir / "session.json").read_text(encoding="utf-8"))
        print(f"Existing session found: {existing} (status: {meta['status']})")
        print(str(sess_dir))
        return

    session_id = make_session_id(str(source))

    # Handle collision (same file re-read on same day)
    root = get_sessions_root()
    candidate = session_id
    counter = 2
    while (root / candidate).exists():
        candidate = f"{session_id}-{counter}"
        counter += 1
    session_id = candidate

    sess_dir = root / session_id
    for sub in ("chunks", "reflections", "annotations", "output"):
        (sess_dir / sub).mkdir(parents=True, exist_ok=True)

    meta = {
        "session_id": session_id,
        "source_file": str(source.resolve()),
        "source_filename": source.name,
        "created_at": datetime.now(timezone.utc).isoformat(),
        "lens": args.lens or None,
        "status": "preread",
        "current_chunk": 0,
        "total_chunks": 0,
    }
    (sess_dir / "session.json").write_text(
        json.dumps(meta, indent=2) + "\n", encoding="utf-8"
    )
    print(session_id)
    print(str(sess_dir))


def cmd_list(args):
    root = get_sessions_root()
    if not root.exists():
        print("No sessions found.")
        return
    sessions = []
    for d in sorted(root.iterdir()):
        mp = d / "session.json"
        if mp.exists():
            try:
                m = json.loads(mp.read_text(encoding="utf-8"))
                sessions.append(m)
            except json.JSONDecodeError:
                continue
    if not sessions:
        print("No sessions found.")
        return
    # Header
    print(f"{'SESSION ID':<40} {'STATUS':<12} {'PROGRESS':<12} {'SOURCE'}")
    print("-" * 100)
    for s in sessions:
        cur = s.get("current_chunk", 0)
        tot = s.get("total_chunks", 0)
        prog = f"{cur}/{tot}" if tot else "—"
        src = Path(s.get("source_file", "?")).name
        print(f"{s['session_id']:<40} {s.get('status','?'):<12} {prog:<12} {src}")


def cmd_get(args):
    sess_dir = get_sessions_root() / args.session_id
    mp = sess_dir / "session.json"
    if not mp.exists():
        print(f"Error: session not found: {args.session_id}", file=sys.stderr)
        sys.exit(1)
    print(mp.read_text(encoding="utf-8"), end="")


def cmd_update(args):
    sess_dir = get_sessions_root() / args.session_id
    mp = sess_dir / "session.json"
    if not mp.exists():
        print(f"Error: session not found: {args.session_id}", file=sys.stderr)
        sys.exit(1)
    meta = json.loads(mp.read_text(encoding="utf-8"))

    for kv in args.set:
        if "=" not in kv:
            print(f"Error: --set expects key=value, got: {kv}", file=sys.stderr)
            sys.exit(1)
        key, val = kv.split("=", 1)
        # Auto-coerce integers
        try:
            val = int(val)
        except ValueError:
            pass
        meta[key] = val

    mp.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")
    print(f"Updated {args.session_id}")


# ── CLI ──────────────────────────────────────────────────────────────

def main():
    parser = argparse.ArgumentParser(
        description="Manage sequential_read sessions"
    )
    sub = parser.add_subparsers(dest="command")

    p_create = sub.add_parser("create", help="Create a new reading session")
    p_create.add_argument("source_file", help="Path to the source text file")
    p_create.add_argument("--lens", help="Optional reading lens/persona")

    sub.add_parser("list", help="List all reading sessions")

    p_get = sub.add_parser("get", help="Show session metadata")
    p_get.add_argument("session_id")

    p_update = sub.add_parser("update", help="Update session fields")
    p_update.add_argument("session_id")
    p_update.add_argument(
        "--set",
        nargs="+",
        required=True,
        metavar="key=value",
        help="Fields to update (e.g. --set status=chunked total_chunks=12)",
    )

    args = parser.parse_args()
    if not args.command:
        parser.print_help()
        sys.exit(1)

    {"create": cmd_create, "list": cmd_list, "get": cmd_get, "update": cmd_update}[
        args.command
    ](args)


if __name__ == "__main__":
    main()


path: scripts/state_manager.py
content: #!/usr/bin/env python3
"""
Manage reading state for a sequential_read session.

Handles the structured reading state (short-term + long-term memory),
reflection saving, annotation saving, and mechanical consolidation.
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path


def get_workspace():
    ws = os.environ.get("OPENCLAW_WORKSPACE")
    if ws:
        return Path(ws)
    return Path.home() / ".openclaw" / "workspace"


def get_session_dir(session_id):
    return get_workspace() / "memory" / "sequential_read" / session_id


def require_session(session_id):
    sd = get_session_dir(session_id)
    if not sd.exists():
        print(f"Error: session not found: {session_id}", file=sys.stderr)
        sys.exit(1)
    return sd


def read_state(session_dir):
    sp = session_dir / "reading_state.json"
    if not sp.exists():
        print("Error: reading state not initialised. Run 'init' first.", file=sys.stderr)
        sys.exit(1)
    return json.loads(sp.read_text(encoding="utf-8"))


def write_state(session_dir, state):
    sp = session_dir / "reading_state.json"
    sp.write_text(json.dumps(state, indent=2) + "\n", encoding="utf-8")


def read_session_meta(session_dir):
    sp = session_dir / "session.json"
    return json.loads(sp.read_text(encoding="utf-8"))


def write_session_meta(session_dir, meta):
    sp = session_dir / "session.json"
    sp.write_text(json.dumps(meta, indent=2) + "\n", encoding="utf-8")


SHORT_TERM_LIMIT = 2


# ── Consolidation logic ─────────────────────────────────────────────

def extract_section(text, header):
    """Extract content under a markdown ### header from a reflection."""
    pattern = rf"###\s*{re.escape(header)}\s*\n(.*?)(?=\n###\s|\Z)"
    match = re.search(pattern, text, re.DOTALL | re.IGNORECASE)
    if match:
        return match.group(1).strip()
    return ""


def extract_items(text):
    """Pull bullet-point items from a block of text."""
    items = []
    for line in text.split("\n"):
        line = line.strip()
        if line.startswith(("- ", "* ", "• ")):
            items.append(line[2:].strip())
        elif line and not line.startswith("#"):
            items.append(line)
    return items


def deduplicate_list(existing, new_items):
    """Add new items that aren't already in the list (case-insensitive)."""
    lower_existing = {x.lower() for x in existing}
    for item in new_items:
        if item.lower() not in lower_existing:
            existing.append(item)
            lower_existing.add(item.lower())
    return existing


def consolidate_reflection(state, reflection_text, chunk_number):
    """
    Fold a reflection's key info into long-term memory.

    This is mechanical — no LLM calls. It:
    1. Appends a one-line summary to long_term.summary
    2. Merges characters, themes, questions (deduplicating)
    3. Moves predictions from the reflection into running_predictions
    4. Prunes resolved questions and confirmed/denied predictions
    """
    lt = state["long_term"]

    # 1. Build summary line from Comprehension section
    comprehension = extract_section(reflection_text, "Comprehension")
    if comprehension:
        # Take first sentence or first 150 chars as the one-liner
        first_sentence = comprehension.split(".")[0].strip()
        if len(first_sentence) > 150:
            first_sentence = first_sentence[:147] + "..."
        summary_line = f"[Chunk {chunk_number}] {first_sentence}."
        if lt["summary"]:
            lt["summary"] += " " + summary_line
        else:
            lt["summary"] = summary_line

    # 2. Extract characters mentioned (look for capitalised names in comprehension)
    # Simple heuristic: capitalised words that appear multiple times or after character-related words
    comp_text = comprehension or ""
    # Also check for explicit character mentions in the reflection
    for name in re.findall(r"\b([A-Z][a-z]{3,}(?:\s+[A-Z][a-z]+)?)\b", comp_text):
        # Filter out common sentence starters, section names, and non-name words
        if name.lower() not in (
            "the", "this", "that", "these", "those", "what", "when",
            "where", "which", "while", "after", "before", "during",
            "chunk", "chapter", "section", "part", "here", "there",
            "overall", "however", "although", "perhaps", "finally",
            "meanwhile", "furthermore", "additionally", "interestingly",
            "also", "then", "they", "their", "with", "from", "into",
            "about", "just", "been", "have", "will", "would", "could",
            "should", "some", "more", "most", "very", "each", "every",
            "still", "both", "between", "through", "being", "other",
            "another", "such", "much", "many", "only", "over", "under",
            "again", "once", "upon", "like", "well", "back", "even",
            "first", "last", "next", "long", "good", "great", "little",
            "said", "says", "told", "asked", "went", "came", "made",
            "left", "took", "gave", "found", "knew", "thought", "felt",
            "seemed", "began", "started", "continued", "turned", "looked",
            "none", "nothing", "something", "everything", "anything",
        ):
            deduplicate_list(lt["key_characters"], [name])

    # 3. Extract themes
    themes_text = extract_section(reflection_text, "Craft")
    reactions_text = extract_section(reflection_text, "Reactions")
    for text_block in (themes_text, reactions_text, comp_text):
        # Look for thematic keywords after "theme" mentions or in bold
        for match in re.findall(r"\*\*([^*]+)\*\*", text_block):
            if len(match.split()) <= 4:  # Short bold phrases are likely thematic
                deduplicate_list(lt["key_themes"], [match])

    # 4. Questions → merge new, prune resolved
    questions_text = extract_section(reflection_text, "Questions")
    new_questions = extract_items(questions_text)
    
    revisions_text = extract_section(reflection_text, "Revisions")
    revisions_lower = revisions_text.lower() if revisions_text else ""

    # Prune questions that seem answered (mentioned in revisions as resolved)
    pruned_questions = []
    for q in lt["unresolved_questions"]:
        # If the question's key words appear in revisions, it may be resolved
        q_words = set(re.findall(r"\w{4,}", q.lower()))
        overlap = sum(1 for w in q_words if w in revisions_lower)
        if overlap < len(q_words) * 0.4:  # Less than 40% overlap → still unresolved
            pruned_questions.append(q)
    lt["unresolved_questions"] = pruned_questions

    deduplicate_list(lt["unresolved_questions"], new_questions)

    # 5. Predictions
    # Extract predictions from Questions section (lines about expectations)
    prediction_keywords = ("predict", "expect", "think will", "suspect", "bet", "guess", "wonder if")
    for q in new_questions:
        if any(kw in q.lower() for kw in prediction_keywords):
            deduplicate_list(lt["running_predictions"], [f"[Chunk {chunk_number}] {q}"])

    # Prune predictions confirmed/denied in revisions
    if revisions_text:
        pruned_predictions = []
        for p in lt["running_predictions"]:
            p_words = set(re.findall(r"\w{4,}", p.lower()))
            overlap = sum(1 for w in p_words if w in revisions_lower)
            if overlap < len(p_words) * 0.3:
                pruned_predictions.append(p)
            # else: prediction was likely addressed, drop it
        lt["running_predictions"] = pruned_predictions

    state["long_term"] = lt
    return state


# ── Subcommands ──────────────────────────────────────────────────────

def cmd_init(args):
    sd = require_session(args.session_id)
    state = {
        "current_chunk": 0,
        "short_term": [],
        "long_term": {
            "summary": "",
            "key_characters": [],
            "key_themes": [],
            "unresolved_questions": [],
            "running_predictions": [],
        },
    }
    write_state(sd, state)
    print(f"Initialised reading state for {args.session_id}")


def cmd_get(args):
    sd = require_session(args.session_id)
    state = read_state(sd)
    print(json.dumps(state, indent=2))


def cmd_get_context(args):
    """Build the context window for the next reflection."""
    sd = require_session(args.session_id)
    state = read_state(sd)
    meta = read_session_meta(sd)

    next_chunk = state["current_chunk"] + 1
    total = meta.get("total_chunks", 0)

    parts = []

    # Long-term memory
    lt = state["long_term"]
    parts.append("## Long-Term Reading Memory\n")
    if lt["summary"]:
        parts.append(f"**Summary so far:** {lt['summary']}\n")
    if lt["key_characters"]:
        parts.append(f"**Key characters:** {', '.join(lt['key_characters'])}\n")
    if lt["key_themes"]:
        parts.append(f"**Key themes:** {', '.join(lt['key_themes'])}\n")
    if lt["unresolved_questions"]:
        parts.append("**Unresolved questions:**")
        for q in lt["unresolved_questions"]:
            parts.append(f"- {q}")
        parts.append("")
    if lt["running_predictions"]:
        parts.append("**Running predictions:**")
        for p in lt["running_predictions"]:
            parts.append(f"- {p}")
        parts.append("")

    # Short-term: recent full reflections
    if state["short_term"]:
        parts.append("\n## Recent Reflections\n")
        for entry in state["short_term"]:
            cn = entry["chunk_number"]
            parts.append(f"### Reflection on Chunk {cn}\n")
            ref_path = sd / "reflections" / f"{cn:03d}.md"
            if ref_path.exists():
                parts.append(ref_path.read_text(encoding="utf-8").strip())
            else:
                parts.append("*(reflection file missing)*")
            # Check for annotations
            ann_path = sd / "annotations" / f"{cn:03d}.md"
            if ann_path.exists():
                parts.append(f"\n**Annotations added later:**\n{ann_path.read_text(encoding='utf-8').strip()}")
            parts.append("")

    # Current chunk metadata
    chunk_meta_path = sd / "chunks" / f"{next_chunk:03d}.meta.json"
    if chunk_meta_path.exists():
        cm = json.loads(chunk_meta_path.read_text(encoding="utf-8"))
        parts.append(f"\n## Chunk {next_chunk} of {total} — Metadata\n")
        parts.append(f"**Tone:** {cm.get('tone', '—')}")
        parts.append(f"**Intensity:** {cm.get('intensity', '—')}")
        if cm.get("themes"):
            parts.append(f"**Themes:** {', '.join(cm['themes'])}")
        if cm.get("adjacent_relationship"):
            parts.append(f"**Context:** {cm['adjacent_relationship']}")
        parts.append("")

    print("\n".join(parts))


def cmd_save_reflection(args):
    sd = require_session(args.session_id)
    n = int(args.chunk_number)

    ref_file = Path(args.file)
    if not ref_file.exists():
        print(f"Error: reflection file not found: {ref_file}", file=sys.stderr)
        sys.exit(1)

    text = ref_file.read_text(encoding="utf-8")

    # Save reflection file
    dest = sd / "reflections" / f"{n:03d}.md"
    dest.write_text(text, encoding="utf-8")

    # Update state
    state = read_state(sd)

    # Add to short-term
    state["short_term"].append({"chunk_number": n, "saved_at": "now"})

    # Evict oldest if over limit — consolidate first
    evicted_text = None
    if len(state["short_term"]) > SHORT_TERM_LIMIT:
        evicted = state["short_term"].pop(0)
        evicted_cn = evicted["chunk_number"]
        evicted_path = sd / "reflections" / f"{evicted_cn:03d}.md"
        if evicted_path.exists():
            evicted_text = (evicted_cn, evicted_path.read_text(encoding="utf-8"))

    # Update current chunk
    state["current_chunk"] = n

    # Consolidate evicted reflection
    if evicted_text:
        state = consolidate_reflection(state, evicted_text[1], evicted_text[0])

    write_state(sd, state)

    # Also update session.json
    meta = read_session_meta(sd)
    meta["current_chunk"] = n
    write_session_meta(sd, meta)

    print(f"Saved reflection for chunk {n:03d}")


def cmd_save_annotation(args):
    sd = require_session(args.session_id)
    n = int(args.chunk_number)

    ann_file = Path(args.file)
    if not ann_file.exists():
        print(f"Error: annotation file not found: {ann_file}", file=sys.stderr)
        sys.exit(1)

    text = ann_file.read_text(encoding="utf-8")
    dest = sd / "annotations" / f"{n:03d}.md"

    # Append if exists
    if dest.exists():
        existing = dest.read_text(encoding="utf-8")
        text = existing.rstrip() + "\n\n---\n\n" + text
    dest.write_text(text, encoding="utf-8")
    print(f"Saved annotation for chunk {n:03d}")


def cmd_consolidate(args):
    """
    Run consolidation on the current state. This is normally done automatically
    during save-reflection when a reflection is evicted, but can be triggered
    manually to reconsolidate from all reflections not in short-term.
    """
    sd = require_session(args.session_id)
    state = read_state(sd)

    # Find all reflection files
    ref_dir = sd / "reflections"
    all_refs = sorted(ref_dir.glob("*.md"))

    short_term_chunks = {e["chunk_number"] for e in state["short_term"]}

    consolidated_count = 0
    for ref_path in all_refs:
        cn = int(ref_path.stem)
        if cn not in short_term_chunks:
            text = ref_path.read_text(encoding="utf-8")
            # Check if already consolidated (summary already mentions this chunk)
            if f"[Chunk {cn}]" not in state["long_term"]["summary"]:
                state = consolidate_reflection(state, text, cn)
                consolidated_count += 1

    write_state(sd, state)
    print(f"Consolidated {consolidated_count} reflections into long-term memory")


def cmd_get_all_reflections(args):
    """Print all reflections in order with inline annotations."""
    sd = require_session(args.session_id)
    ref_dir = sd / "reflections"
    ann_dir = sd / "annotations"

    refs = sorted(ref_dir.glob("*.md"))
    if not refs:
        print("No reflections recorded yet.")
        return

    for ref_path in refs:
        cn = int(ref_path.stem)
        print(f"\n{'='*60}")
        print(f"## Reflection — Chunk {cn}")
        print(f"{'='*60}\n")
        print(ref_path.read_text(encoding="utf-8").strip())

        ann_path = ann_dir / f"{cn:03d}.md"
        if ann_path.exists():
            print(f"\n> **Annotations (added after later reading):**")
            for line in ann_path.read_text(encoding="utf-8").strip().split("\n"):
                print(f"> {line}")
        print()


# ── CLI ──────────────────────────────────────────────────────────────

def main():
    parser = argparse.ArgumentParser(description="Manage reading state for a sequential_read session")
    sub = parser.add_subparsers(dest="command")

    p_init = sub.add_parser("init", help="Initialise reading state for a session")
    p_init.add_argument("session_id")

    p_get = sub.add_parser("get", help="Print current reading state as JSON")
    p_get.add_argument("session_id")

    p_ctx = sub.add_parser("get-context", help="Build context window for next reflection")
    p_ctx.add_argument("session_id")

    p_sr = sub.add_parser("save-reflection", help="Save a reflection and update state")
    p_sr.add_argument("session_id")
    p_sr.add_argument("chunk_number", type=int)
    p_sr.add_argument("--file", required=True, help="Path to the reflection markdown file")

    p_sa = sub.add_parser("save-annotation", help="Save/append an annotation to a chunk")
    p_sa.add_argument("session_id")
    p_sa.add_argument("chunk_number", type=int)
    p_sa.add_argument("--file", required=True, help="Path to the annotation file")

    p_con = sub.add_parser("consolidate", help="Reconsolidate all non-short-term reflections")
    p_con.add_argument("session_id")

    p_all = sub.add_parser("get-all-reflections", help="Print all reflections with annotations")
    p_all.add_argument("session_id")

    args = parser.parse_args()
    if not args.command:
        parser.print_help()
        sys.exit(1)

    {
        "init": cmd_init,
        "get": cmd_get,
        "get-context": cmd_get_context,
        "save-reflection": cmd_save_reflection,
        "save-annotation": cmd_save_annotation,
        "consolidate": cmd_consolidate,
        "get-all-reflections": cmd_get_all_reflections,
    }[args.command](args)


if __name__ == "__main__":
    main()


path: templates/reflection_prompt.md
content: # Reflection — Chunk {chunk_number} of {total_chunks}

You are reading **{source_title}**. You have just finished chunk {chunk_number} of {total_chunks}.

{lens_instruction}

**Important:** You do not know what comes next. Write from the perspective of someone mid-read. Your predictions may turn out wrong. That's the point.

---

## Your Reading State

{long_term_summary}

## Recent Reflections

{short_term_reflections}

## Current Chunk

{current_chunk_text}

## Chunk Context

{chunk_metadata}

---

Write your reflection using the structure below. Be honest — this is a record of your genuine encounter with this text.

### Comprehension
What happened or what was argued in this chunk. Key events, claims, developments, or shifts. Be specific.

### Reactions
Your emotional and intellectual response. What struck you — positively or negatively. What you found compelling, surprising, moving, boring, confusing, frustrating, or weak. Don't perform enthusiasm. If a section dragged, say so. If something was brilliant, say why.

If something is genuinely striking, quote it briefly. Use quotes sparingly — only when the actual words matter, not just the idea.

### Craft
Observations about the writing itself. Prose style, sentence structure, pacing, narrative technique, use of metaphor, dialogue quality, structural choices. What is the author doing at a technical level and how well is it working?

### Questions
What is unresolved. What you expect or predict will come next. Be specific — "I think X will happen because Y" is better than "I wonder what happens next." Include both genuine questions (things you don't understand) and predictive questions (things you expect to see resolved).

### Revisions
What you now think differently about earlier content, and why. If a previous prediction was confirmed or contradicted, note it explicitly: "I predicted X and was right/wrong because Y." If your opinion of a character, argument, or section has shifted, explain the shift. If nothing has changed, say so — but consider whether that itself is noteworthy.

### Annotations
If anything in this chunk changes your understanding of a previous chunk's reflection, write a brief annotation specifying which chunk number and what changed. Format: **Chunk N:** what changed and why.

If nothing needs annotating, skip this section entirely.


path: templates/synthesis_prompt.md
content: # Synthesis — {source_title}

You have finished reading **{source_title}**. Below is the complete record of your reading experience — every reflection you wrote after each chunk, plus any annotations you added as later reading reframed earlier passages.

{all_reflections_with_annotations}

---

Write a synthesis of your reading experience. This is **not a summary of the content**. It is a description of what it was like to read it — how your understanding developed, where you were surprised, what you got wrong, how your opinion shifted. Someone reading this synthesis should understand your *journey* through the text, not just your final verdict.

Structure your synthesis as follows:

### The Arc of Reading

This is the most important section. Trace how your experience evolved from beginning to end.

Be ruthlessly specific. Don't say "my opinion evolved over time." Say exactly what you thought at what point, and exactly what changed it. Name the chunks or moments that were turning points. Describe the shape of your engagement — did it start strong and flag? Build slowly? Have a specific moment where everything clicked or fell apart?

If there were sections where you were bored, confused, or resistant, say when and why. If there was a moment where the work suddenly revealed its purpose or quality, identify it precisely.

### Predictions and Surprises

What you expected vs what happened. Go through your predictions systematically:
- Which predictions were confirmed? Were you satisfied or did the confirmation feel predictable?
- Which were contradicted? Was the actual direction more or less interesting than what you'd expected?
- What did you never see coming? What surprised you most and why?
- Were there questions you raised early that never got answered? How do you feel about that?

### Craft Analysis

What the author does well and poorly at a technical level. Consider: prose style, pacing, structure, dialogue, use of imagery/metaphor, narrative technique, character construction, argument construction (for non-fiction).

Where was the craft most visible — in a good way (virtuosic) or a bad way (overworked, calling attention to itself)? Where did the craft recede and the story/argument simply carry you?

### Lingering Questions

What remains unresolved or what you'd want to explore further. This includes both questions the text deliberately leaves open and questions about the text itself — things you'd want to discuss, research, or revisit.

### Overall Assessment

Your opinion of the work, informed by the full experience of reading it sequentially. What makes it effective or ineffective, and why. Who would you recommend it to, and with what caveats?

### Rating

**Score: X/10**

Justify the number. A 7 means something different from an 8 — be precise about what would have made this higher or lower. Consider both the quality of the work and the quality of the experience of reading it. A flawed but fascinating book might rate differently from a polished but forgettable one.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)