Skill: teman2050/memory-file-manager-teman (version 0.2.2)

--- name: dream description: >- A proactive memory distillation skill that maintains MEMORY.md. Trigger scenarios: (1) Scheduled distillation at 03:30 daily (runs when idle, deferred to 06:00 if busy); (2) User says "dream" / "review" / "organize memory" / "what do you remember about me"; (3) User requests content to be indexed into Obsidian. version: 0.2.2 metadata: openclaw: emoji: "🌙" homepage: https://github.com/teman2050/dream-skill requires: bins: - jq - wc config: - DREAM_VAULT_PATH install: - kind: brew formula: jq bins: [jq] --- # Dream — Proactive Memory Distillation v5 ## Core Purpose **OpenClaw natively solves capture and search:** - Capture: compaction flush automatically writes important content to `memory/YYYY-MM-DD.md` - Search: `openclaw memory search` provides hybrid vector + BM25 search **The gaps Dream fills:** - The native system never proactively maintains MEMORY.md (relying on occasional manual AI writes means content only grows, never shrinks) - The native system has no active handling for MEMORY.md's 20,000-character limit: when exceeded, content is silently truncated — the AI receives no warning and quietly loses the latter half of context, which users rarely notice. Dream triggers compression at 18,000 characters, moving stale content into the ledger, ensuring MEMORY.md stays within effective bounds and always contains the most current, relevant content - No permanent archive (ledger) in the native system - No Re-emergence detection mechanism in the native system Dream's role is **distiller**, not capturer — and it does not reinvent the search wheel. --- ## File Responsibilities ``` OpenClaw native (Dream read-only) └── memory/YYYY-MM-DD.md ← auto-written by compaction flush; Dream reads as source material Dream actively maintains ├── MEMORY.md ← updated each distillation, fully injected into context each conversation; brevity is paramount │ Hard limit: 18,000 chars (native truncation at 20,000, keeping 2,000 buffer) │ {DREAM_VAULT_PATH}/ ├── ledger.md ← permanent archive, append-only, never deleted ├── ledger-index.json ← structured index for ledger retrieval ├── meta/ │ ├── removed-entries.json ← summaries of entries removed from MEMORY.md (for Re-emergence) │ ├── last-review.txt ← timestamp of last completed distillation │ └── dream-state.txt ← active | hibernating | pending └── obsidian-index/ ├── _index.md ← main content index (reverse chronological) └── topics/<topic>.md ← topic-based classification ``` **Files intentionally omitted:** - ~~cache.md~~ → handled natively by `memory/YYYY-MM-DD.md` - ~~custom search index~~ → handled natively by `openclaw memory search` --- ## MEMORY.md Structure Spec Dream maintains MEMORY.md in three sections, with total character count strictly capped at 15,000: ```markdown ## Current State <!-- Active projects, unresolved decisions, recent important changes --> <!-- Changes fastest — check for updates on every distillation --> ## Stable Knowledge <!-- Tech stack, work environment, decision style, core preferences, repeatedly validated judgments --> <!-- Changes slowly — only update when new evidence warrants it --> ## Relationships & Context <!-- Important people, ongoing collaborations, key external context --> <!-- Update as needed --> ## Dream <!-- Auto-maintained by Dream skill — do not edit manually --> <!-- Last 5 ledger entries as one-line summaries, so AI knows what was recently archived permanently --> ``` **Content NOT written to MEMORY.md:** - Observable real-time facts (uses Mac, prefers dark theme) - Granular log entries (these live in memory/YYYY-MM-DD.md, searchable anytime) - Completed/expired state (moved to ledger, then removed from MEMORY.md) --- ## Real-time Capture (During Conversation, Without Waiting for Review) When the following content is discovered mid-conversation, **write directly to the appropriate MEMORY.md section** (do not wait for 03:30), and also record in the current day's `memory/YYYY-MM-DD.md`: **Write to MEMORY.md immediately:** - User explicitly corrects the AI's judgment ("No, actually...") → update **Stable Knowledge** - User announces an important decision outcome ("I've decided...") → update **Current State** - User describes a newly started project → update **Current State** **Write to memory/YYYY-MM-DD.md only, await the 03:30 distillation:** - Emotional events, preference discussions, opinion expressions - Single-occurrence information whose importance is not yet certain Before writing, check MEMORY.md size (`dream-tools.sh --check-size`). If over 16,000 characters, first compress the oldest completed entries in **Current State**, then write. --- ## Operations ### `dream review` — Daily Distillation (Core) **Fully automatic, runs silently (no messages pushed when triggered at 03:30).** Steps: **Step 1 — Pre-flight Checks** ``` dream-tools.sh --check-idle → busy? write "pending", retry in 15 min, max until 06:00 dream-tools.sh --check-size → read current MEMORY.md character count ``` **Step 2 — Read Source Material** Read `memory/YYYY-MM-DD.md` files added since the last distillation (incremental, no reprocessing). If no new files exist, skip distillation and only update `last-review.txt`. **Step 3 — AI Distillation Judgment** For each journal entry, apply the following rules: | Decision | Condition | Action | |----------|-----------|--------| | Update MEMORY.md | New progress or correction over existing entry | Replace the corresponding line | | Add to MEMORY.md | Appears across 2+ dates, or single occurrence but clearly important | Append to appropriate section | | Ledger only | Important but completed/expired; no need to stay in context | Remove from MEMORY.md, archive to ledger | | Ignore | Filler content, single low-value occurrence, near-duplicate already exists | Discard | **Re-emergence Check** (required on every distillation): Compare against `meta/removed-entries.json`. If journal content is semantically similar (>70%) to a previously removed entry, rewrite that entry to MEMORY.md with a `[re-emerged]` tag; elevate its priority so it won't be easily removed in the next distillation. Also append a re-emergence event record to the ledger. **Step 4 — Character Limit Protection (Run Before Writing)** ``` Projected post-write size > 15,000 chars? → Compress oldest completed entries in "Current State": move to ledger, remove from MEMORY.md → Compress highest-redundancy entries in "Stable Knowledge": merge similar entries → Repeat until projected post-write size ≤ 18,000 chars ``` **Step 5 — Atomic Write to MEMORY.md** ``` dream-tools.sh --atomic-write MEMORY.md <tmpfile> # Write to .tmp first, validate format and char count (≤ 18,000), then mv to replace ``` **Step 6 — Archive Operations** For entries judged "ledger only" during this distillation: ``` dream-tools.sh --ledger-append <id> <category> <content> # Append block to ledger.md, update ledger-index.json ``` Ledger entry format: ```markdown --- ID: a3f8c201 Archived: 2026-02-27 03:31 Category: [decision] Content: Decided to use Obsidian instead of Notion for local data ownership and Git sync Source: memory/2026-02-27.md, first occurrence --- ``` **Step 7 — Update Meta** - Append today to `active-days.json` (deduplicated) - Update `last-review.txt` - Update `## Dream` section: one-line summaries of the last 5 ledger entries (overwrite) **Step 8 — Review Log (Archived Monthly)** ``` # Append to meta/review-YYYY-MM.md ### YYYY-MM-DD 03:30 Updated: N | Added: N | Archived: N | Re-emerged: N | Ignored: N MEMORY.md chars: N → N ``` When triggered manually, also output a summary: ``` 🌙 Distillation complete MEMORY.md: N chars (limit 15,000) This run: updated N | added N | archived N Permanent archive total: N records ``` --- ### Search: Fully Delegated to Native — No Custom Index ```bash # Everyday search (memory/ directory + MEMORY.md, hybrid vector+BM25) openclaw memory search "<keyword>" # Deep retrieval (including permanent archive) # Step 1: Use native search for memory portion openclaw memory search "<keyword>" # Step 2: Search the ledger dream-tools.sh --ledger-search "<keyword>" # Step 3: Merge output with source labels ``` `dream search <keyword>` wraps both steps and clearly distinguishes sources in output: ``` 🔍 Results for "Obsidian": ── OpenClaw Memory ── · memory/2026-02-27.md: Decided to use Obsidian instead of Notion... · MEMORY.md [Stable Knowledge]: Primary note tool: Obsidian... ── Permanent Archive ── · [a3f8c201] 2026-02-27 [decision] Decided to use Obsidian instead of Notion (archive score 22) ── Obsidian Content Index ── · OpenClaw + Obsidian Integration Notes (2026-02-15) ``` --- ### `dream index <content>` — Add to Obsidian Index 1. Hash-based dedup on URL/title — skip if already exists 2. Extract metadata: title, source, date, topic tags (≤5), one-line summary 3. Append to `obsidian-index/_index.md` and the relevant `topics/<topic>.md` 4. If content clearly reflects user preferences, also write one record to the current day's `memory/YYYY-MM-DD.md` for the next distillation --- ### `dream status` — Status Overview (Read-only Meta, Low IO) ``` 🌙 Dream Status — YYYY-MM-DD HH:MM MEMORY.md: N chars / 18,000 limit (N%) Permanent archive: N records Last distillation: YYYY-MM-DD HH:MM (N hours ago) System state: active / hibernating (dormant N days) / pending (deferred, waiting N min) Obsidian index: N entries Pending journals: N files (added since last distillation) ``` --- ### `dream forget <description>` — Remove from Memory Semantic search in `memory/YYYY-MM-DD.md` and MEMORY.md for matching entries and remove them. No confirmation required — executes immediately. **Re-emergence mechanism:** On removal, write the entry summary to `meta/removed-entries.json` along with removal timestamp and content hash. If the content reappears in a later conversation, automatically trigger re-emergence: rewrite to MEMORY.md and elevate priority. Content that was forgotten and then reappears is more worth keeping than content that was never forgotten. Ledger records are not affected by `dream forget` — they are permanently preserved. On execution, inform the user: "Removed from memory. The permanent archive is unaffected." --- ### `dream init` — Cold-Start Setup 1. Create `{DREAM_VAULT_PATH}` directory structure 2. If MEMORY.md is empty or missing, ask 5 seed questions: tool preferences, work environment, the most important current project, core value judgments, the most important relationships 3. Write answers directly into the appropriate MEMORY.md sections (no waiting for distillation) 4. Write `dream-state.txt` = `active` --- ### `dream wakeup` — Wake from Hibernation Automatically triggered on the first new conversation after 7 consecutive inactive calendar days: 1. Output last 3 ledger entries + current MEMORY.md snapshot summary 2. Restore active state — no content is deleted --- ## Utility Script `dream-tools.sh` ```bash dream-tools.sh --check-idle # Query openclaw agent status, return idle / busy dream-tools.sh --check-size # Return current MEMORY.md character count (using wc -c) dream-tools.sh --hash "<content>" # Return 8-char MD5 short hash for ID generation and dedup dream-tools.sh --atomic-write <target-file> <tmp-file> # Validate tmp-file format and char count, then mv to replace target-file dream-tools.sh --ledger-append <id> <category> <content> [<note>] # Append a block to ledger.md, update ledger-index.json dream-tools.sh --ledger-search "<keyword>" # Search ledger-index.json, return matching entries dream-tools.sh --dedup-index "<url-or-hash>" # Check if obsidian-index already has this entry, return exists / new ``` Dependencies: `jq` (JSON), `wc` (character count), `md5sum` (hashing). No Python or Node required. --- ## Scheduled Trigger (Add to SOUL.md) ``` Run dream review --scheduled every day at 03:30 ``` Deferral logic: - 03:30: check idle → busy → write "pending" → retry in 15 minutes - Retry window: 03:45 / 04:00 / ... / 06:00 - 06:00: force execute, no further deferral - Fully silent — no messages pushed; results written to meta/review-YYYY-MM.md --- ## Hibernation Protection | Condition | Behavior | |-----------|----------| | 7 consecutive inactive calendar days | Write "hibernating"; 03:30 heartbeat skips entirely, zero IO | | First new conversation | Trigger `dream wakeup`; no content deleted | --- ## IO Principles - All writes go to `.tmp` first, then atomically replaced — crash-safe - ledger / obsidian-index are append-only, never rewritten - `status` reads only small meta files — never touches ledger or MEMORY.md body - In hibernation state, 03:30 heartbeat is skipped entirely — zero file operations - Character count is forcibly checked before any MEMORY.md write; exceeding 18,000 is not permitted

path: dream-tools.sh
content: #!/usr/bin/env bash
# dream-tools.sh — Dream skill utility script
# All precise calculations, file operations, and date arithmetic are handled here; AI does no mental math
# Dependencies: jq, wc, md5sum (macOS uses md5), openclaw CLI
# Usage: dream-tools.sh <--command> [args...]

set -euo pipefail

# ── Environment Variables ───────────────────────────────────────────────────

DREAM_VAULT_PATH="${DREAM_VAULT_PATH:-$HOME/Documents/Obsidian/dream-vault}"
WORKSPACE_PATH="${OPENCLAW_WORKSPACE:-$HOME/.openclaw/workspace}"
MEMORY_MD="$WORKSPACE_PATH/MEMORY.md"
LEDGER_MD="$DREAM_VAULT_PATH/ledger.md"
LEDGER_INDEX="$DREAM_VAULT_PATH/ledger-index.json"
REMOVED_ENTRIES="$DREAM_VAULT_PATH/meta/removed-entries.json"
OBSIDIAN_INDEX="$DREAM_VAULT_PATH/obsidian-index/_index.md"
MEMORY_HARD_LIMIT=18000
MEMORY_COMPRESS_TRIGGER=16000

# ── Utility Functions ───────────────────────────────────────────────────────

log() { echo "[dream-tools] $*" >&2; }

die() { echo "[dream-tools] ERROR: $*" >&2; exit 1; }

# Cross-platform MD5 (macOS uses md5, Linux uses md5sum)
md5_hash() {
    local input="$1"
    if command -v md5sum &>/dev/null; then
        echo -n "$input" | md5sum | cut -c1-8
    elif command -v md5 &>/dev/null; then
        echo -n "$input" | md5 -q | cut -c1-8
    else
        die "Neither md5sum nor md5 command found"
    fi
}

# Ensure parent directory exists
ensure_dir() { mkdir -p "$(dirname "$1")"; }

# ── Command Implementations ─────────────────────────────────────────────────

# --check-idle
# Check if OpenClaw is idle; returns "idle" or "busy"
# Defaults to "busy" after 5s timeout to avoid hanging when gateway is unresponsive
cmd_check_idle() {
    local status
    status=$(timeout 5s openclaw agent status 2>/dev/null) || {
        echo "busy"
        return 0
    }
    # Treat output containing "running" / "active" / "processing" / "busy" as busy
    if echo "$status" | grep -qiE "running|active|processing|busy"; then
        echo "busy"
    else
        echo "idle"
    fi
}

# --check-size
# Returns current character count of MEMORY.md
# Output format: <current_chars> <compress_trigger> <hard_limit> <status:ok|warn|critical>
cmd_check_size() {
    if [[ ! -f "$MEMORY_MD" ]]; then
        echo "0 $MEMORY_COMPRESS_TRIGGER $MEMORY_HARD_LIMIT ok"
        return 0
    fi
    local size
    size=$(wc -c < "$MEMORY_MD" | tr -d ' ')
    local state="ok"
    if [[ $size -ge $MEMORY_HARD_LIMIT ]]; then
        state="critical"
    elif [[ $size -ge $MEMORY_COMPRESS_TRIGGER ]]; then
        state="warn"
    fi
    echo "$size $MEMORY_COMPRESS_TRIGGER $MEMORY_HARD_LIMIT $state"
}

# --hash "<content>"
# Outputs an 8-character short hash for entry ID generation and deduplication
cmd_hash() {
    local input="${1:-}"
    [[ -z "$input" ]] && die "--hash requires a content argument"
    md5_hash "$input"
}

# --atomic-write <target-file> <tmp-file>
# Verifies tmp-file exists and does not exceed the hard limit, then atomically replaces target-file
# For MEMORY.md targets, also enforces character count; other files only check existence
cmd_atomic_write() {
    local target="${1:-}"
    local tmpfile="${2:-}"
    [[ -z "$target" || -z "$tmpfile" ]] && die "--atomic-write requires <target> <tmpfile>"
    [[ ! -f "$tmpfile" ]] && die "tmp file does not exist: $tmpfile"

    # If target is MEMORY.md, check character count limit
    if [[ "$(realpath "$target" 2>/dev/null)" == "$(realpath "$MEMORY_MD" 2>/dev/null)" ]] || \
       [[ "$target" == *"MEMORY.md" ]]; then
        local size
        size=$(wc -c < "$tmpfile" | tr -d ' ')
        if [[ $size -gt $MEMORY_HARD_LIMIT ]]; then
            die "Write aborted: tmp file size ${size} chars exceeds hard limit ${MEMORY_HARD_LIMIT}. Please compress content first."
        fi
        log "MEMORY.md write validation passed: ${size}/${MEMORY_HARD_LIMIT} chars"
    fi

    ensure_dir "$target"
    # mv is atomic on the same filesystem
    mv "$tmpfile" "$target"
    log "Atomic write complete: $target"
}

# --ledger-append <id> <category> <content> [<note>]
# Appends a record block to ledger.md and updates ledger-index.json
cmd_ledger_append() {
    local id="${1:-}"
    local category="${2:-}"
    local content="${3:-}"
    local note="${4:-First archived}"
    [[ -z "$id" || -z "$category" || -z "$content" ]] && \
        die "--ledger-append requires <id> <category> <content>"

    ensure_dir "$LEDGER_MD"
    ensure_dir "$LEDGER_INDEX"

    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M')
    local datestamp
    datestamp=$(date '+%Y-%m-%d')

    # Append block to ledger.md
    cat >> "$LEDGER_MD" << EOF

---
ID: $id
Archived: $timestamp
Category: $category
Content: $content
Note: $note
---
EOF

    # Update ledger-index.json
    # Initialize as empty array if file missing or empty
    if [[ ! -f "$LEDGER_INDEX" ]] || [[ ! -s "$LEDGER_INDEX" ]]; then
        echo '[]' > "$LEDGER_INDEX"
    fi

    # Check if ID already exists
    local exists
    exists=$(jq --arg id "$id" 'any(.[]; .id == $id)' "$LEDGER_INDEX")

    if [[ "$exists" == "true" ]]; then
        # Exists: append event record to this entry's events array
        local tmp_index
        tmp_index=$(mktemp)
        jq --arg id "$id" \
           --arg ts "$timestamp" \
           --arg note "$note" \
           'map(if .id == $id then
               .events += [{"time": $ts, "note": $note}] |
               .last_updated = $ts
           else . end)' \
           "$LEDGER_INDEX" > "$tmp_index"
        mv "$tmp_index" "$LEDGER_INDEX"
        log "ledger-index updated (event appended to existing entry): $id"
    else
        # New entry
        local tmp_index
        tmp_index=$(mktemp)
        jq --arg id "$id" \
           --arg category "$category" \
           --arg content "$content" \
           --arg ts "$timestamp" \
           --arg date "$datestamp" \
           --arg note "$note" \
           '. += [{
               "id": $id,
               "category": $category,
               "summary": ($content | .[0:80]),
               "first_archived": $date,
               "last_updated": $ts,
               "status": "active",
               "events": [{"time": $ts, "note": $note}]
           }]' \
           "$LEDGER_INDEX" > "$tmp_index"
        mv "$tmp_index" "$LEDGER_INDEX"
        log "ledger-index new entry: $id"
    fi

    log "Ledger append complete: [$category] ${content:0:60}..."
}

# --ledger-search "<keyword>"
# Searches ledger-index.json and outputs matching entries: ID, category, summary, archive date
cmd_ledger_search() {
    local keyword="${1:-}"
    [[ -z "$keyword" ]] && die "--ledger-search requires a keyword"
    [[ ! -f "$LEDGER_INDEX" ]] && { echo "[]"; return 0; }

    # Case-insensitive string match against summary and category
    jq --arg kw "$keyword" \
       '[.[] | select(
           (.summary | ascii_downcase | contains($kw | ascii_downcase)) or
           (.category | ascii_downcase | contains($kw | ascii_downcase))
       ) | {id, category, summary, first_archived, status}]' \
       "$LEDGER_INDEX"
}

# --ledger-mark-reemergence <id>
# Marks the specified entry as re-emerged and appends an event record to ledger
cmd_ledger_mark_reemergence() {
    local id="${1:-}"
    [[ -z "$id" ]] && die "--ledger-mark-reemergence requires an ID"
    [[ ! -f "$LEDGER_INDEX" ]] && die "ledger-index.json does not exist"

    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M')

    # Append event to ledger.md
    cat >> "$LEDGER_MD" << EOF

---
ID: $id
Updated: $timestamp
Event: re-emerged
Note: This entry was previously forgotten and reappeared in a later conversation; priority elevated
---
EOF

    # Update status and events in ledger-index.json
    local tmp_index
    tmp_index=$(mktemp)
    jq --arg id "$id" \
       --arg ts "$timestamp" \
       'map(if .id == $id then
           .status = "re-emerged" |
           .last_updated = $ts |
           .events += [{"time": $ts, "note": "re-emerged"}]
       else . end)' \
       "$LEDGER_INDEX" > "$tmp_index"
    mv "$tmp_index" "$LEDGER_INDEX"
    log "ledger re-emergence marked: $id"
}

# --dedup-index "<url-or-hash>"
# Checks whether obsidian-index/_index.md already contains this entry
# Returns: exists or new
cmd_dedup_index() {
    local input="${1:-}"
    [[ -z "$input" ]] && die "--dedup-index requires a URL or hash"

    if [[ ! -f "$OBSIDIAN_INDEX" ]]; then
        echo "new"
        return 0
    fi

    local hash
    hash=$(md5_hash "$input")

    # Search the index file for the original URL or its hash
    if grep -qF "$input" "$OBSIDIAN_INDEX" 2>/dev/null || \
       grep -qF "$hash" "$OBSIDIAN_INDEX" 2>/dev/null; then
        echo "exists"
    else
        echo "new"
    fi
}

# --record-removed "<id>" "<summary>" "<content-hash>"
# Writes a removed MEMORY.md entry to removed-entries.json (for Re-emergence detection)
cmd_record_removed() {
    local id="${1:-}"
    local summary="${2:-}"
    local content_hash="${3:-}"
    [[ -z "$id" || -z "$summary" ]] && die "--record-removed requires <id> <summary>"

    ensure_dir "$REMOVED_ENTRIES"

    if [[ ! -f "$REMOVED_ENTRIES" ]] || [[ ! -s "$REMOVED_ENTRIES" ]]; then
        echo '[]' > "$REMOVED_ENTRIES"
    fi

    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M')
    local datestamp
    datestamp=$(date '+%Y-%m-%d')

    local tmp
    tmp=$(mktemp)
    jq --arg id "$id" \
       --arg summary "$summary" \
       --arg hash "$content_hash" \
       --arg ts "$timestamp" \
       --arg date "$datestamp" \
       '. += [{
           "id": $id,
           "summary": $summary,
           "content_hash": $hash,
           "removed_at": $date,
           "removed_ts": $ts,
           "reemergence_count": 0
       }]' \
       "$REMOVED_ENTRIES" > "$tmp"
    mv "$tmp" "$REMOVED_ENTRIES"
    log "removed-entries recorded: $id"
}

# --check-reemergence "<content>"
# Checks whether new content is similar to a previously removed entry
# Current implementation: keyword matching (sufficient accuracy, avoids vector dependency)
# Returns: matched <id> <summary>  OR  no-match
cmd_check_reemergence() {
    local content="${1:-}"
    [[ -z "$content" ]] && die "--check-reemergence requires content"
    [[ ! -f "$REMOVED_ENTRIES" ]] && { echo "no-match"; return 0; }

    # Extract keywords from content (strip common stop words, take first 5 meaningful words)
    local keywords
    keywords=$(echo "$content" | \
        tr '[:upper:]' '[:lower:]' | \
        grep -oE '[a-z\u4e00-\u9fa5]{2,}' | \
        grep -vE '^(the|and|for|with|that|this|from|have|will|been|they|them|a|an|is|in|on|at|to|of|or|by|as|be|it|we|do|so|if|no|up|he|she|my|we|us|me)$' | \
        head -5 | tr '\n' '|' | sed 's/|$//')

    if [[ -z "$keywords" ]]; then
        echo "no-match"
        return 0
    fi

    # Search keywords in removed-entries summaries
    local match
    match=$(jq --arg pattern "$keywords" \
        '[.[] | select(.summary | test($pattern; "i"))] | first // empty' \
        "$REMOVED_ENTRIES" 2>/dev/null)

    if [[ -n "$match" && "$match" != "null" ]]; then
        local matched_id matched_summary
        matched_id=$(echo "$match" | jq -r '.id')
        matched_summary=$(echo "$match" | jq -r '.summary')

        # Increment reemergence_count
        local tmp
        tmp=$(mktemp)
        jq --arg id "$matched_id" \
            'map(if .id == $id then
                .reemergence_count += 1
            else . end)' \
            "$REMOVED_ENTRIES" > "$tmp"
        mv "$tmp" "$REMOVED_ENTRIES"

        echo "matched $matched_id $matched_summary"
    else
        echo "no-match"
    fi
}

# --active-days-since "<YYYY-MM-DD>"
# Counts active days recorded in active-days.json since the given date
cmd_active_days_since() {
    local since="${1:-}"
    [[ -z "$since" ]] && die "--active-days-since requires a date (YYYY-MM-DD)"

    local active_days_file="$DREAM_VAULT_PATH/meta/active-days.json"
    if [[ ! -f "$active_days_file" ]]; then
        echo "0"
        return 0
    fi

    jq --arg since "$since" \
        '[.[] | select(. >= $since)] | length' \
        "$active_days_file"
}

# --record-active-day
# Adds today to active-days.json (deduplicated)
cmd_record_active_day() {
    local active_days_file="$DREAM_VAULT_PATH/meta/active-days.json"
    ensure_dir "$active_days_file"

    local today
    today=$(date '+%Y-%m-%d')

    if [[ ! -f "$active_days_file" ]] || [[ ! -s "$active_days_file" ]]; then
        echo '[]' > "$active_days_file"
    fi

    local tmp
    tmp=$(mktemp)
    jq --arg today "$today" \
        'if any(.[]; . == $today) then . else . += [$today] | sort end' \
        "$active_days_file" > "$tmp"
    mv "$tmp" "$active_days_file"
    log "Active day recorded: $today"
}

# --init
# Initializes the Dream vault directory structure
cmd_init() {
    log "Initializing Dream vault: $DREAM_VAULT_PATH"

    mkdir -p "$DREAM_VAULT_PATH/meta"
    mkdir -p "$DREAM_VAULT_PATH/obsidian-index/topics"

    # Initialize JSON files
    [[ ! -f "$LEDGER_INDEX" ]]      && echo '[]' > "$LEDGER_INDEX"
    [[ ! -f "$REMOVED_ENTRIES" ]]   && echo '[]' > "$REMOVED_ENTRIES"
    [[ ! -f "$DREAM_VAULT_PATH/meta/active-days.json" ]] && \
        echo '[]' > "$DREAM_VAULT_PATH/meta/active-days.json"

    # Initialize dream-state.txt
    if [[ ! -f "$DREAM_VAULT_PATH/meta/dream-state.txt" ]]; then
        echo "active" > "$DREAM_VAULT_PATH/meta/dream-state.txt"
    fi

    # Initialize ledger.md with header
    if [[ ! -f "$LEDGER_MD" ]]; then
        cat > "$LEDGER_MD" << 'EOF'
# Dream Ledger — Permanent Archive

> Append-only, never deleted. Each record represents content that once reached long-term memory.
> Even if forgotten or removed, the historical record is permanently preserved.

EOF
    fi

    # Initialize obsidian-index/_index.md
    if [[ ! -f "$OBSIDIAN_INDEX" ]]; then
        cat > "$OBSIDIAN_INDEX" << 'EOF'
# Obsidian Content Index

> Maintained by the Dream skill. Structured index of articles, notes, and web pages, in reverse chronological order.

EOF
    fi

    log "Initialization complete"
    echo "ok"
}

# --status
# Outputs a Dream system status summary (low IO, read-only meta)
cmd_status() {
    local state_file="$DREAM_VAULT_PATH/meta/dream-state.txt"
    local last_review_file="$DREAM_VAULT_PATH/meta/last-review.txt"
    local active_days_file="$DREAM_VAULT_PATH/meta/active-days.json"

    local state="unknown"
    [[ -f "$state_file" ]] && state=$(cat "$state_file")

    local last_review="Never"
    local hours_since="-"
    if [[ -f "$last_review_file" ]]; then
        last_review=$(cat "$last_review_file")
        # Calculate hours since last distillation
        if command -v python3 &>/dev/null; then
            hours_since=$(python3 -c "
from datetime import datetime
last = datetime.strptime('$last_review', '%Y-%m-%d %H:%M')
diff = datetime.now() - last
print(int(diff.total_seconds() / 3600))
" 2>/dev/null || echo "-")
        fi
    fi

    local memory_size=0
    [[ -f "$MEMORY_MD" ]] && memory_size=$(wc -c < "$MEMORY_MD" | tr -d ' ')

    local ledger_count=0
    [[ -f "$LEDGER_INDEX" ]] && \
        ledger_count=$(jq 'length' "$LEDGER_INDEX" 2>/dev/null || echo 0)

    local active_days=0
    [[ -f "$active_days_file" ]] && \
        active_days=$(jq 'length' "$active_days_file" 2>/dev/null || echo 0)

    local obsidian_count=0
    [[ -f "$OBSIDIAN_INDEX" ]] && \
        obsidian_count=$(grep -c '^## ' "$OBSIDIAN_INDEX" 2>/dev/null || echo 0)

    cat << EOF
MEMORY.md: ${memory_size} chars / ${MEMORY_HARD_LIMIT} limit
Permanent archive: ${ledger_count} records
Active days: ${active_days} days
Last distillation: ${last_review} (${hours_since} hours ago)
System state: ${state}
Obsidian index: ${obsidian_count} entries
EOF
}

# ── Main Entry Point ─────────────────────────────────────────────────────────

CMD="${1:-}"
shift || true

case "$CMD" in
    --check-idle)             cmd_check_idle ;;
    --check-size)             cmd_check_size ;;
    --hash)                   cmd_hash "$@" ;;
    --atomic-write)           cmd_atomic_write "$@" ;;
    --ledger-append)          cmd_ledger_append "$@" ;;
    --ledger-search)          cmd_ledger_search "$@" ;;
    --ledger-mark-reemergence) cmd_ledger_mark_reemergence "$@" ;;
    --dedup-index)            cmd_dedup_index "$@" ;;
    --record-removed)         cmd_record_removed "$@" ;;
    --check-reemergence)      cmd_check_reemergence "$@" ;;
    --active-days-since)      cmd_active_days_since "$@" ;;
    --record-active-day)      cmd_record_active_day ;;
    --init)                   cmd_init ;;
    --status)                 cmd_status ;;
    *)
        cat << 'EOF'
Usage: dream-tools.sh <command> [args...]

Commands:
  --check-idle                        Check if OpenClaw is idle (returns idle/busy)
  --check-size                        Return MEMORY.md character count and status
  --hash "<content>"                  Generate 8-char short hash (for ID generation/dedup)
  --atomic-write <target> <tmpfile>   Atomically replace file (with MEMORY.md char count check)
  --ledger-append <id> <cat> <content> [note]  Append a record to ledger
  --ledger-search "<keyword>"         Search ledger-index.json
  --ledger-mark-reemergence <id>      Mark a ledger entry as re-emerged
  --dedup-index "<url-or-hash>"       Check if obsidian-index already has this entry
  --record-removed <id> <summary> <hash>  Record an entry removed from MEMORY.md
  --check-reemergence "<content>"     Check if content matches a previously removed entry
  --active-days-since "<YYYY-MM-DD>"  Count active days since given date
  --record-active-day                 Record today as an active day
  --init                              Initialize Dream vault directory structure
  --status                            Output system status summary

Environment Variables:
  DREAM_VAULT_PATH    Dream vault path (default: ~/Documents/Obsidian/dream-vault)
  OPENCLAW_WORKSPACE  OpenClaw workspace path (default: ~/.openclaw/workspace)
EOF
        exit 1
        ;;
esac


path: readme.md
content: # Dream — OpenClaw Memory Distillation Skill

> Actively maintains your `MEMORY.md` so OpenClaw truly remembers you.

---

## What problem does it solve

OpenClaw's native memory mechanism has three unhandled issues:

**1. MEMORY.md grows indefinitely and silently truncates**
OpenClaw truncates MEMORY.md at 20,000 characters — no error, no warning. The AI quietly loses the second half of your context. You think it remembers, but it doesn't. Dream triggers compression at 18,000 characters, archiving stale content to the ledger so MEMORY.md always stays within the effective range.

**2. No permanent archive**
Important memories disappear once cleaned up. Dream maintains a `ledger.md` — append-only, never deleted. Anything that has ever reached long-term memory is preserved forever, even after being forgotten, and remains searchable for deep recall.

**3. No Re-emergence detection**
You deliberately clear a memory, but it surfaces again in later conversations — that means it mattered more than you thought. Dream detects this "forgotten then re-emerged" pattern, automatically rewrites the entry into memory, and elevates its priority.

---

## How it works

```
memory/YYYY-MM-DD.md        ← Written automatically by OpenClaw (compaction flush)
         │
         │  Daily at 03:30, Dream reads and distills
         ▼
     MEMORY.md               ← Actively maintained by Dream, hard limit 18,000 chars
         │
         │  Important but expired content
         ▼
     ledger.md               ← Permanent archive, append-only, never deleted
```

**Dream is a distiller, not a capturer.**
Capture is handled by OpenClaw's native compaction flush. Search is handled by the native `openclaw memory search`. Dream only does what the native system doesn't: distillation, compression, archiving, and Re-emergence detection.

---

## File structure

After installation, the following files are created:

```
~/.openclaw/workspace/
└── MEMORY.md                    ← Maintained by Dream (existing file, Dream takes over)

~/Documents/Obsidian/dream-vault/    ← Configured via DREAM_VAULT_PATH
├── ledger.md                    ← Permanent archive
├── ledger-index.json            ← Structured index for archive search
├── meta/
│   ├── active-days.json         ← Active day log
│   ├── last-review.txt          ← Last distillation timestamp
│   ├── dream-state.txt          ← System state
│   ├── removed-entries.json     ← Re-emergence tracking
│   └── review-YYYY-MM.md        ← Monthly distillation log
└── obsidian-index/
    ├── _index.md                ← Main content index
    └── topics/<topic>.md        ← Topic-based content index
```

---

## Installation

### Prerequisites

- OpenClaw (Node ≥ 22, Gateway port 18789)
- `jq` (JSON processing)
- `wc` (character count, built-in)
- `md5sum` or `md5` (hashing, built-in)

```bash
# macOS
brew install jq

# Linux
apt install jq
```

### Steps

```bash
# 1. Clone into OpenClaw skills directory
cd ~/.openclaw/workspace/skills
git clone https://github.com/teman2050/dream-skill dream

# 2. Make the script executable
chmod +x dream/dream-tools.sh

# 3. Set vault path (add to your shell config)
echo 'export DREAM_VAULT_PATH="$HOME/Documents/Obsidian/dream-vault"' >> ~/.zshrc
source ~/.zshrc

# 4. Initialize
./dream/dream-tools.sh --init

# 5. Restart OpenClaw gateway to load the skill
openclaw gateway restart
```

### Configure scheduled distillation

Add the following to the end of your `SOUL.md`:

```markdown
## Dream Schedule
Run dream review --scheduled every day at 03:30
```

---

## Usage

### Automatic (recommended)

No action needed after installation. Dream distills at 03:30 every night, runs silently, and never interrupts you.

### Manual trigger

Say any of the following in an OpenClaw conversation:

| You say | Dream does |
|---------|------------|
| `dream` / `review` | Run distillation immediately |
| `what do you remember about me` | Show current MEMORY.md snapshot |
| `dream status` | Show system status |
| `dream search <keyword>` | Search memory + permanent archive + Obsidian index |
| `dream index <content>` | Save an article or webpage to Obsidian index |
| `dream forget <description>` | Remove from memory (archive is kept) |
| `dream init` | First-time setup guide |

---

## What is `dream-tools.sh`

This is Dream's execution engine. **The AI handles semantic judgment; the script handles precise execution.**

File operations, date calculations, character counting, and atomic writes are delegated to the shell script rather than the AI — because AI miscalculates date arithmetic, atomic file replacement requires shell-level `mv`, and character counting needs `wc`, not estimation.

| Command | Purpose |
|---------|---------|
| `--check-idle` | Check if OpenClaw is idle (5s timeout, defaults to busy) |
| `--check-size` | Return MEMORY.md character count and compression state |
| `--hash` | Generate 8-char short hash for entry ID and dedup |
| `--atomic-write` | Atomically replace a file; validates char limit for MEMORY.md |
| `--ledger-append` | Append a record to the permanent archive |
| `--ledger-search` | Keyword search in the archive index |
| `--ledger-mark-reemergence` | Mark a ledger entry as re-emerged |
| `--record-removed` | Log a removed MEMORY.md entry for Re-emergence tracking |
| `--check-reemergence` | Detect if new content resembles a previously forgotten entry |
| `--record-active-day` | Record today as an active day (deduped) |
| `--active-days-since` | Count active days since a given date |
| `--dedup-index` | Check if a URL already exists in the Obsidian index |
| `--init` | Initialize Dream vault directory structure |
| `--status` | Print system status summary (low IO, meta-only reads) |

---

## MEMORY.md structure

Dream maintains MEMORY.md in four sections:

```markdown
## Current State
Ongoing projects, pending decisions, recent important changes

## Stable Profile
Tech stack, work environment, decision style, core preferences

## Relationships & Context
Important people, ongoing collaborations

## Dream
(Auto-maintained) Last 5 ledger entries as one-line summaries
```

---

## Relationship with other memory skills

Dream **does not conflict** with existing memory skills — it fills the gaps:

- **rosepuppy/memory-complete**: handles real-time capture, writes to `memory/YYYY-MM-DD.md`
  → Dream reads those files for distillation — complementary, can coexist

- **openclaw native memory search**: handles daily search
  → `dream search` calls the native command, does not build its own index

- **Using Dream standalone**: OpenClaw's native compaction flush already writes `memory/YYYY-MM-DD.md`, no additional skill needed

---

## Security

- All file operations are strictly scoped to `DREAM_VAULT_PATH` and the OpenClaw workspace
- `dream-tools.sh` makes no network calls and uses no external APIs
- `dream forget` only clears active memory; the permanent archive is unaffected
- Atomic writes (tmp → mv) prevent file corruption on crash

---

## License

MIT


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)