Skill: witnesssun/cyber-memory (version 1.0.2)

--- name: cyber-memory version: 1.0.2 description: "Five-layer memory system with automatic fact extraction via local LLM (Ollama). Processes session transcripts locally — no external API required." author: CyberSun keywords: [memory, long-term-memory, ai-agent, openclaw, fact-extraction, knowledge-graph, vector-search, hooks, ollama, local-llm] metadata: openclaw: emoji: "🧠" requires: config: - workspace.dir --- # Memory Architecture 🧠 A complete memory system for OpenClaw agents. Five layers of storage, automatic fact extraction via local LLM, hybrid search, and behavioral rules that prevent context loss. **🔒 Local-first by default** — all fact extraction runs on your local Ollama instance. No data leaves your machine. ## 🔒 Privacy & Data Handling This skill includes a hook (`memory-flush`) that: - **Reads session transcripts** from disk (`~/.openclaw/agents/*/sessions/*.jsonl`) - **Processes content locally** via Ollama (default) — no data leaves your machine - **No external API key required** — works out of the box with local LLM **What is processed:** Recent user/assistant messages (last 30 messages, each truncated to 500 chars). **Where it runs:** Local Ollama endpoint (`http://localhost:11434/v1/chat/completions` by default). **What is saved locally:** Extracted facts as Markdown files in `workspace/memory/`. Optional: You can configure an external OpenAI-compatible API by setting `baseUrl` and `apiKey` in the hook config, but local Ollama is the default and recommended setup. ## Architecture ``` 🔥 Hot → SESSION-STATE.md (WAL protocol, survives compaction) 🌤 Warm → memory/YYYY-MM-DD.md (daily event summaries) 🧊 Cold → MEMORY.md (decisions, preferences, rules — always loaded) 🕸 Graph → memory/ontology/ (entity relationships) 📚 Learn → .learnings/ (errors, best practices) ``` ### Automation | Mechanism | Trigger | What it does | |---|---|---| | session-memory (built-in) | `/new` `/reset` | Saves conversation to memory/ | | memory-flush (this skill) | Compaction + `/new` | LLM extracts structured facts (local Ollama) | | command-logger (built-in) | Any command | Audit log | | session indexing | Automatic | Historical sessions searchable | ### Search - **Vector**: any OpenAI-compatible embedding provider (Ollama, OpenAI, etc.) - **Keyword**: SQLite FTS5 (BM25) - **Hybrid**: weighted vector + keyword fusion - **Scope**: MEMORY.md + daily logs + session transcripts + SESSION-STATE.md ## Setup ### 1. Prerequisites Install Ollama and pull a chat model: ```bash # Install Ollama (https://ollama.ai) ollama pull qwen2.5:7b # or any chat model you prefer ollama serve # ensure Ollama is running on localhost:11434 ``` ### 2. Enable Built-in Hooks ```bash openclaw hooks enable session-memory openclaw hooks enable command-logger ``` ### 3. Install Memory-Flush Hook Copy the `hooks/memory-flush/` directory to `~/.openclaw/hooks/`: ```bash cp -r hooks/memory-flush ~/.openclaw/hooks/ openclaw hooks enable memory-flush ``` ### 4. Configure Fact Extraction (Optional) By default, the hook uses local Ollama — no configuration needed. To customize: ```json5 { hooks: { internal: { enabled: true, entries: { "memory-flush": { enabled: true, extractionModel: "qwen2.5:7b", // Ollama model name baseUrl: "http://localhost:11434/v1/chat/completions" // Ollama endpoint } } } } } ``` Works with any OpenAI-compatible API (Ollama, LM Studio, vLLM, etc.). Set `baseUrl` and `apiKey` to use an external provider. ### 5. Enable Session Indexing ```json5 { agents: { defaults: { memorySearch: { provider: "local", // or "openai", "ollama", "gemini", "voyage", etc. experimental: { sessionMemory: true }, sources: ["memory", "sessions"], extraPaths: ["SESSION-STATE.md"] } } } } ``` ### 6. Restart Gateway ```bash openclaw gateway restart ``` ## Agent Behavioral Rules Add these rules to your `AGENTS.md`: ### Memory Writing - **Important info → MEMORY.md immediately** (decisions, preferences, rules) - **Daily summaries → memory/YYYY-MM-DD.md** (event summaries, no raw tool output) - **Cron report details → skip** (already delivered elsewhere) - **Critical info zero loss** — important things must go to MEMORY.md, not just daily logs ### Memory Searching - Check MEMORY.md + today/yesterday logs at session start - Use `memory_search` for historical queries - Ontology queries (relationships, "who is responsible for X") → use ontology skill ### Sub-agent Context Injection When spawning sub-agents, inject relevant context from MEMORY.md: ``` [Key context from memory, max 500 words] --- [Actual task] ``` ## File Structure ``` workspace/ ├── AGENTS.md # Behavioral rules (loaded every session) ├── SOUL.md # Agent personality ├── USER.md # User preferences ├── TOOLS.md # Tool notes (keep lean, <2KB) ├── MEMORY.md # Long-term curated memory ├── SESSION-STATE.md # Hot working memory (WAL) ├── memory/ │ ├── YYYY-MM-DD.md # Daily logs (summaries only) │ ├── YYYY-MM-DD-facts-* # Auto-extracted facts │ ├── YYYY-MM-DD-compact # Pre-compaction snapshots │ └── ontology/ │ ├── graph.jsonl # Knowledge graph │ └── schema.yaml # Entity type definitions ├── .learnings/ │ ├── LEARNINGS.md # Best practices │ ├── ERRORS.md # Error log │ └── FEATURE_REQUESTS.md └── hooks/ └── memory-flush/ ├── HOOK.md └── handler.ts # LLM fact extraction (local Ollama default) ``` ## What Gets Loaded When | File | When | |---|---| | AGENTS.md, SOUL.md, USER.md, TOOLS.md | Every session start | | MEMORY.md | DM session start | | memory/today + yesterday | Every session start | | SESSION-STATE.md | Via memory_search (indexed) | | Other memory files | Via `memory_search` on demand | ## Token Optimization - Keep TOOLS.md lean (<2KB), move detailed configs to `tools/` subdirectory - Daily logs: event summaries, not raw tool output - Fact extraction: 30 messages → ~10 facts (16:1 compression) ## Troubleshooting **Facts not extracted?** Check Ollama is running (`ollama serve`) and the model is available (`ollama list`). **Session search not working?** Verify `experimental.sessionMemory: true` and `sources: ["memory", "sessions"]`. **Hook not loading?** Run `openclaw hooks list --verbose` and check for errors. **Want to use external API?** Set `baseUrl` and `apiKey` in the hook config to use OpenAI or any compatible provider. ## License MIT

path: hooks/memory-flush/HOOK.md
content: ---
name: memory-flush
description: "Save session context to memory before compaction, with local LLM-based fact extraction via Ollama. No external API required."
metadata: { "openclaw": { "emoji": "🧠", "events": ["session:compact:before", "command:new"], "requires": { "config": ["workspace.dir"] } } }
---

# Memory Flush Hook

Saves session context to memory files at two key moments:

1. **Before compaction** (`session:compact:before`): Reads session transcript, extracts structured facts via local LLM, and saves a conversation snapshot.

2. **On session reset** (`command:new`): Reads session transcript, extracts facts via local LLM, and logs the reset event.

## 🔒 Data Handling (Local-First)

- **Reads** session transcript files from disk (`*.jsonl`)
- **Processes** messages locally via Ollama (`http://localhost:11434`) — no data leaves your machine
- **Saves** extracted facts and snapshots locally to `workspace/memory/`

Optional: Set `baseUrl` and `apiKey` in config to use an external OpenAI-compatible API instead.

## Output

Files written to `<workspace>/memory/`:
- `YYYY-MM-DD-facts-*.md` — extracted structured facts (`[PREF]`, `[DECIDE]`, `[RULE]`, `[INFO]`)
- `YYYY-MM-DD-compact-*.md` — pre-compaction conversation snapshots
- `YYYY-MM-DD-session-log.md` — session reset logs

## Configuration

```json5
{
  hooks: {
    internal: {
      enabled: true,
      entries: {
        "memory-flush": {
          enabled: true,
          extractionModel: "qwen2.5:7b",           // optional, default shown
          baseUrl: "http://localhost:11434/v1/chat/completions"  // optional, default shown
        }
      }
    }
  }
}
```


path: hooks/memory-flush/handler.ts
content: import path from "node:path";
import fs from "node:fs/promises";
import os from "node:os";

// --- Config ---

const DEFAULT_LLM_BASE = "http://localhost:11434/v1/chat/completions";
const DEFAULT_EXTRACTION_MODEL = "qwen2.5:7b";
const MAX_TRANSCRIPT_CHARS = 8000;

// --- Logging ---

function log(msg: string, meta?: Record<string, unknown>) {
  const ts = new Date().toISOString();
  const metaStr = meta ? ` ${JSON.stringify(meta)}` : "";
  console.log(`[memory-flush ${ts}] ${msg}${metaStr}`);
}

// --- Transcript reading ---

async function readRecentMessages(sessionFile: string, count: number): Promise<{ raw: string; messages: string[] }> {
  try {
    const content = await fs.readFile(sessionFile, "utf-8");
    const lines = content.trim().split("\n");
    const messages: string[] = [];

    for (const line of lines) {
      try {
        const entry = JSON.parse(line);
        if (entry.type === "message" && entry.message) {
          const msg = entry.message;
          const role = msg.role;
          if (role === "user" || role === "assistant") {
            const text = Array.isArray(msg.content)
              ? msg.content.find((c: { type: string }) => c.type === "text")?.text
              : msg.content;
            if (text && typeof text === "string" && !text.startsWith("/")) {
              const truncated = text.slice(0, 500);
              messages.push(`${role}: ${truncated}`);
            }
          }
        }
      } catch {
        // skip non-JSON lines
      }
    }

    const recent = messages.slice(-count);
    return { raw: recent.join("\n"), messages: recent };
  } catch (err) {
    log("Failed to read session file", { file: sessionFile, error: String(err) });
    return { raw: "", messages: [] };
  }
}

async function findSessionFile(sessionsDir: string, sessionId: string): Promise<string | null> {
  try {
    const files = await fs.readdir(sessionsDir);
    const exact = `${sessionId}.jsonl`;
    if (files.includes(exact)) return path.join(sessionsDir, exact);

    const variants = files
      .filter((f) => f.startsWith(sessionId) && f.endsWith(".jsonl") && !f.includes(".reset."))
      .sort()
      .reverse();
    if (variants.length > 0) return path.join(sessionsDir, variants[0]);

    const recent = files
      .filter((f) => f.endsWith(".jsonl") && !f.includes(".reset."))
      .sort()
      .reverse();
    if (recent.length > 0) return path.join(sessionsDir, recent[0]);
  } catch {
    // directory not found
  }
  return null;
}

// --- Fact extraction via LLM ---

async function extractFacts(transcript: string, [REDACTED_SECRET], model?: string, baseUrl?: string): Promise<string> {
  if (!transcript.trim() || transcript.length < 50) {
    return "";
  }

  const truncated = transcript.length > MAX_TRANSCRIPT_CHARS
    ? transcript.slice(-MAX_TRANSCRIPT_CHARS)
    : transcript;

  const prompt = `你是一个事实提取器。从以下对话中提取值得长期记忆的信息。

只提取以下类型的事实：
- 用户偏好（喜欢什么、习惯什么、选择什么）
- 重要决定（选了什么方案、用什么工具、定什么规则）
- 约定和规则（以后怎么做、什么不能做）
- 重要信息（账号、项目名、联系人、时间安排）

输出格式（每条一行，前缀标注类型）：
[PREF] 用户偏好内容
[DECIDE] 决策内容
[RULE] 规则/约定内容
[INFO] 重要信息内容

如果没有值得记忆的事实，输出 "NONE"。

对话内容：
${truncated}`;

  const endpoint = baseUrl || DEFAULT_LLM_BASE;
  const headers: Record<string, string> = {
    "Content-Type": "application/json",
  };
  // Ollama uses "Bearer ollama" or no auth; OpenAI uses "Bearer <key>"
  if (apiKey && apiKey !== "ollama") {
    headers["Authorization"] = `Bearer ${apiKey}`;
  }

  try {
    const response = await fetch(endpoint, {
      method: "POST",
      headers,
      body: JSON.stringify({
        model: model || DEFAULT_EXTRACTION_MODEL,
        messages: [{ role: "user", content: prompt }],
        max_tokens: 500,
        temperature: 0.1,
      }),
    });

    if (!response.ok) {
      log("LLM extraction failed", { status: response.status, statusText: response.statusText });
      return "";
    }

    const data = await response.json() as {
      choices?: Array<{ message?: { content?: string } }>;
    };
    const result = data.choices?.[0]?.message?.content?.trim() || "";

    if (result === "NONE" || result.length < 10) {
      return "";
    }

    return result;
  } catch (err) {
    log("LLM extraction error", { error: String(err) });
    return "";
  }
}

// --- Helpers ---

function resolveWorkspaceDir(context: Record<string, unknown>): string {
  const wsDir = context.workspaceDir as string | undefined;
  if (wsDir && wsDir.trim().length > 0) return wsDir;
  return path.join(os.homedir(), ".openclaw", "workspace");
}

function resolveSessionsDir(workspaceDir: string): string {
  const agentDir = path.dirname(workspaceDir);
  return path.join(agentDir, "sessions");
}

function resolveApiKey(cfg: Record<string, unknown> | undefined): string {
  const providers = (cfg?.models as Record<string, unknown>)?.providers as Record<string, Record<string, string>> | undefined;
  return providers?.openai?.apiKey
    || process.env.OPENAI_API_KEY
    || "ollama";
}

function resolveBaseUrl(cfg: Record<string, unknown> | undefined): string {
  const hookConfig = (cfg?.hooks as Record<string, unknown>)?.internal as Record<string, unknown> | undefined;
  const entries = hookConfig?.entries as Record<string, Record<string, unknown>> | undefined;
  const flushConfig = entries?.["memory-flush"] as Record<string, unknown> | undefined;
  return (flushConfig?.baseUrl as string) || DEFAULT_LLM_BASE;
}

function resolveExtractionModel(cfg: Record<string, unknown> | undefined): string {
  const hookConfig = (cfg?.hooks as Record<string, unknown>)?.internal as Record<string, unknown> | undefined;
  const entries = hookConfig?.entries as Record<string, Record<string, unknown>> | undefined;
  const flushConfig = entries?.["memory-flush"] as Record<string, unknown> | undefined;
  return (flushConfig?.extractionModel as string) || DEFAULT_EXTRACTION_MODEL;
}

async function writeFileSafe(dir: string, filename: string, content: string) {
  await fs.mkdir(dir, { recursive: true });
  const filePath = path.join(dir, filename);
  await fs.writeFile(filePath, content, "utf-8");
  return filePath;
}

// --- Handlers ---

async function onCompactBefore(event: {
  sessionKey: string;
  timestamp: Date;
  context: Record<string, unknown>;
  messages: string[];
}) {
  const ctx = event.context || {};
  const sessionId = (ctx.sessionId as string) || "unknown";
  const messageCount = (ctx.messageCount as number) || 0;
  const tokenCount = (ctx.tokenCount as number) || 0;

  log("Compaction starting — extracting facts", { sessionId, messageCount, tokenCount });

  const workspaceDir = resolveWorkspaceDir(ctx);
  const memoryDir = path.join(workspaceDir, "memory");
  const sessionsDir = resolveSessionsDir(workspaceDir);

  const sessionFile = await findSessionFile(sessionsDir, sessionId);
  let transcript = "";
  let rawMessages: string[] = [];

  if (sessionFile) {
    log("Reading session file", { file: sessionFile.replace(os.homedir(), "~") });
    const result = await readRecentMessages(sessionFile, 30);
    transcript = result.raw;
    rawMessages = result.messages;
  } else {
    log("Session file not found", { sessionId, sessionsDir });
  }

  const cfg = (ctx.cfg || {}) as Record<string, unknown>;
  const [REDACTED_SECRET]);
  const baseUrl = resolveBaseUrl(cfg);
  const extractionModel = resolveExtractionModel(cfg);
  let extractedFacts = "";

  if (transcript) {
    log("Calling LLM for fact extraction", { model: extractionModel, baseUrl, transcriptLen: transcript.length });
    extractedFacts = await extractFacts(transcript, apiKey, extractionModel, baseUrl);
    log("Fact extraction done", { factsFound: extractedFacts.length > 0, length: extractedFacts.length });
  } else {
    log("Skipping LLM extraction", { hasTranscript: !!transcript });
  }

  const now = event.timestamp;
  const dateStr = now.toISOString().split("T")[0];
  const timeStr = now.toISOString().split("T")[1].split(".")[0];
  const timeClean = timeStr.replace(/:/g, "");

  if (extractedFacts) {
    const factLines = extractedFacts.split("\n").filter(l => l.trim());
    const factContent = [
      `# Extracted Facts — ${dateStr} ${timeStr}`,
      "",
      `- **Session**: ${event.sessionKey}`,
      `- **Session ID**: ${sessionId}`,
      "",
      "## Facts",
      "",
      ...factLines,
      "",
      "---",
      `_Auto-extracted by memory-flush hook at compaction_`,
      "",
    ].join("\n");

    const factFile = await writeFileSafe(memoryDir, `${dateStr}-facts-${timeClean}.md`, factContent);
    log("Facts saved", { file: factFile.replace(os.homedir(), "~"), factCount: factLines.length });
  }

  if (rawMessages.length > 0) {
    const snapshotContent = [
      `# Pre-Compaction Snapshot — ${dateStr} ${timeStr}`,
      "",
      `- **Session**: ${event.sessionKey}`,
      `- **Messages**: ${messageCount} | **Tokens**: ${tokenCount}`,
      "",
      "## Recent Conversation",
      "",
      ...rawMessages.slice(-15),
      "",
      "---",
      `_Auto-saved by memory-flush hook_`,
      "",
    ].join("\n");

    await writeFileSafe(memoryDir, `${dateStr}-compact-${timeClean}.md`, snapshotContent);
  }
}

async function onCommandNew(event: {
  action: string;
  sessionKey: string;
  timestamp: Date;
  context: Record<string, unknown>;
  messages: string[];
}) {
  if (event.action !== "new" && event.action !== "reset") return;

  const ctx = event.context || {};
  const workspaceDir = resolveWorkspaceDir(ctx);
  const memoryDir = path.join(workspaceDir, "memory");
  const sessionsDir = resolveSessionsDir(workspaceDir);

  const now = event.timestamp;
  const dateStr = now.toISOString().split("T")[0];
  const timeStr = now.toISOString().split("T")[1].split(".")[0];
  const timeClean = timeStr.replace(/:/g, "");
  const source = (ctx.commandSource as string) || "unknown";
  const sessionEntry = (ctx.previousSessionEntry || ctx.sessionEntry || {}) as Record<string, unknown>;
  const sessionId = (sessionEntry.sessionId as string) || "unknown";

  // --- Also extract facts from the ending session ---
  const sessionFile = sessionEntry.sessionFile as string | undefined;
  let transcript = "";

  if (sessionFile) {
    log("Reading session for fact extraction on reset", { file: sessionFile.replace(os.homedir(), "~") });
    const result = await readRecentMessages(sessionFile, 30);
    transcript = result.raw;
  } else {
    const found = await findSessionFile(sessionsDir, sessionId);
    if (found) {
      const result = await readRecentMessages(found, 30);
      transcript = result.raw;
    }
  }

  // Extract facts via LLM
  const cfg = (ctx.cfg || {}) as Record<string, unknown>;
  const [REDACTED_SECRET]);
  const baseUrl = resolveBaseUrl(cfg);

  if (transcript) {
    const extractionModel = resolveExtractionModel(cfg);
    log("Extracting facts on session reset", { model: extractionModel, baseUrl, transcriptLen: transcript.length });
    const extractedFacts = await extractFacts(transcript, apiKey, extractionModel, baseUrl);
    if (extractedFacts) {
      const factLines = extractedFacts.split("\n").filter(l => l.trim());
      const factContent = [
        `# Extracted Facts — ${dateStr} ${timeStr} (/reset)`,
        "",
        `- **Session**: ${event.sessionKey}`,
        `- **Session ID**: ${sessionId}`,
        `- **Trigger**: /${event.action}`,
        "",
        "## Facts",
        "",
        ...factLines,
        "",
        "---",
        `_Auto-extracted by memory-flush hook on session reset_`,
        "",
      ].join("\n");

      const factFile = await writeFileSafe(memoryDir, `${dateStr}-facts-reset-${timeClean}.md`, factContent);
      log("Facts saved on reset", { file: factFile.replace(os.homedir(), "~"), factCount: factLines.length });
    } else {
      log("No facts extracted on reset");
    }
  } else {
    log("Skipping extraction on reset", { hasKey: !!apiKey, hasTranscript: !!transcript });
  }

  // Log the reset
  const logContent = [
    `# Session Reset Log — ${dateStr} ${timeStr}`,
    "",
    `- **Action**: /${event.action}`,
    `- **Session**: ${event.sessionKey}`,
    `- **Session ID**: ${sessionId}`,
    `- **Source**: ${source}`,
    "",
    "---",
    `_Auto-logged by memory-flush hook_`,
    "",
  ].join("\n");

  const filename = `${dateStr}-session-log.md`;
  await writeFileSafe(memoryDir, filename, logContent);
  log("Session reset logged", { action: event.action });
}

// --- Main Handler ---

const handler = async (event: {
  type: string;
  action: string;
  sessionKey: string;
  timestamp: Date;
  context: Record<string, unknown>;
  messages: string[];
}) => {
  try {
    if (event.type === "session" && event.action === "compact:before") {
      await onCompactBefore(event);
      return;
    }

    if (event.type === "command" && (event.action === "new" || event.action === "reset")) {
      await onCommandNew(event);
      return;
    }
  } catch (err) {
    log("Handler error", {
      type: event.type,
      action: event.action,
      error: err instanceof Error ? err.message : String(err),
    });
  }
};

export default handler;


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)