Skill: braibaud/memento (version 0.6.0)

--- name: memento description: Local persistent memory for OpenClaw agents. Captures conversations, extracts structured facts via LLM, and auto-recalls relevant knowledge before each turn. Privacy-first, all stored data stays local in SQLite. metadata: version: "0.6.0" author: braibaud license: MIT repository: https://github.com/braibaud/Memento openclaw: emoji: "🧠" kind: plugin requires: node: ">=18.0.0" optionalEnv: - name: ANTHROPIC_API_KEY when: "Using anthropic/* models for extraction" - name: OPENAI_API_KEY when: "Using openai/* models for extraction" - name: MISTRAL_API_KEY when: "Using mistral/* models for extraction" - name: MEMENTO_API_KEY when: "Generic fallback for any provider" - name: CLAUDE_CODE_OAUTH_TOKEN when: "OpenClaw OAuth token for model routing (auto-used when running inside OpenClaw)" - name: MEMENTO_WORKSPACE_MAIN when: "Migration only: path to agent workspace for bootstrapping" - name: MEMENTO_AGENT_PATHS when: "Deep consolidation CLI: explicit agent:path mappings" dataFiles: - path: "~/.engram/conversations.sqlite" purpose: "Main database — conversations, facts, embeddings (local only, never uploaded)" - path: "~/.engram/segments/*.jsonl" purpose: "Human-readable conversation backups (local only)" - path: "~/.engram/migration-config.json" purpose: "Optional: agent workspace paths for one-time migration bootstrap" install: - id: npm kind: node package: "@openclaw/memento" label: "Install Memento plugin (npm)" extensions: - "./src/index.ts" keywords: - memory - knowledge-base - recall - conversation - extraction - embeddings - sqlite - privacy - local - cross-agent --- # Memento — Local Persistent Memory for OpenClaw Agents Memento gives your agents long-term memory. It captures conversations, extracts structured facts using an LLM, and auto-injects relevant knowledge before each AI turn. **All stored data stays on your machine — no cloud sync, no subscriptions.** Extraction uses your configured LLM provider; use a local model (Ollama) for fully air-gapped operation. > ⚠️ **Privacy note:** When `autoExtract` is enabled, conversation segments are sent to your configured LLM provider for fact extraction. If you use a cloud provider (Anthropic, OpenAI, Mistral), that text leaves your machine. For fully local operation, set `extractionModel` to `ollama/<model>` and keep Ollama running locally. ## What It Does 1. **Captures** every conversation turn, buffered per session 2. **Extracts** structured facts (preferences, decisions, people, action items) via configurable LLM (opt-in — see Privacy section) 3. **Recalls** relevant facts before each AI turn using FTS5 keyword search + optional semantic embeddings (BGE-M3) 4. **Respects privacy** — facts are classified as `shared`, `private`, or `secret` based on content, with hard overrides for sensitive categories (medical, financial, credentials) 5. **Cross-agent knowledge** — shared facts flow between agents with provenance tags; private/secret facts never cross boundaries ## Quick Start Install the plugin, restart your gateway, and Memento starts capturing automatically. Extraction is **off by default** — enable it explicitly when ready. ### Optional: Semantic Search Download a local embedding model for richer recall: ```bash mkdir -p ~/.node-llama-cpp/models curl -L -o ~/.node-llama-cpp/models/bge-m3-Q8_0.gguf \ "https://huggingface.co/gpustack/bge-m3-GGUF/resolve/main/bge-m3-Q8_0.gguf" ``` ## Environment Variables All environment variables are **optional** — you only need the one matching your chosen LLM provider: | Variable | When Needed | |----------|-------------| | `ANTHROPIC_API_KEY` | Using `anthropic/*` models for extraction | | `OPENAI_API_KEY` | Using `openai/*` models for extraction | | `MISTRAL_API_KEY` | Using `mistral/*` models for extraction | | `MEMENTO_API_KEY` | Generic fallback for any provider | | `MEMENTO_WORKSPACE_MAIN` | Migration only: path to agent workspace for bootstrapping | No API key needed for `ollama/*` models (local inference). ## Configuration Add to your `openclaw.json` under `plugins.entries.memento.config`: ```json { "memento": { "autoCapture": true, "extractionModel": "anthropic/claude-sonnet-4-6", "extraction": { "autoExtract": true, "minTurnsForExtraction": 3 }, "recall": { "autoRecall": true, "maxFacts": 20, "crossAgentRecall": true, "autoQueryPlanning": false } } } ``` > **`autoExtract: true`** is an explicit opt-in (default: `false`). When enabled, conversation segments are sent to the configured `extractionModel` for LLM-based fact extraction. Omit or set to `false` to keep everything local. > **`autoQueryPlanning: true`** is an explicit opt-in (default: `false`). When enabled, a fast LLM call runs before each recall search to expand the query with synonyms and identify relevant categories — improving precision at the cost of one extra LLM call per turn. ## Data Storage Memento stores all data locally: | Path | Contents | |------|----------| | `~/.engram/conversations.sqlite` | Main database: conversations, facts, embeddings | | `~/.engram/segments/*.jsonl` | Human-readable conversation backups | | `~/.engram/migration-config.json` | Optional: migration workspace paths (only for bootstrapping) | ## Privacy & Data Flow | Feature | Data leaves machine? | Details | |---------|---------------------|---------| | `autoCapture` (default: `true`) | ❌ No | Writes to local SQLite + JSONL only | | `autoExtract` (default: `false`) | ⚠️ Yes, if cloud LLM | Sends conversation text to configured provider. Use `ollama/*` for local. | | `autoRecall` (default: `true`) | ❌ No | Reads from local SQLite only | | Secret facts | ❌ Never | Filtered from extraction context — never sent to any LLM | | Migration | ❌ No | Reads local workspace files, writes to local SQLite | ## Migration (Bootstrap from Existing Memory Files) Migration is an **optional, one-time** process to seed Memento from existing agent memory/markdown files. It is user-initiated only — never runs automatically. ### What it reads Migration reads **only** the files you explicitly list in the config. It does **not** scan your filesystem, read arbitrary files, or access anything outside the configured paths. ### Setup 1. Create `~/.engram/migration-config.json` or set `MEMENTO_WORKSPACE_MAIN`: ```json { "agents": [ { "agentId": "main", "workspace": "/path/to/your-workspace", "paths": ["MEMORY.md", "memory/*.md"] } ] } ``` 2. **Always dry-run first** to verify exactly which files will be read: ```bash npx tsx src/extraction/migrate.ts --all --dry-run ``` The dry-run prints every file path it would read — review this before proceeding. 3. Run the actual migration: ```bash npx tsx src/extraction/migrate.ts --all ``` ### Security notes - Migration only reads files matching the glob patterns you configure - Extracted facts inherit visibility classification (shared/private/secret) - Secret-classified facts are **never** sent to cloud LLM providers - Migration config file is optional — if absent, migration is completely inert - The migration script has no network access beyond the configured extraction LLM ## Architecture - **Capture layer** — hooks `message:received` + `message:sent`, buffers multi-turn segments - **Extraction layer** — async LLM extraction with deduplication, occurrence tracking, temporal state transitions (`previous_value`), and knowledge graph relations (including causal edges with `causal_weight`) - **Storage layer** — SQLite schema v7 (better-sqlite3) with FTS5 full-text search + optional vector embeddings; knowledge graph (`fact_relations` with `causal_weight`), multi-layer clusters, and temporal transition tracking (`previous_value`) - **Recall layer** — optional LLM query planning pre-pass (`autoQueryPlanning`), multi-factor scoring (recency × frequency × category weight), 1-hop graph traversal with causal edge 1.5× boost, injected via `before_prompt_build` hook ## Requirements - OpenClaw 2026.2.20+ - Node.js 18+ - An API key for your preferred LLM provider (for extraction — not needed if extraction is disabled or using Ollama) - Optional: GPU for accelerated embedding search (falls back to CPU gracefully) ## Install ```bash # From ClawHub clawhub install memento # Or for local development git clone https://github.com/braibaud/Memento cd Memento npm install ``` Note: `better-sqlite3` includes native bindings that compile during `npm install`. This is expected behavior for SQLite access.

path: CHANGELOG.md
content: # Changelog

## v0.6.0 — 2026-03-01

### Added
- **Phase 1 — History-agnostic extraction**: LLM no longer receives existing facts during extraction. Prompt loaded from `prompts/extraction.md` at runtime (editable without recompile). New durability filter: "would this still matter in 6 months?"
- **Phase 2 — Embedding-based deduplication**: `deduplicator.ts` fully rewritten. Each extracted fact is compared against existing DB facts using cosine similarity (BGE-M3): ≥0.97 = duplicate (increment occurrence), ≥0.82 = update/supersede, <0.82 = new fact. LLM `duplicate_of`/`supersedes` hints no longer used.
- **Immediate embedding**: New facts are embedded immediately after insertion (no async backfill needed going forward). `EmbeddingEngine` now passed through `ExtractionTrigger` → `deduplicator`.
- **Phase 3 — Relation sweep** (`src/consolidation/relation-sweep.ts`): Background job that builds the knowledge graph from embedding similarity. Three tiers: same-category ≥0.85 auto-linked, cross-category ≥0.88 auto-linked, cross-category 0.72–0.88 → Haiku batch classification. Runs on-demand or can be scheduled nightly. Secret facts are never included.
- **Relation classify prompt** (`prompts/relation-classify.md`): Haiku prompt for batch relation type classification (6 types: caused_by, precondition_of, part_of, related_to, contradicts, superseded_by) with rationale field for auditability.
- **New DB helpers**: `ConversationDB.getAllRelations()` and `getDistinctAgentIds()`.
- **Dedup sweep script** (`src/storage/dedup-sweep.ts`): One-shot pairwise similarity sweep for existing facts — flags pairs 0.88–0.97 for review, auto-merges ≥0.97.
- **Backfill script** (`src/storage/backfill-embeddings.ts`): Populates embeddings for facts added before embedding support was introduced.

### Changed
- `extractFacts()` no longer passes existing facts to the LLM. `_existingFacts` parameter kept for API compatibility (prefixed `_`, unused).
- `processExtractedFacts()` is now `async` — all call sites updated.
- Batch/CLI tools (`migrate.ts`, `ingest.ts`, `process-raw-sessions.ts`) pass `null` for `embeddingEngine` (graceful fallback to new-fact insertion without embedding dedup).
- `includeExistingFactsCount` config option is now deprecated (dedup moved to Phase 2 embeddings).

### Fixed
- Removed hardcoded personal path (`~/greg/.engram/`) from `backfill-embeddings.ts`.

### Security
- All Phase 3 relation sweep operations exclude `secret`-visibility facts — they are never sent to any LLM for classification.
- No new env vars required for core plugin operation; `CLAUDE_CODE_OAUTH_TOKEN` is an OpenClaw internal token (auto-used when running inside OpenClaw, never required by users).

## v0.5.1 — 2026-02-28

### Added
- **Causality edges in knowledge graph** — new `precondition_of` relation type alongside existing `caused_by`; extraction prompt instructs the LLM to prefer causal relation types when a clear cause-and-effect or prerequisite relationship exists
- **Causal weight field** — `causal_weight` column added to `fact_relations` (schema v7); deduplicator sets `causal_weight = 1.5` for `caused_by` and `precondition_of` edges vs. `1.0` for associative edges
- **1.5× graph traversal boost for causal edges** — `search.ts` now applies a `1.5×` score multiplier (vs. `0.4×` for non-causal edges) when traversing `caused_by` / `precondition_of` graph edges during recall
- **Query planning** — new `planQuery()` function in `search.ts` runs an LLM pre-pass to expand the query with synonyms and identify relevant categories/entities before FTS search
- **`recall.autoQueryPlanning` config option** — opt-in (default `false`); when enabled, calls the LLM once per recall turn to improve search precision
- **Temporal state transitions** — `previous_value` column added to `facts` table (schema v7); `db.supersedeFact()` automatically captures the old fact's content as `previous_value` on the new fact
- **Previous value shown in recall context** — `context-builder.ts` appends `_(previously: ...)_` to recalled facts that carry a `previous_value`
- **Schema v7 migration** — `MIGRATE_V6_TO_V7` adds `causal_weight` to `fact_relations` and `previous_value` to `facts`; migration is safe to re-run (duplicate column errors are caught)

### Fixed
- **OpenClaw model routing not invoked during extraction** — `ExtractionTrigger` was constructing with `openClawConfig` but not passing it through to `extractFacts()`; this meant every extraction silently fell back to direct API calls, bypassing the v0.5.0 model routing feature entirely

## v0.5.0 — 2026-02-28

### Changed
- **OpenClaw model routing**: Extraction now delegates to OpenClaw's `runEmbeddedPiAgent` instead of making direct HTTP calls to LLM providers. Memento transparently inherits the agent's configured model, fallback chain, and auth profile.
- **No more standalone API key management**: `MEMENTO_API_KEY` / provider-specific env vars are no longer needed when running inside OpenClaw. Direct API fallback remains for CLI/standalone usage.
- **New module**: `src/extraction/embedded-runner.ts` — thin wrapper around `runEmbeddedPiAgent`.
- **README**: Updated privacy note and model config docs to reflect OpenClaw routing behaviour.

## v0.4.0 — 2026-02-XX

### Changed
- ClawHub trust score improvements: honest messaging, `optionalEnv` + `dataFiles` declarations
- `@anthropic-ai/sdk` removed from dependencies
- `autoExtract: false` default (opt-in extraction)
- Hardcoded paths removed → auto-discovery via `os.homedir()`
- Secret-filtering security comments expanded
- Published to ClawHub as `memento@0.4.0`

### Added
- Side-learning: ingest pipeline from Claude Code JSONL sessions
- Knowledge graph: `fact_relations` table with 6 typed relation types, 1-hop recall traversal
- Fact clusters: `fact_clusters` + `cluster_members` tables, incremental + deep consolidation
- Mission Control tab: Knowledge Graph SVG visualisation (force-directed, cluster hulls)
- Deep consolidation: soft confidence decay, cluster merging, runs at 3 AM via cron

## v0.3.0

- Renamed from Engram → Memento
- Git history rewritten (no personal data)
- Pushed to GitHub: https://github.com/braibaud/Memento

## v0.2.0

- Provider-agnostic refactor: Anthropic, OpenAI, Mistral, Ollama, OpenAI-compatible
- Configurable extraction model
- Temporal recency weighting in recall
- Category priority weights (decisions > routine)
- Confidence threshold (0.6 minimum)
- Per-agent memory + master KB (shared facts)

## v0.1.0

- Initial release: fact extraction, SQLite storage, FTS5 search, auto-recall hook


path: DESIGN.md
content: # OpenClaw Local Supermemory Plugin

## Overview
A fully local, privacy-first persistent memory system for OpenClaw agents. No cloud dependencies, no subscriptions. All data stays on the machine.

This is designed as an **official OpenClaw plugin** from day one.

## Architecture

### Four Layers

```
┌─────────────────────────────────────────────────┐
│ CAPTURE LAYER (Hooks: message events)           │
│ • Buffers all messages per conversation          │
│ • Detects conversation pauses / topic shifts    │
│ • Triggers extraction when segment is complete  │
│ • Multi-turn aware (sliding window, not single) │
└──────────────────┬──────────────────────────────┘
                   │
┌──────────────────▼──────────────────────────────┐
│ EXTRACTION LAYER (Configurable LLM - Sonnet)    │
│ • Receives full conversation segment            │
│ • Extracts: preferences, decisions, facts,      │
│   action items, people info, corrections        │
│ • Deduplicates against existing knowledge       │
│ • Detects fact UPDATES (not just additions)     │
│ • Flags recurring topics (temporal signal)      │
│ • Assigns visibility: private/shared/secret     │
└──────────────────┬──────────────────────────────┘
                   │
┌──────────────────▼──────────────────────────────┐
│ STORAGE LAYER (SQLite + embeddings)             │
│ • Structured facts with categories, timestamps  │
│ • Occurrence tracking (when/how often mentioned)│
│ • Embeddings for semantic search                │
│ • Visibility levels (private/shared/secret)     │
│ • memory/*.md files (human-readable journal)    │
└──────────────────┬──────────────────────────────┘
                   │
┌──────────────────▼──────────────────────────────┐
│ RECALL LAYER (before each AI turn)              │
│ • Semantic search over knowledge base           │
│ • Injects relevant facts as context             │
│ • Temporal awareness ("discussed on Jan 29th")  │
│ • Pattern detection (recurring = important)     │
└─────────────────────────────────────────────────┘
```

### Multi-Agent Architecture

```
┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│   main   │  │ drjones  │  │   bob    │  │  frank   │
│ (main)   │  │(drjones) │  │  (bob)   │  │ (frank)  │
│ memory   │  │ memory   │  │ memory   │  │ memory   │
└────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │             │             │
     └─────────────┴──────┬──────┴─────────────┘
                          │
                   ┌──────▼──────┐
                   │   MASTER    │
                   │ KNOWLEDGE   │
                   │   BASE      │
                   └─────────────┘
```

Each agent searches: own memory → master KB. Never other agents' private stores.

### Visibility Levels
- **`private`** — stays in agent's own store (default for operational details)
- **`shared`** — propagates to master KB (user preferences, family, home setup)
- **`secret`** — never propagated, potentially encrypted (credentials, medical, financial)

### Temporal Pattern Detection
When a topic recurs, track:
- **What**: the fact/topic
- **When**: timestamps of each occurrence
- **Why**: context of each mention (was it a reminder? frustration? update?)
- **Frequency**: occurrence count → importance weight

Recurring topics = core values/unresolved issues. Not just duplicates to squash.

## Phase Plan

### Phase 1: Conversation Capture (THIS PHASE)
- OpenClaw plugin structure (`openclaw.plugin.json`, `package.json`)
- Hook on `message:received` + `message:sent` events
- Buffer conversations per session in memory
- Write raw conversation segments to append-only log files
- Detect conversation pauses (configurable timeout, default 5 min)
- Detect topic shifts (heuristic: long gap or explicit topic change)
- When segment completes, write to `<workspace>/memory/conversations/YYYY-MM-DD-HH-MM-slug.jsonl`
- SQLite schema for conversation segments (metadata, timestamps, session info)

### Phase 2: Extraction + Knowledge Base
- Sonnet extraction of structured facts from conversation segments
- SQLite knowledge base with categories, embeddings, visibility levels
- Occurrence tracking (temporal dimension)
- Deduplication with "recurring topic" detection
- Initial migration: ingest existing memory/*.md files

### Phase 3: Auto-Recall
- Hook on `before_agent_start` (or equivalent) to inject relevant facts
- Semantic search over knowledge base
- Temporal context injection ("you discussed this on...")
- Pattern-aware recall (recurring topics get priority)

### Phase 4: Master Knowledge Base
- Cross-agent fact sharing (respecting visibility)
- Conflict resolution (latest wins, with provenance)
- Master KB aggregation/merge logic

### Phase 5: Package & Publish
- Full test suite
- Documentation
- npm package
- ClawHub listing

## Technical Details

### Plugin Structure
```
openclaw-memento-local/
├── package.json                    # npm package with openclaw.extensions
├── openclaw.plugin.json            # Plugin manifest + config schema
├── src/
│   ├── index.ts                    # Plugin entry point (register hooks + tools)
│   ├── hooks/
│   │   ├── capture.ts              # message:received + message:sent handler
│   │   └── recall.ts               # before_agent_start handler (Phase 3)
│   ├── capture/
│   │   ├── buffer.ts               # In-memory conversation buffer
│   │   ├── segmenter.ts            # Topic shift / pause detection
│   │   └── writer.ts               # Write segments to disk + SQLite
│   ├── extraction/                 # Phase 2
│   │   ├── extractor.ts            # LLM-based fact extraction
│   │   ├── deduplicator.ts         # Fact deduplication + recurrence detection
│   │   └── classifier.ts           # Visibility level assignment
│   ├── storage/
│   │   ├── schema.ts               # SQLite schema definitions
│   │   ├── db.ts                   # Database operations
│   │   └── embeddings.ts           # Local embedding generation
│   ├── recall/                     # Phase 3
│   │   ├── search.ts               # Semantic search
│   │   └── context-builder.ts      # Build injection context
│   └── config.ts                   # Configuration types + defaults
├── HOOK.md                         # Hook metadata (for hook discovery)
└── README.md
```

### Configuration (in openclaw.json)
```json
{
  "plugins": {
    "entries": {
      "memento-local": {
        "enabled": true,
        "config": {
          "extractionModel": "anthropic/claude-sonnet-4-6",
          "embeddingModel": "hf:BAAI/bge-m3-gguf",
          "pauseTimeoutMs": 300000,
          "maxBufferTurns": 50,
          "autoCapture": true,
          "autoRecall": true,
          "visibility": {
            "defaultLevel": "shared",
            "rules": {
              "medical": "secret",
              "credentials": "secret",
              "operational": "private"
            }
          }
        }
      }
    }
  }
}
```

### SQLite Schema (Phase 1 - Conversations)
```sql
-- Conversation segments (raw capture)
CREATE TABLE conversations (
  id TEXT PRIMARY KEY,
  agent_id TEXT NOT NULL,
  session_key TEXT NOT NULL,
  channel TEXT,
  started_at INTEGER NOT NULL,
  ended_at INTEGER NOT NULL,
  turn_count INTEGER NOT NULL,
  raw_text TEXT NOT NULL,
  metadata TEXT -- JSON
);

-- Individual messages within segments
CREATE TABLE messages (
  id TEXT PRIMARY KEY,
  conversation_id TEXT NOT NULL REFERENCES conversations(id),
  role TEXT NOT NULL, -- 'user' or 'assistant'
  content TEXT NOT NULL,
  timestamp INTEGER NOT NULL,
  message_id TEXT, -- provider message ID
  metadata TEXT -- JSON
);
```

### SQLite Schema (Phase 2 - Knowledge Base)
```sql
-- Extracted facts
CREATE TABLE facts (
  id TEXT PRIMARY KEY,
  agent_id TEXT NOT NULL,
  category TEXT NOT NULL, -- preference, decision, person, action_item, etc.
  content TEXT NOT NULL,
  summary TEXT, -- one-line summary
  visibility TEXT NOT NULL DEFAULT 'shared', -- private, shared, secret
  confidence REAL DEFAULT 1.0,
  first_seen_at INTEGER NOT NULL,
  last_seen_at INTEGER NOT NULL,
  occurrence_count INTEGER DEFAULT 1,
  embedding TEXT, -- vector embedding
  supersedes TEXT, -- ID of fact this updates/replaces
  metadata TEXT -- JSON (extra context)
);

-- Fact occurrences (temporal tracking)
CREATE TABLE fact_occurrences (
  id TEXT PRIMARY KEY,
  fact_id TEXT NOT NULL REFERENCES facts(id),
  conversation_id TEXT NOT NULL REFERENCES conversations(id),
  timestamp INTEGER NOT NULL,
  context_snippet TEXT, -- what was said around this mention
  sentiment TEXT -- neutral, frustration, confirmation, correction, etc.
);

-- Master knowledge base (Phase 4)
CREATE TABLE master_facts (
  id TEXT PRIMARY KEY,
  source_agent_id TEXT NOT NULL,
  source_fact_id TEXT NOT NULL,
  -- same columns as facts, minus private ones
  category TEXT NOT NULL,
  content TEXT NOT NULL,
  summary TEXT,
  confidence REAL DEFAULT 1.0,
  first_seen_at INTEGER NOT NULL,
  last_seen_at INTEGER NOT NULL,
  occurrence_count INTEGER DEFAULT 1,
  embedding TEXT,
  metadata TEXT
);
```

## Embedding Model
- **Target**: BGE-M3 (GGUF format)
- **Dimensions**: 1024
- **Why**: Multilingual (EN/FR), state of the art, fits easily in 16GB VRAM
- **Runtime**: GPU-accelerated via node-llama-cpp or llama.cpp server

## Extraction Prompt Design (Phase 2)
The extraction prompt should:
1. Receive the full conversation segment (multi-turn)
2. Extract structured facts with categories
3. Check against existing facts (provided as context) for deduplication
4. Flag updates to existing facts (e.g., "FTP changed from 250W to 260W")
5. Detect recurring topics and assign importance
6. Assign visibility levels based on content category
7. Return structured JSON output

## Key Design Principles
1. **Privacy first** — all data local, no external calls except LLM API
2. **Plugin-ready** — proper OpenClaw plugin from day one
3. **Multi-turn aware** — capture conversation arcs, not single exchanges
4. **Temporal intelligence** — track when and how often topics arise
5. **Configurable** — model, thresholds, visibility rules all configurable
6. **Graceful degradation** — if extraction fails, raw capture still works
7. **Human-readable** — all data inspectable in files + SQLite


path: MIGRATION-SPEC.md
content: # Migration: Existing Memory Files → Memento Knowledge Base

## Overview
One-time migration to bootstrap Memento's knowledge base from existing agent memory files.
Can also be re-run incrementally (skips already-processed files).

## Agent Inventories

Agent workspace paths are configured per-environment in `~/.engram/migration-config.json`
(see the Migration Script section below). The paths below are illustrative examples.

### Main agent
- Workspace: `~/your-workspace/` (set via `MEMENTO_WORKSPACE_MAIN` or migration-config.json)
- Files: `MEMORY.md` + `memory/*.md`
- Content: User preferences, family, tech setup, career, everything

### Specialist agents (optional)
- Workspace: configured per agent in `~/.engram/migration-config.json`
- Files: `MEMORY.md` + `memory/*.md` + domain-specific files
- Content: Domain-specific knowledge (medical records, project files, etc.)

### Visibility Classification
**IMPORTANT**: Visibility is determined by the CONTENT of each fact, NOT by which agent produced it.
The classifier analyzes each extracted fact independently:
- Medical diagnoses, prescriptions, health data → `secret`
- Credentials, passwords, API keys → `secret`
- User preferences, family info, general knowledge → `shared`
- Project-specific technical details → `private`
- A preference discussed with another agent is still `shared`
- A password mentioned to the main agent is still `secret`

No `defaultVisibility` per agent — let the classifier decide per fact.

## Migration Script (`src/extraction/migrate.ts`)

### Interface
```typescript
interface MigrationConfig {
  agents: Array<{
    agentId: string;
    workspace: string;
    paths: string[];           // glob patterns relative to workspace
    defaultVisibility: 'private' | 'shared' | 'secret';
  }>;
  extractionModel: string;     // from plugin config
  dryRun?: boolean;            // log what would be extracted without calling LLM
  skipAlreadyProcessed?: boolean; // default true
}
```

### Default Configuration

Agent workspaces are loaded from `~/.engram/migration-config.json` (created by the user).
Example config:

```json
{
  "agents": [
    {
      "agentId": "main",
      "workspace": "/home/yourname/your-workspace",
      "paths": ["MEMORY.md", "memory/*.md"]
    },
    {
      "agentId": "specialist",
      "workspace": "/home/yourname/other-workspace",
      "paths": ["MEMORY.md", "memory/*.md", "domain/*.md"]
    }
  ]
}
```

Alternatively, set `MEMENTO_WORKSPACE_MAIN` environment variable for the main agent.

```typescript
// NOTE: No defaultVisibility per agent — the classifier determines
// visibility from the content of each extracted fact.
const DEFAULT_EXTRACTION_MODEL = 'anthropic/claude-sonnet-4-6';
```

### Process
1. For each agent, glob the specified paths
2. For each file:
   a. Check if already in `extraction_log` (by creating a synthetic conversation_id from the file path hash)
   b. If not processed (or force=true), read the file content
   c. Create a synthetic conversation entry in the DB (source='migration', raw_text=file content)
   d. Run the extraction prompt with the file content
   e. Insert extracted facts with the agent's default visibility (overridable by classifier)
   f. Log in extraction_log
3. Report summary: files processed, facts extracted, per agent

### Special Handling
- **SOUL.md, AGENTS.md, USER.md, IDENTITY.md**: Skip these — they're system files, not memory
- **HEARTBEAT.md, TOOLS.md**: Skip — operational config, not knowledge
- **Files > 50KB**: Split into chunks before extraction
- **Dr Robby medical files**: Force visibility=secret regardless of classifier output

### CLI Integration
Should be runnable as:
```bash
# Via openclaw plugin command (if registered)
openclaw memento migrate --dry-run
openclaw memento migrate --agent main
openclaw memento migrate --all

# Or directly via node
node -e "require('./src/extraction/migrate.ts').runMigration()"
```

## Tracking
- Each migrated file creates a `conversations` entry with `channel='migration'`
- The `extraction_log` tracks processing status
- Re-running skips already-processed files unless `--force` is passed


path: PHASE2-SPEC.md
content: # Phase 2: Extraction + Knowledge Base

## Overview
Add LLM-powered fact extraction from captured conversation segments, and a structured knowledge base with temporal tracking.

## What to Build

### 1. SQLite Schema Extension (`src/storage/schema.ts`)
Add Phase 2 tables to the existing schema:

```sql
-- Extracted facts
CREATE TABLE IF NOT EXISTS facts (
  id TEXT PRIMARY KEY,
  agent_id TEXT NOT NULL,
  category TEXT NOT NULL, -- preference, decision, person, action_item, correction, technical, etc.
  content TEXT NOT NULL,
  summary TEXT, -- one-line summary
  visibility TEXT NOT NULL DEFAULT 'shared', -- private, shared, secret
  confidence REAL DEFAULT 1.0,
  first_seen_at INTEGER NOT NULL,
  last_seen_at INTEGER NOT NULL,
  occurrence_count INTEGER DEFAULT 1,
  supersedes TEXT, -- ID of fact this updates/replaces
  is_active INTEGER DEFAULT 1, -- 0 if superseded
  metadata TEXT -- JSON
);

-- Fact occurrences (temporal tracking)
CREATE TABLE IF NOT EXISTS fact_occurrences (
  id TEXT PRIMARY KEY,
  fact_id TEXT NOT NULL REFERENCES facts(id),
  conversation_id TEXT NOT NULL REFERENCES conversations(id),
  timestamp INTEGER NOT NULL,
  context_snippet TEXT, -- what was said around this mention
  sentiment TEXT -- neutral, frustration, confirmation, correction, update
);

-- Extraction log (track what's been processed)
CREATE TABLE IF NOT EXISTS extraction_log (
  conversation_id TEXT PRIMARY KEY REFERENCES conversations(id),
  extracted_at INTEGER NOT NULL,
  model_used TEXT NOT NULL,
  facts_extracted INTEGER DEFAULT 0,
  facts_updated INTEGER DEFAULT 0,
  facts_deduplicated INTEGER DEFAULT 0,
  error TEXT -- null if successful
);
```

Add indexes on `facts.category`, `facts.agent_id`, `facts.visibility`, `fact_occurrences.fact_id`.

### 2. Extraction Engine (`src/extraction/extractor.ts`)
Core extraction logic:

- Takes a conversation segment (from the `conversations` table)
- Fetches existing relevant facts from the DB (for deduplication context)
- Calls the configured LLM (default: Sonnet 4.6) with an extraction prompt
- Parses the structured JSON response
- Returns an array of extracted facts

**LLM Call**: Use the OpenClaw API to call the configured model. The plugin API provides:
```typescript
// api.runtime should have an LLM call method, but if not available,
// use a direct HTTP call to the Anthropic API via the configured provider.
// For now, use Node's native fetch to call the Anthropic Messages API.
// The API key should come from the environment (ANTHROPIC_API_KEY).
```

Actually, the simplest approach: use `child_process.execSync` to run:
```bash
openclaw chat --model <model> --message "<prompt>" --json
```

Or better: use the Node.js `fetch` API to call Anthropic directly. The API key is available in the environment.

**Extraction Prompt Design** (critical):
```
You are a memory extraction system. Analyze this conversation segment and extract structured facts.

For each fact, provide:
- category: one of [preference, decision, person, action_item, correction, technical, routine, emotional]
- content: the full fact description
- summary: one-line summary (max 100 chars)
- visibility: one of [private, shared, secret]
  - shared: user preferences, family info, general knowledge (default)
  - private: agent-specific operational details
  - [REDACTED_SECRET], medical info, financial details
- confidence: 0.0-1.0 how certain this is a real fact
- sentiment: one of [neutral, frustration, confirmation, correction, update]

EXISTING FACTS (check for duplicates/updates):
<existing_facts_json>

CONVERSATION:
<conversation_text>

Rules:
1. If a fact UPDATES an existing fact, set supersedes=<existing_fact_id> and note what changed
2. If a fact is a DUPLICATE of an existing fact, note it with duplicate_of=<existing_fact_id> and increment_occurrence=true
3. Extract DECISIONS and AGREEMENTS explicitly — these are high priority
4. Note recurring topics — if something was discussed before, flag it
5. Don't extract trivial exchanges ("hi", "thanks", "ok")
6. For preferences, be specific: "solar report format: emoji headers, import/export breakdown with costs"
7. For people, capture relationships: "Alice is the user's best friend, partner Carol"

Return JSON array of facts.
```

### 3. Deduplication Engine (`src/extraction/deduplicator.ts`)
- Before inserting new facts, check for semantic similarity with existing facts
- If a fact is essentially the same: increment occurrence_count, update last_seen_at, add to fact_occurrences
- If a fact UPDATES an existing one: create new fact with supersedes reference, mark old as inactive
- If a fact is genuinely new: insert normally
- **Key insight**: duplicate detection is a SIGNAL, not just cleanup. Track it in fact_occurrences with context.

### 4. Visibility Classifier (`src/extraction/classifier.ts`)
- Default rules based on category:
  - medical → secret
  - credentials/passwords/tokens → secret
  - operational/workflow → private
  - everything else → shared
- Content-based heuristics (keywords like "password", "token", "diagnosis")
- Configurable override rules from plugin config

### 5. Database Operations (`src/storage/db.ts`)
Add methods to ConversationDB:
- `getUnextractedConversations()` — conversations not in extraction_log
- `getRelevantFacts(agentId, limit)` — active facts for context
- `insertFact(fact)` — insert a new fact
- `updateFactOccurrence(factId, conversationId, context, sentiment)` — add occurrence
- `supersedeFact(oldId, newFact)` — mark old inactive, insert new
- `logExtraction(conversationId, result)` — record extraction attempt
- `getFactsByCategory(agentId, category)` — for recall
- `searchFacts(agentId, query)` — text search via FTS5

### 6. FTS5 Full-Text Search
Add FTS5 virtual table for fact content search:
```sql
CREATE VIRTUAL TABLE IF NOT EXISTS facts_fts USING fts5(
  content, summary, 
  content=facts, content_rowid=rowid
);
```
With triggers to keep it in sync.

### 7. Extraction Trigger (`src/extraction/trigger.ts`)
- Hook into the capture layer: after a segment is written, queue it for extraction
- Run extraction asynchronously (don't block the message flow)
- Respect rate limits (configurable max extractions per minute)
- Skip very short segments (< 3 turns by default, configurable)

### 8. Initial Migration (`src/extraction/migrate.ts`)
- One-time script to process existing memory/*.md files
- Read each file, create a synthetic "conversation" from its content
- Run extraction on each to populate the knowledge base
- This gives us a "warm start" with all existing knowledge

## Configuration Additions
```json
{
  "extraction": {
    "minTurnsForExtraction": 3,
    "maxExtractionsPerMinute": 10,
    "includeExistingFactsCount": 50,
    "autoExtract": true
  }
}
```

## Technical Constraints
- Use Node's native `fetch` for Anthropic API calls (no extra deps)
- The Anthropic API key comes from environment: `process.env.ANTHROPIC_API_KEY`
- All extraction is async and must never block message capture
- Extraction errors should be logged but never crash the plugin
- Use the model from config (`extractionModel` field, default "anthropic/claude-sonnet-4-6")
- Parse the model string: if it starts with "anthropic/", use the Anthropic Messages API
- Model name mapping: "anthropic/claude-sonnet-4-6" → API model "claude-sonnet-4-6"

## What NOT to Build Yet
- Embeddings / vector search (Phase 2b or Phase 3)
- Auto-recall injection (Phase 3)
- Master knowledge base (Phase 4)
- Migration script for existing memory files (defer to after core works)

## File Structure
```
src/extraction/
├── extractor.ts      # LLM-based fact extraction
├── deduplicator.ts   # Fact dedup + recurrence detection  
├── classifier.ts     # Visibility level assignment
└── trigger.ts        # Extraction trigger (post-capture hook)
```


path: README.md
content: # Memento — Local Persistent Memory for OpenClaw Agents

**Memento** gives OpenClaw agents a long-term memory — structured, private, and stored entirely on your machine. All stored data stays local — no cloud sync, no subscriptions. Extraction uses your configured LLM provider; use a local model (Ollama) for fully air-gapped operation.

> ⚠️ **Privacy note:** When `autoExtract` is enabled, conversation segments are sent to your configured LLM provider for fact extraction. Memento uses OpenClaw's built-in model routing — it inherits whichever model you have configured for your agent (including fallbacks). If you use a cloud provider (Anthropic, OpenAI, Mistral), that text leaves your machine. For fully local operation, configure your OpenClaw agent to use Ollama.

---

## Features

- **📥 Capture** — buffers every conversation turn per session, auto-flushes on pause or session end
- **🧠 Extraction** — asynchronously extracts structured facts (preferences, decisions, people, etc.) using the configured LLM; **history-agnostic** — existing facts are NOT passed to the LLM (Phase 1)
- **🔍 Recall** — injects relevant facts before each AI turn via FTS5 keyword search + optional semantic embedding search
- **🔒 Privacy-first** — facts are classified by visibility (`shared` / `private` / `secret`); secret facts never leave your machine or cross agent boundaries
- **🌐 Cross-agent KB** — shared facts from multiple agents are surfaced with provenance tags in recall
- **📊 Temporal intelligence** — recency, frequency, and category weights govern recall ranking; `previous_value` shown when a fact has changed
- **🔗 Knowledge graph** — typed weighted relations between facts (`related_to`, `elaborates`, `contradicts`, `supports`, `caused_by`, `part_of`, `precondition_of`) with 1-hop graph traversal during recall; causal edges get a **1.5× score boost**; Phase 3 background sweep auto-builds relations from embedding similarity
- **🔮 Query planning** — optional LLM pre-pass before retrieval expands queries with synonyms and identifies relevant categories/entities for higher precision recall
- **📦 Multi-layer memory** — facts cluster into higher-level summaries; incremental consolidation after each extraction + periodic deep "sleep" passes with confidence decay

---

## Architecture

```
                ┌──────────────────────────────────────────────────────────────┐
                │                       OpenClaw Agent                         │
                │                                                              │
  Conversation  │   message_received ──► ConversationBuffer ──► SegmentWriter  │
  Flow          │                                                     │        │
                │                                           ExtractionTrigger  │
                │                                                     │        │
                │                                           extractFacts (LLM) │
                │                                                     │        │
                │                                     processExtractedFacts    │
                │                                          │          │        │
                │                              fact_relations    SQLite DB      │
                │                              (graph edges,    (facts, FTS5,  │
                │                              causal_weight)    previous_val) │
                │                                          │          │        │
                │                              incrementalConsolidate          │
                │                              (cluster assignment)            │
                │                                                              │
  Deep Sleep    │   cron (3 AM) ──► deepConsolidate ──► decay + merge + refresh│
  Phase 3       │   cron (3 AM) ──► relationSweep ──► embedding sim → typed edges   │
                │                                                              │
  Recall        │   before_prompt_build ──► [planQuery?] ──► searchRelevantFacts│
                │                           + 1-hop graph traversal ──► inject │
                └──────────────────────────────────────────────────────────────┘
```

### Key Modules

| Module | Description |
|--------|-------------|
| `src/index.ts` | Plugin entry point — registers hooks and services |
| `src/capture/buffer.ts` | In-memory session buffer with auto-flush |
| `src/capture/writer.ts` | Persists segments to SQLite + JSONL |
| `src/extraction/extractor.ts` | **Phase 1**: LLM-based fact extraction — history-agnostic (no existing facts passed); prompt loaded from `prompts/extraction.md` at runtime |
| `src/extraction/embedded-runner.ts` | OpenClaw model routing — delegates LLM calls to `runEmbeddedPiAgent` |
| `src/extraction/deduplicator.ts` | **Phase 2**: embedding-based dedup — cosine similarity classifies each extracted fact as duplicate (≥0.97), update (≥0.82), or new (<0.82); embeds new facts immediately |
| `src/extraction/trigger.ts` | Async extraction scheduling with rate limiting |
| `src/extraction/classifier.ts` | Visibility classification with hard overrides |
| `src/recall/search.ts` | FTS5 + semantic search with multi-factor scoring; optional `planQuery` pre-pass; causal edge 1.5× boost |
| `src/recall/context-builder.ts` | Formats recalled facts for injection; shows `previous_value` for changed facts |
| `src/storage/db.ts` | SQLite database layer (better-sqlite3) |
| `src/storage/embeddings.ts` | Local embedding engine via node-llama-cpp |
| `src/storage/schema.ts` | SQLite schema v7, migrations, row types |
| `src/consolidation/consolidator.ts` | Incremental consolidation — assigns facts to clusters after extraction |
| `src/consolidation/deep-consolidator.ts` | Deep "sleep" consolidation — decay, cluster merging, summary refresh |
| `src/consolidation/relation-sweep.ts` | **Phase 3**: background relation linking — pairwise embedding similarity → typed graph edges; same-category ≥0.85 auto-linked; cross-category 0.72–0.88 → Haiku batch classification |
| `src/cli/deep-consolidate.ts` | CLI entry point for deep consolidation (cron-compatible) |
| `prompts/extraction.md` | LLM prompt for fact extraction — edit to tune quality without recompiling |
| `prompts/relation-classify.md` | Haiku prompt for relation type classification (Phase 3) |
| `src/config.ts` | Plugin configuration with defaults |
| `src/types.ts` | Shared TypeScript types |

### Schema (v7)

The SQLite database at `~/.engram/conversations.sqlite` uses schema version 7:

| Table | Purpose |
|-------|---------|
| `conversations` | Raw conversation segments (one row per flushed buffer) |
| `messages` | Individual messages within a segment |
| `facts` | Extracted knowledge base — includes `previous_value` (v7) for temporal transitions |
| `fact_occurrences` | Temporal tracking of each fact mention |
| `extraction_log` | Which conversations have been processed |
| `fact_relations` | Knowledge graph edges — includes `causal_weight` (v7) for causal edge boosting |
| `fact_clusters` | Higher-level memory clusters (Layer 2+) |
| `cluster_members` | Membership edges linking facts → clusters |
| `ingest_tokens` | Auth tokens for remote JSONL ingest |
| `ingest_file_log` | Dedup log for ingested files |

---

## Installation

Memento is a plugin for OpenClaw. Install it from the plugin directory:

```bash
openclaw plugin install @openclaw/memento
```

Or for local development, clone and link:

```bash
git clone https://github.com/openclaw/memento
cd memento
npm install
npm link
openclaw plugin link /path/to/memento
```

---

## Configuration

Add to your `openclaw.json` under `plugins.entries.memento.config`:

> **`extractionModel`** accepts any of these provider prefixes:
> `anthropic/` · `openai/` · `mistral/` · `ollama/` (or any OpenAI-compatible string as fallback).
> Set the matching API key env var: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `MISTRAL_API_KEY`.
> Ollama runs locally — no key needed.

```json
{
  "memento": {
    "autoCapture": true,
    "dataDir": ".engram",
    "extractionModel": "anthropic/claude-sonnet-4-6",
    "embeddingModel": "hf:BAAI/bge-m3-gguf",
    "agentDisplay": {
      "main": "Main Assistant",
      "medbot": "Medical Bot"
    },
    "extraction": {
      "autoExtract": true,
      "minTurnsForExtraction": 3,
      "maxExtractionsPerMinute": 10,
      "includeExistingFactsCount": 50
    },
    "recall": {
      "autoRecall": true,
      "maxFacts": 20,
      "maxContextChars": 4000,
      "minQueryLength": 5,
      "crossAgentRecall": true,
      "autoQueryPlanning": false
    }
  }
}
```

> **`autoExtract: true`** is an explicit opt-in (default: `false`). When enabled, conversation segments are sent to the configured `extractionModel` for LLM-based fact extraction. Omit or set to `false` to keep everything local (capture + recall still work via SQLite without any LLM calls).

### Configuration Reference

| Key | Default | Description |
|-----|---------|-------------|
| `autoCapture` | `true` | Buffer and persist all conversation messages |
| `dataDir` | `.engram` | Data directory (relative to workspace) |
| `extractionModel` | `anthropic/claude-sonnet-4-6` | LLM for fact extraction — supports `anthropic/`, `openai/`, `mistral/`, `ollama/` prefixes |
| `embeddingModel` | `hf:BAAI/bge-m3-gguf` | Local embedding model (HuggingFace URI or path) |
| `agentDisplay` | `{}` | Human-readable names for agents in cross-agent recall tags |
| `extraction.autoExtract` | `false` | **Opt-in**: automatically extract facts after each segment — sends conversation text to the configured LLM provider |
| `extraction.minTurnsForExtraction` | `3` | Skip very short segments |
| `extraction.maxExtractionsPerMinute` | `10` | Rate limit for LLM calls |
| `extraction.includeExistingFactsCount` | `50` | _(Deprecated — no longer used)_ Dedup is now handled by Phase 2 embedding similarity, not by the extraction LLM |
| `recall.autoRecall` | `true` | Inject relevant facts before each AI turn |
| `recall.maxFacts` | `20` | Maximum facts to inject per turn |
| `recall.maxContextChars` | `4000` | Maximum characters for the injected context block |
| `recall.crossAgentRecall` | `true` | Include shared facts from other agents |
| `recall.autoQueryPlanning` | `false` | **Opt-in**: run a fast LLM pre-pass to expand the query with synonyms/entities before FTS search — improves recall precision at the cost of an extra LLM call per turn |

---

## Local Development Setup

### Prerequisites

- Node.js 18+
- TypeScript 5.7+
- OpenClaw installed globally

### TypeScript path resolution

The `openclaw/plugin-sdk` module type declarations are resolved from the OpenClaw installation. After cloning:

```bash
# Option 1: npm link (recommended for local dev)
npm link openclaw

# Option 2: adjust tsconfig.json paths to point to your global install
# $(npm root -g)/openclaw/dist/plugin-sdk/index.d.ts
```

### Type-check

```bash
npx tsc --noEmit
```

---

## Fact Migration (Bootstrap from Existing Memory Files)

Migration is an **optional, one-time** process to seed Memento from existing agent memory/markdown files. It is user-initiated only — never runs automatically.

> **Tip:** Always run with `--dry-run` first to preview what will be imported without making any writes. Migration reads **only** the files you explicitly list in the config — it does not scan your filesystem or access anything outside the configured paths.

1. Create `~/.engram/migration-config.json`:

```json
{
  "agents": [
    {
      "agentId": "main",
      "workspace": "/home/yourname/your-workspace",
      "paths": ["MEMORY.md", "memory/*.md"]
    }
  ]
}
```

Or set `MEMENTO_WORKSPACE_MAIN` environment variable.

2. Run the migration:

```bash
# Dry run (no writes)
npx tsx src/extraction/migrate.ts --all --dry-run

# Process all agents
npx tsx src/extraction/migrate.ts --all

# Single agent
npx tsx src/extraction/migrate.ts --agent main
```

---

## Privacy & Security

| Feature | Data location | Leaves your machine? |
|---------|--------------|----------------------|
| `autoCapture: true` | SQLite + JSONL on disk | ❌ Never |
| Relation sweep (Phase 3) | Sends fact _summaries_ to Haiku for relation classification; **secret facts are never included** | ✅ Yes (unless using Ollama) |
| `autoExtract: true` | Sends conversation segments to `extractionModel` | ✅ Yes (unless using Ollama) |
| `autoRecall: true` | Reads from local SQLite | ❌ Never |
| `autoQueryPlanning: true` | Sends the user's message to `extractionModel` for query expansion | ✅ Yes (unless using Ollama) |
| Secret facts (`credentials`, `medical`, `financial`) | Filtered **before** extraction context | ❌ Never sent to LLM |

- **Secret facts** (`credentials`, `medical`, `financial`) are never sent to external APIs — they are filtered from the extraction dedup context before LLM calls
- **Private facts** stay within the agent that created them — they are never shared across agents
- **Shared facts** can appear in cross-agent recall (with provenance tags)
- All stored data lives locally in `~/.engram/conversations.sqlite` (or your configured `dataDir`)
- No cloud sync, telemetry, or external storage

**For fully local / air-gapped operation:** set `extractionModel` to `ollama/<model>` (e.g. `ollama/llama3`) and keep Ollama running on localhost. No API keys needed.

---

## Semantic Search (Embeddings)

Memento supports optional local semantic search via BGE-M3 (or any compatible GGUF embedding model):

```bash
# Download the BGE-M3 model (~1.3GB, Q8 quantization)
mkdir -p ~/.node-llama-cpp/models
curl -L -o ~/.node-llama-cpp/models/bge-m3-Q8_0.gguf \
  "https://huggingface.co/gpustack/bge-m3-GGUF/resolve/main/bge-m3-Q8_0.gguf"
```

The embedding engine uses GPU (CUDA) when available, falling back to CPU automatically. If no model is found, keyword (FTS5) search still works.

---

## License

MIT


path: ROADMAP.md
content: # Memento — Roadmap & Ideas Tracker

## ✅ Done

### Core Plugin
- [x] Fact extraction from conversations (capture + extract pipeline)
- [x] SQLite storage with FTS5 full-text search
- [x] Visibility model: shared / private / secret (keyword + LLM classifier)
- [x] Duplicate detection & fact superseding with provenance chain
- [x] Auto-recall injection before each AI turn (`before_prompt_build` hook)
- [x] Per-agent memory + master KB aggregating "shared" facts
- [x] Provider-agnostic refactor (Anthropic, OpenAI, Mistral, Ollama, OpenAI-compatible)
- [x] Configurable extraction model (default: `anthropic/claude-sonnet-4-6`)
- [x] Temporal recency weighting in recall scoring
- [x] Category priority weights (decisions & corrections > routine)
- [x] Confidence scoring (0-1) with minimum threshold (0.6)

### ClawHub Publishing
- [x] Published as `memento@0.3.0` on ClawHub (@braibaud)
- [x] Renamed from Engram → Memento
- [x] Git history rewritten (no personal info leaks)
- [x] Removed credential snooping (no more reading auth-profiles.json)
- [x] TypeScript: zero errors

### Mission Control Tab
- [x] Dashboard with 6 hero stat cards, category bars, donut charts, growth timeline
- [x] Extraction pipeline health monitor
- [x] Top 10 most-reinforced facts
- [x] Recent 20 facts stream with category pills, visibility/agent badges
- [x] Expandable fact cards (full content, metadata, timestamps)
- [x] Feedback actions: 👍 Helpful, 👎 Not relevant, 🗑️ Remove, 💬 Correct
- [x] Toast notifications, inline confirmation for destructive actions
- [x] Auto-refresh every 60s, category click-to-filter

## ✅ Recently Completed (v0.4.0)

### ClawHub Trust Score — DONE
- [x] Honest messaging (no "Everything stays local"), recommends Ollama for local
- [x] Structured `optionalEnv` + `dataFiles` declarations in SKILL.md
- [x] `@anthropic-ai/sdk` removed from package-lock
- [x] `autoExtract: false` default (opt-in)
- [x] Migration docs: explicit filesystem access + security notes
- [x] Secret-filtering: expanded security comments for audit trail
- [x] Hardcoded paths removed from CLI → auto-discovery via os.homedir()
- [x] Published as `memento@0.4.0`

### Git / GitHub — DONE  
- [x] Git history rewritten (no personal info)
- [x] Pushed to `https://github.com/braibaud/Memento`

---

## 🔧 In Progress

### Side-Learning (Claude Code → Memento bridge)
- [x] Staging schema (v1 JSON format for fact exports)
- [x] Ingest pipeline: reads `~/.engram/staging/*.json`, validates, deduplicates, promotes to KB
- [x] `/memento-export` slash command installed at `~/.claude/commands/`
- [ ] **Automatic bridge** — scan `~/.claude/projects/**/*.jsonl`, extract facts via LLM, push to staging. Works regardless of macOS app vs CLI. No user action required.
- [ ] Cron job to run ingest pipeline periodically

### Knowledge Graph & Multi-Layer Memory
- [x] Schema v4: `fact_relations` table — typed weighted edges between facts
- [x] 6 relation types: related_to, elaborates, contradicts, supports, caused_by, part_of
- [x] Extraction prompt identifies relations to existing facts
- [x] Relations stored during deduplication pipeline
- [x] Recall enriched with 1-hop graph traversal (graph-sourced facts at 40% parent score)
- [x] Context builder shows graph-sourced facts with 🔗 provenance tag
- [x] Visibility filtering in graph traversal (shared < private < secret)
- [x] Schema v5: `fact_clusters` + `cluster_members` tables
- [x] Incremental consolidation after each extraction (no LLM, fast)
  - Assigns unclustered facts to existing clusters via graph edges
  - Creates new clusters from 3+ unclustered facts in same category
- [x] Deep consolidation ("sleep" pass)
  - Soft confidence decay: 0.5%/day after 30-day grace, floor 0.3
  - Clustered facts decay 50% slower, high-occurrence 30% slower
  - Cluster summary refresh from member facts
  - Cluster merging when Jaccard overlap > 60%
- [x] Deep consolidation runs at 3 AM via cron
- [ ] LLM-assisted semantic grouping in deep consolidation
- [ ] Layer 3+ meta-cluster creation
- [x] MC tab: Knowledge Graph visualization (force-directed SVG, cluster hulls, hover/click/drag, collapsible section)

## 📋 Planned — Next Up

### Interview / Onboarding Process
Structured conversation where an agent "interviews" the user to rapidly build a knowledge base. Solves the cold-start problem.
- Targeted questions across categories: identity, family, preferences, tools, work, home
- Facts from interviews start at lower confidence (~0.7)
- Confidence reinforced when facts confirmed in real conversations (natural trust-building)
- Track interview completion by category
- Trigger via CLI command or Telegram `/memento interview`
- **Priority: HIGH** — biggest UX impact for new users

### Value Measurement / Analytics
How do we know Memento is actually helping?
- **Recall precision**: % of injected facts that get 👍 vs 👎 (foundation already built with feedback buttons)
- **Question avoidance**: does the agent ask fewer clarifying questions over time?
- **User correction rate**: do "no, I told you..." messages decrease?
- **Fact staleness tracking**: how many recalled facts are thumbed down as outdated?
- **Feedback analytics panel** on MC tab (once enough data exists)
- Target: >80% recall precision = clearly adding value
- **Priority: MEDIUM** — needs data from feedback buttons first

### Fact Discussion / Conversational Correction (V3)
Full multi-turn conversation about a specific fact, beyond the current inline correction.
- Clicking "Discuss" opens a chat-like interface (or triggers a Telegram message)
- Agent can ask clarifying questions, propose corrections, negotiate final fact
- Could be embedded in MC tab or route to Telegram with fact pre-loaded as context
- Current V2 inline correction gets ~80% of the value
- **Priority: LOW** — V2 correction covers most use cases

## 💡 Ideas / Backlog

### Visibility Model Improvements
- Secret vs Private distinction is muddy — consider renaming "secret" → "sensitive"?
- Keyword classifier is too aggressive (e.g., "Ben prefers his medication in the morning" = preference, not medical secret)
- Consider user-configurable sensitivity rules

### Feedback Loop into Extraction
- Thumbs up/down patterns should feed back into extraction quality
- If certain categories consistently get thumbed down → tune extraction prompt
- If certain agents produce more low-quality facts → diagnostic signal
- Track feedback analytics per category and per agent

### Embedding Improvements
- [ ] BGE-M3 embedding backfill (blocked: needs `sudo apt install cmake`)
- Semantic search for recall (currently FTS5 keyword + recency)
- Embedding-based duplicate detection

### Data Migration
- [ ] Migrate all agents' `memory/*.md` files into Memento knowledge base (warm start)
- Migration script exists but hasn't been run on production data

### Packaging & Documentation
- [ ] Blog post covering architecture and design decisions
- [ ] Logo generation (blocked: OpenAI API key expired)
- [ ] Phase 5 final packaging review

### MC Tab Enhancements
- Feedback analytics panel (charts showing precision over time)
- Fact search/filter by content text
- Agent-specific views (filter by agent)
- Fact timeline view (chronological, not just "recent 20")
- Export facts as JSON/CSV

---
*Created: 2026-02-25 by Greg*
*Last updated: 2026-02-25 08:20*

## ✅ Recently Completed (v0.5.0)

### OpenClaw Model Routing Integration
- [x] Extraction now uses OpenClaw's `runEmbeddedPiAgent` instead of direct provider HTTP calls
- [x] Memento transparently inherits the agent's configured model (provider, fallbacks, auth)
- [x] No more standalone API key management in Memento — all delegated to OpenClaw
- [x] Graceful fallback to direct API if running outside OpenClaw (CLI mode)
- [x] New `src/extraction/embedded-runner.ts` module wraps the OpenClaw integration
- [x] `ExtractionTrigger` and `extractFacts()` updated to thread `api.config` through
- [x] README and docs updated to reflect model routing behaviour

## ✅ Recently Completed (v0.5.1)

### AMA-Agent Inspired Features — ALL DONE (from arxiv 2602.22769)

#### 1. Causality Edges in fact_relations — DONE
- [x] Added `precondition_of` edge type (alongside `caused_by`)
- [x] Extraction prompt instructs LLM to prefer causal types for cause-effect / prerequisite relationships
- [x] `causal_weight` column added to `fact_relations` (schema v7); set to `1.5` for causal edges
- [x] Graph traversal in `search.ts` applies **1.5× score boost** for `caused_by` / `precondition_of` edges vs. 0.4× for associative edges
- [x] `deduplicator.ts` stores `causal_weight` when persisting relation edges

#### 2. Query Planning Before Retrieval — DONE
- [x] `planQuery()` function in `search.ts` calls LLM to expand query with synonyms/entities/categories
- [x] `recall.autoQueryPlanning` config option (default: `false`, opt-in)
- [x] Graceful fallback: if planning fails or times out, falls back to raw FTS query
- [x] Uses OpenClaw model routing via `runViaOpenClaw`

#### 3. Temporal State Transitions — DONE
- [x] `previous_value` column added to `facts` table (schema v7)
- [x] `db.supersedeFact()` automatically captures old fact's content as `previous_value` on the new fact
- [x] `context-builder.ts` shows `_(previously: ...)_` in recall output for facts that have changed
- [x] `deduplicator.buildFactRow()` initialises `previous_value: null`; actual value set atomically in `supersedeFact`

### What was NOT implemented (by design)
Full trajectory-based reasoning from AMA-Bench — designed for agent-environment interactions (web nav, code editing), not personal assistant conversations. Memento's dialogue-centric approach is correct for this use case.

## ✅ Recently Completed (v0.5.2)

### Mission Control — Memento Tabs
- [x] `/memento` tab: Agent Memory view (current agent's facts, category bars, graph visualization)
- [x] `/memento/global` tab: Global Memory across all agents
- [x] `/memento/ingest` tab: Ingest pipeline management — approve/reject/extract projects
- [x] Old `/ingest` route redirects to `/memento/ingest`
- [x] Knowledge graph visualization: force-directed SVG, cluster hulls, tooltips, collapsible

### Bug Fix: trigger.ts openClawConfig
- [x] `trigger.ts` was passing `undefined` for `openClawConfig` to `extractFacts()` — silently broken since v0.5.0
- [x] Fixed: config now threaded correctly through `handleAfterResponse`

### Mission Control Extraction Pipeline
- [x] Direct Anthropic API call (Bearer OAuth) replaces broken `openclaw agent` CLI approach
- [x] `openclaw agent --agent main` deadlocks from inside main session (file lock) — bypassed
- [x] drjones treating extraction as heartbeat during quiet hours — bypassed
- [x] OAuth token requires `Authorization: Bearer` + `anthropic-beta: oauth-2025-04-20` header
- [x] `content_received_at` orphan bug fixed: reset timestamps for projects with 0 content files
- [x] Pending query robust against empty-content edge case
- [x] Mission Control now runs as systemd service (auto-restart, survives shell death)

### Warm-Start Ingestion
- [x] 93 Claude Code JSONL session files pushed from Mac Mini → Mission Control
- [x] 268 new facts extracted, 27 updated, 757 total active facts in Engram DB
- [x] memento-ingest CLI on Mac Mini patched: shows `⚠️ (folder deleted)` for missing project folders

---

## 🔍 Quality Analysis — 2026-03-01 Findings

### Problem: Memento stores too much, filters too little

After analyzing the live DB (1007 total facts, 533 active in `technical` category alone):

**Findings:**
1. **Confidence ≠ importance.** Almost everything scores 0.9+. A Feb-4 Drive folder upload has the same weight as Ben's family tree.
2. **`technical` is a dump category.** 533 active technical facts — meeting notes, transient statuses, one-off findings live alongside durable architectural decisions.
3. **Action items are never marked done.** 145 active action_items, many clearly stale: "gog CLI hanging since Feb 7", "Engram Phase 3/4/5 TODO" (already shipped), etc.
4. **All warm-start facts share the same timestamp (02-23)** — deduplication can't operate on them over time, no staleness signal.
5. **No durability filter.** The extraction prompt doesn't ask "would this matter in 6 months?" It asks "is this a fact?" — a much lower bar.

**Root cause:** The LLM extraction prompt is tuned for completeness, not curation. It's a journalist (capture everything), not a librarian (keep what matters).

---

## 📋 Planned — Next Up (revised priority)

### 🔴 HIGH: Fact Quality & Relevancy Improvements

#### 1. Stricter Extraction Prompt
- Add durability filter: "Would this fact still be relevant in 6 months?"
- Reject meeting notes, transient statuses, completed TODOs, session artifacts
- Require minimum information density: no facts that are just "X happened on date Y"
- Categories to be more selective: `technical` should only store architectural decisions, not implementation logs
- **Actionable test:** before storing, ask "would this help me answer a future question I can't predict?"

#### 2. TTL / Auto-Expiry for Action Items
- `action_item` facts that aren't reinforced within 30 days → auto-set `is_active=0`
- "DONE/COMPLETED/RESOLVED" prefix in summary → immediately deactivate
- Weekly consolidation pass: scan action_items older than 14 days, check if still valid via LLM
- This alone would eliminate ~50% of current noise

#### 3. Importance Score (separate from Confidence)
- Add `importance` field (1-5) alongside `confidence`
- Confidence = "how sure are we this is true" (current)
- Importance = "how useful will this be for future recall" (new)
- Scoring heuristics: person/family/preference facts → 4-5; architectural decisions → 4; meeting notes → 1-2; transient status → 1
- Recall injection: only facts with importance ≥ 3 unless query specifically relevant

#### 4. MC Bulk Curation UI
- "Review" mode in `/memento/ingest` or new `/memento/review` tab
- Shows facts sorted by (low importance × low reinforcement × age)
- Quick actions: Keep / Archive / Delete / Recategorize
- Batch archive: "archive all action_items older than 30 days"
- This gives Ben direct control without touching code

### 🟡 MEDIUM: Value Measurement / Analytics
*(unchanged from previous plan)*

### 🟡 MEDIUM: Interview / Onboarding Process  
*(unchanged from previous plan)*

### 🟢 LOW: Fact Discussion / Conversational Correction (V3)
*(unchanged from previous plan)*

---

## 💡 Ideas / Backlog (updated)

### Visibility Model Improvements
- Secret vs Private distinction is muddy — consider renaming "secret" → "sensitive"?
- Keyword classifier is too aggressive (e.g., medication timing = preference, not medical secret)
- Consider user-configurable sensitivity rules

### Feedback Loop into Extraction
- Thumbs up/down patterns should feed back into extraction quality
- If certain categories consistently get thumbed down → tune extraction prompt
- Track feedback per category and per agent

### Embedding Improvements
- [ ] BGE-M3 embedding backfill (blocked: needs `sudo apt install cmake`)
- Semantic search for recall (currently FTS5 keyword + recency)
- Embedding-based duplicate detection

### Data Migration
- [ ] Migrate all agents' `memory/*.md` files into Memento knowledge base (warm start)

### Packaging & Documentation
- [ ] Blog post covering architecture and design decisions
- [ ] Logo generation (blocked: OpenAI API key expired)

### MC Tab Enhancements
- Feedback analytics panel
- Fact search/filter by content text
- Agent-specific views
- Fact timeline view
- Export facts as JSON/CSV

---
*Last updated: 2026-03-01 by Greg*

---

## 🏗️ Architecture Redesign — 2026-03-01

### Decision: 3-Phase Extraction Pipeline

Based on analysis of current extraction quality issues, the pipeline is being redesigned into 3 cleanly separated phases:

#### Phase 1 — LLM Extraction (history-agnostic)
- **What**: Extract facts from the conversation segment only
- **What NOT**: No existing facts passed, no deduplication rules, no relation detection
- **Output**: Raw fact candidates (category, content, confidence, visibility)
- **Key rule**: "Would this fact still be relevant in 6 months?" — if no, skip
- **Prompt**: Moved to dedicated `.md` file (not embedded in code)
- **Status**: 🔲 TODO

#### Phase 2 — Embedding-Based Deduplication
- **What**: Embed each candidate fact, cosine similarity against DB
- **Dedup threshold**: ~0.92 cosine → mark as duplicate/supersede
- **Update threshold**: ~0.75–0.91 → merge/update existing fact
- **No LLM needed here at all**
- **Prerequisite**: All active facts must have embeddings
- **Status**: 🔲 TODO
- **Blocker**: 392/867 active facts currently missing embeddings → backfill needed

#### Phase 3 — Relation Linking (async, embedding-first)
- **What**: Dedicated background job linking facts to each other
- **Approach**: Embedding similarity across full DB → candidate pairs → typed edges
- **Not in the hot path**: Runs during consolidation (3 AM cron or similar)
- **LLM involvement**: Optional secondary pass to classify relation type on high-similarity pairs
- **Why not LLM-primary**: Would need full DB context; current windowed approach produces noisy edges
- **Status**: 🔲 TODO (design agreed, implementation pending)

### Other Decisions (2026-03-01)
- **Prompts in .md files**: All LLM prompts extracted from code to dedicated markdown files (editable without recompile)
- **Existing facts NOT passed to extractor**: Removes history dependency, reduces token cost, improves extraction quality
- **Embeddings are prerequisite for Phase 2**: Need backfill job for 392 facts missing embeddings
- **CUDA symlink confirmed**: `/usr/local/cuda-12.6/lib64/libcudart.so.12` → exists ✅
- **CUDA GPU status**: CUDA build present (`linux-x64-cuda` binaries exist), but no explicit `ggml_cuda_init` confirmation in logs — CPU fallback possible. Needs verification.
- **Embedding coverage**: 475/867 active facts have embeddings (55%) — insufficient for Phase 2

### Embedding Backfill — TODO
- Write a one-time backfill script: iterate facts WHERE `embedding IS NULL`, embed via bge-m3, store
- Run during off-peak (can be slow — 392 facts × ~50ms each ≈ ~20s)
- Required before Phase 2 deduplication is viable

---
*Architecture section added: 2026-03-01 by Greg*

---

## ✅ Recently Completed (v0.6.0 — Pipeline Redesign)

### Phase 1: History-Agnostic Extraction — DONE
- [x] `prompts/extraction.md` — new extraction prompt with durability filter ("would this matter in 6 months?")
- [x] Explicit anti-patterns: meeting notes, transient status, completed TODOs, implementation details
- [x] Tighter category guidance for `technical` (architectural facts only) and `action_item` (open + urgent only)
- [x] Prompt loaded from file at runtime — editable without recompile
- [x] `extractor.ts`: existing facts NO LONGER passed to LLM (removed `{{existing_facts}}` placeholder)
- [x] `_existingFacts` param kept for API compat but prefixed `_` (unused)

### Phase 2: Embedding-Based Deduplication — DONE
- [x] `deduplicator.ts` fully rewritten — LLM hints (`duplicate_of`, `supersedes`) no longer used
- [x] Cosine similarity against existing embedded facts:
  - `>= 0.97` → duplicate (increment occurrence_count)
  - `>= 0.82` → update/supersede (old fact deactivated, new inserted)
  - `< 0.82`  → new fact
- [x] Same-category comparison (avoids cross-category false positives)
- [x] New facts embedded immediately after insertion (no async backfill needed going forward)
- [x] `processExtractedFacts` now `async` — all call sites updated
- [x] `EmbeddingEngine` now passed to `ExtractionTrigger` and forwarded to deduplicator
- [x] Batch/CLI call sites (`migrate.ts`, `ingest.ts`, `process-raw-sessions.ts`) pass `null` gracefully (fall back to new-fact insertion)

### Embedding Backfill — DONE
- [x] `backfill-embeddings.ts` script — ran successfully
- [x] 392 missing embeddings generated; 870/870 active facts now have embeddings (100% coverage)
- [x] CPU mode (CUDA init fails: driver version insufficient for runtime version — node-llama-cpp CUDA issue)

### Dedup Sweep — DONE
- [x] `dedup-sweep.ts` — pairwise cosine similarity sweep across all embedded facts
- [x] 1 auto-merged (sim=0.98), 26 flagged for review (0.88–0.97)
- [x] DB confirmed clean at near-exact level; noise is breadth, not duplicates

---

## 🔲 Phase 3: Relation Linking (Next)

### Design (agreed 2026-03-01)
- Dedicated background job (runs at 3AM or on demand, not in hot path)
- Input: all active facts with embeddings
- Step 1: Find candidate pairs via embedding similarity (threshold ~0.75 — lower than dedup)
- Step 2: For high-similarity cross-category pairs → assign `related_to` edge
- Step 3: Optional LLM pass on top-N pairs to classify specific relation type (caused_by, part_of, etc.)
- Step 4: Store edges in `fact_relations` table with `created_by: "relation-sweep"`
- Avoids re-processing already-linked pairs (track via `metadata` or separate log)

### Status: ✅ DONE (v0.6.0)

Implemented:
- `src/consolidation/relation-sweep.ts`: full background job (pairwise cosine, 3 tiers, Haiku batch)
- `prompts/relation-classify.md`: structured Haiku prompt (6 relation types, rationale field)
- `db.ts`: `getAllRelations()` + `getDistinctAgentIds()` helpers
- Mission Control: `POST /api/memento/relation-sweep` endpoint + UI button on Memento page
- Secret facts excluded from all sweep operations



path: assets/LOGO-BRIEF.md
content: # Memento Logo Brief — "The Synapse"

## Concept
A single neural memory trace: one central node radiating synaptic connections outward. Represents the moment a memory is encoded — a spark becoming permanent.

## Composition

### Central Node (The Memento)
- **Shape**: Perfect circle, slightly larger than surrounding nodes (~20% of total logo width)
- **Color**: Warm gold (#FFD54F) with a subtle radial glow fading to transparent
- **Glow**: Soft halo around the node, suggesting energy/activation — not harsh, more like a warm ember
- **Position**: Slightly off-center (golden ratio) or dead center, depending on what feels more dynamic

### Neural Branches (3-4 branches)
- **Shape**: Organic but geometric — not perfectly straight, with gentle curves suggesting neural pathways. Think circuit-board-meets-biology
- **Color**: Electric blue (#4FC3F7), solid near the center, fading slightly toward endpoints
- **Width**: Thin lines (~2-3px at icon size), tapering slightly at the ends
- **Pattern**: Asymmetric — branches should NOT be evenly spaced. One branch longer than others. One might fork into two sub-branches. This asymmetry makes it feel organic, not mechanical
- **Small nodes at endpoints**: Each branch terminates in a smaller circle (~8% of logo width), same electric blue but slightly dimmer than the central gold node. These are the connected memories

### Optional: Micro-dots
- Between the main branches, 2-3 tiny dots (2-3px) scattered sparsely — suggesting latent connections not yet formed. Very subtle, barely visible. Same blue at ~30% opacity.

## Background
- **Primary**: Deep indigo/dark navy (#1A1A2E or #0D1B2A)
- **Alternative**: Transparent (for light backgrounds, branches become dark navy, central node stays gold)
- Must work on both dark and light backgrounds — provide both variants

## Style
- **Flat design** with subtle glow effects (not 3D, not skeuomorphic)
- **Minimalist** — no unnecessary decoration
- **Geometric but organic** — clean lines with slight curves, not rigid grid
- Think: the aesthetic of Stripe, Linear, or Vercel — modern tech, not sci-fi

## Sizes & Variants Needed
1. **Icon** (square, 1:1): 512×512, 256×256, 128×128, 64×64, 32×32, 16×16
   - At 16-32px: simplify to just the central node + 3 short stubs (branches won't read at this size)
2. **Logo mark** (square, padded): Icon with breathing room for use as app icon / npm avatar
3. **Logo + wordmark** (horizontal): Icon left, "Memento" text right
   - Font: Clean geometric sans-serif (Inter, Geist, or similar)
   - "Memento" in white (#FFFFFF) on dark, or dark navy (#0D1B2A) on light
   - Letter-spacing slightly expanded (+2-3%)
4. **Monochrome**: Single-color version (white on dark, black on light) for contexts where color isn't available

## Color Palette
| Name | Hex | Usage |
|------|-----|-------|
| Gold (accent) | #FFD54F | Central node, highlights |
| Electric Blue | #4FC3F7 | Branches, secondary nodes |
| Deep Indigo | #1A1A2E | Background (dark variant) |
| Dark Navy | #0D1B2A | Background (darker variant) |
| White | #FFFFFF | Text, monochrome on dark |
| Near Black | #121212 | Monochrome on light |

## What to Avoid
- ❌ No brain silhouettes (too literal, too medical)
- ❌ No full neural networks with dozens of nodes (too complex, doesn't scale)
- ❌ No lightning bolts or electricity symbols (too aggressive)
- ❌ No gradients on the branches (keep them solid)
- ❌ No text inside the logo mark itself
- ❌ No drop shadows (flat design only)

## Mood References
- The simplicity of the Electron.js logo (atom with orbits)
- The warmth of the Firebase logo (amber glow)
- The clean tech feel of the Supabase or Prisma logos
- A touch of the biological/neural feel of Neuralink's branding

## File Formats
- SVG (vector, primary)
- PNG (raster, all sizes above, with transparency)
- ICO (for favicon use)

---

*"Every memory starts with a spark."*


path: docs/PROJECT.md
content: # Memento — Local Persistent Memory for AI Agents

> *"The physical trace of a memory, encoded in the brain."*

## The Problem

Every AI agent has the same fatal flaw: amnesia.

No matter how brilliant the conversation, how important the decision, how many times you've told it your preferences — the next session starts blank. The agent wakes up, reads a few files, and hopes for the best. If something didn't get written down, it's gone forever.

This isn't a theoretical concern. It's a daily frustration for anyone running a persistent AI assistant. You have a long conversation. You agree on a format for daily reports. You explain a preference. The session ends, or the context window compacts, and the next day your agent asks you the same question again. Or worse — it confidently does the wrong thing because it never remembered the right thing.

The core issue isn't that AI agents *can't* remember. It's that they rely on *themselves* to write things down, and they don't always do it. It's like asking someone with short-term memory loss to maintain their own journal — the very capacity they're missing is the one required to fix the problem.

## The Landscape

### How Memory Works Today

Most AI agent frameworks treat memory as an afterthought. The typical approach:

1. **Manual file writing**: The agent is instructed to write important things to markdown files. This works *when the agent remembers to do it*. Which is exactly the problem.

2. **Session dumps**: Save the raw conversation transcript when a session ends. Useful for search, but not for building structured knowledge. And if the session crashes or the user doesn't explicitly close it? Lost.

3. **Cloud memory services**: Products like Supermemory offer automatic capture and recall — the gold standard for reliability. But they require paid subscriptions and send all your conversation data to a third-party cloud. For a personal assistant with access to your email, calendar, family details, and financial information? That's a non-starter for many users.

4. **Vector databases**: Some setups use Pinecone, Weaviate, or Qdrant to store embeddings of past conversations. Powerful but complex — you're now running infrastructure for what should be a feature, not a platform.

### What's Missing

None of these approaches solve the fundamental challenge: **building genuine intuition over time**.

Storing facts is easy. Knowing *which* facts matter is hard. Understanding *why* a topic keeps coming up — that's the real intelligence.

When a human assistant has worked with you for years, they don't just remember your preferences. They notice patterns: "You always get stressed about this project in Q4." "You've mentioned wanting to change this three times — maybe it's time to actually do it." "Last time we tried this approach, it didn't work — here's why."

That's not retrieval. That's intuition. And it's what we're building.

## Memento: The Design

Memento is a fully local, privacy-first persistent memory system for [OpenClaw](https://openclaw.ai) AI agents. No cloud dependencies, no subscriptions, no data leaving your machine.

### Core Principles

1. **Privacy first** — all data stays local. No external API calls except to the LLM for extraction.
2. **Automatic capture** — no reliance on the agent remembering to write things down.
3. **Multi-turn awareness** — captures conversation arcs, not single exchanges.
4. **Temporal intelligence** — tracks *when* and *how often* topics arise, not just *what*.
5. **Intuition through patterns** — recurring topics aren't duplicates to squash; they're signals of importance.
6. **Multi-agent ready** — each agent has its own memory, with a shared master knowledge base.
7. **Human-readable** — all data inspectable in files and SQLite. No black boxes.
8. **Plugin-ready** — built as a proper OpenClaw plugin from day one.

### Architecture: Four Layers

Memento operates as a pipeline with four distinct layers, each handling a different aspect of the memory lifecycle:

```
┌─────────────────────────────────────────────────────┐
│           CAPTURE LAYER (Hook-based)                │
│                                                     │
│  • Hooks into message:received + message:sent       │
│  • Buffers all messages per conversation session    │
│  • Detects conversation pauses (configurable)       │
│  • Detects topic shifts                             │
│  • Triggers extraction when a segment completes     │
│  • Multi-turn: captures full conversation arcs      │
└──────────────────────┬──────────────────────────────┘
                       │ conversation segment
┌──────────────────────▼──────────────────────────────┐
│           EXTRACTION LAYER (LLM-powered)            │
│                                                     │
│  • Receives the full conversation segment           │
│  • Extracts structured facts:                       │
│    - Preferences ("report format should be...")     │
│    - Decisions ("we agreed to use SQLite")          │
│    - People ("Lily studies in Montréal")            │
│    - Action items ("update the cron job")           │
│    - Corrections ("actually, it's 260W not 250W")  │
│  • Deduplicates against existing knowledge          │
│  • Detects fact UPDATES (not just additions)        │
│  • Flags recurring topics as importance signals     │
│  • Assigns visibility: private / shared / secret    │
└──────────────────────┬──────────────────────────────┘
                       │ structured facts
┌──────────────────────▼──────────────────────────────┐
│           STORAGE LAYER (SQLite + embeddings)       │
│                                                     │
│  • Structured facts with categories + timestamps    │
│  • Occurrence tracking (temporal dimension)         │
│  • Vector embeddings for semantic search            │
│  • Visibility levels (private/shared/secret)        │
│  • Human-readable markdown backup                   │
│  • Per-agent databases + master knowledge base      │
└──────────────────────┬──────────────────────────────┘
                       │ relevant context
┌──────────────────────▼──────────────────────────────┐
│           RECALL LAYER (Pre-turn injection)         │
│                                                     │
│  • Fires before each AI turn                        │
│  • Semantic search over knowledge base              │
│  • Injects relevant facts as context                │
│  • Temporal awareness: "you discussed this on..."   │
│  • Pattern detection: recurring = higher priority   │
│  • Agent profile: persistent user understanding     │
└─────────────────────────────────────────────────────┘
```

### Why Multi-Turn Matters

Most memory systems — including Supermemory — capture on a per-turn basis. After each AI response, they grab the last user+assistant exchange and process it.

This is fundamentally insufficient.

Consider this real conversation:

> **User**: "That's not the format we agreed on."  
> **Agent**: "I can't find what format you're referring to."  
> **User**: "We already had that conversation multiple times, I'm getting worried."  
> **Agent**: "You're right, I'll do better."  
> **User**: *forwards the original message with the agreed format*  

No single turn captures the full meaning here. The meaning lives in the *arc*: there was a prior agreement, it was forgotten, the user is frustrated, the format is being re-established, and this is a *recurring* failure. A per-turn system would see each exchange in isolation. Memento captures the complete segment and extracts the real signal: "Solar report format was agreed on Feb 22, re-confirmed Feb 23 after being forgotten. This is a recurring issue — priority: critical."

### The Temporal Dimension

When Memento detects a recurring topic, it doesn't just note the duplication. It tracks three dimensions:

- **What**: the fact or topic being discussed
- **When**: timestamps of every occurrence
- **Why**: the context of each mention — was it a reminder? an expression of frustration? a correction? an update?

This transforms flat knowledge into *weighted understanding*. A preference mentioned once is a data point. A preference mentioned three times in two weeks with increasing frustration is a **core value** — something the agent should never forget and should proactively respect.

This is the closest thing to intuition an AI agent can develop: not just knowing facts, but understanding which facts matter most and why.

### Privacy and Visibility

Not all facts should be shared freely. Memento implements three visibility levels:

| Level | Scope | Example |
|-------|-------|---------|
| **`shared`** | Propagates to master KB, visible to all agents | User preferences, family info, home setup |
| **`private`** | Stays in the originating agent's store | Agent-specific workflows, operational details |
| **`secret`** | Never propagated, potentially encrypted at rest | Credentials, medical info, financial details |

The extraction layer assigns default visibility based on content category (medical → secret, preferences → shared), but this is overridable — either explicitly ("keep this private") or via per-agent policy rules.

### Multi-Agent Architecture

In a multi-agent setup, each agent maintains its own memory store. But agents often need shared context: if one agent learns about a user's preference, others should benefit from that knowledge too.

Memento solves this with a **master knowledge base**:

```
┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│  Agent A │  │  Agent B │  │  Agent C │  │  Agent D │
│  memory  │  │  memory  │  │  memory  │  │  memory  │
└────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │             │             │
     └─────────────┴──────┬──────┴─────────────┘
                          │ shared facts only
                   ┌──────▼──────┐
                   │   MASTER    │
                   │ KNOWLEDGE   │
                   │    BASE     │
                   └─────────────┘
```

When an agent searches memory, it queries:
1. Its own private store (all visibility levels)
2. The master knowledge base (shared facts from all agents)

It never accesses another agent's private store directly. Privacy boundaries are enforced at the architecture level, not by convention.

### Storage: Why SQLite, Not PostgreSQL

For a personal assistant running on one machine, SQLite is the right choice:

- **Zero infrastructure**: no server, no daemon, no port management
- **Atomic writes**: crash-safe by default
- **FTS5**: full-text search built in, fast and capable
- **Single-file database**: trivially backupable
- **Already in the ecosystem**: OpenClaw uses SQLite for its existing memory index

The schema tracks both raw conversations and extracted knowledge:

**Conversations** (raw capture — never loses data):
- Conversation segments with metadata (session, channel, timestamps, turn count)
- Individual messages within segments

**Facts** (extracted knowledge):
- Categorized entries (preference, decision, person, action_item, etc.)
- Visibility levels
- Occurrence count and temporal tracking
- Vector embeddings for semantic search
- Supersession tracking (which fact replaced which)

**Fact Occurrences** (temporal intelligence):
- Links facts to the conversations where they appeared
- Context snippets showing what was said
- Sentiment classification (neutral, frustration, confirmation, correction)

### Embeddings: BGE-M3

Memento uses [BGE-M3](https://huggingface.co/BAAI/bge-m3) for local vector embeddings:

- **Multilingual**: handles mixed-language conversations (English/French, etc.) natively
- **State of the art**: top-performing open-source embedding model
- **1024 dimensions**: good balance of precision and storage
- **GGUF format**: runs locally via llama.cpp, GPU-accelerated
- **~1.1 GB in FP16**: fits easily alongside other workloads

Semantic search means asking "what did we decide about the report format?" finds the answer even if the exact words never appear in the stored fact. This is critical for natural recall — humans don't search their memory with keywords.

### Extraction Model

Fact extraction uses a configurable LLM — defaulting to Claude Sonnet 4.6, but swappable as models improve. The extraction prompt receives:

1. The full conversation segment (multi-turn)
2. Existing relevant facts (for deduplication)
3. Instructions to categorize, classify visibility, and detect updates

The output is structured JSON: facts with categories, confidence scores, visibility levels, and relationships to existing knowledge. Cost is minimal — a short extraction call on a few hundred words of conversation costs ~$0.01-0.02.

### Configuration

Everything is configurable via OpenClaw's standard plugin config:

```json
{
  "plugins": {
    "entries": {
      "memento": {
        "enabled": true,
        "config": {
          "extractionModel": "anthropic/claude-sonnet-4-6",
          "embeddingModel": "hf:BAAI/bge-m3-gguf",
          "pauseTimeoutMs": 300000,
          "maxBufferTurns": 50,
          "autoCapture": true,
          "autoRecall": true,
          "visibility": {
            "defaultLevel": "shared",
            "rules": {
              "medical": "secret",
              "credentials": "secret",
              "operational": "private"
            }
          }
        }
      }
    }
  }
}
```

## Implementation Roadmap

### Phase 1: Conversation Capture *(in progress)*
The foundation. Every message sent and received is captured automatically via OpenClaw hooks. Conversations are buffered in memory, segmented by topic/pause, and written to SQLite + human-readable logs. Even before extraction exists, this ensures no conversation is ever lost again.

### Phase 2: Extraction + Knowledge Base
Sonnet-powered fact extraction from conversation segments. Structured knowledge base with categories, temporal tracking, deduplication, and recurring topic detection. Initial migration of existing memory files into the knowledge base.

### Phase 3: Auto-Recall
Before each AI turn, semantic search over the knowledge base to inject relevant context. The agent automatically "remembers" relevant past conversations, preferences, and decisions. Temporal awareness: "you discussed this on January 29th."

### Phase 4: Master Knowledge Base
Cross-agent fact sharing with privacy enforcement. Shared facts propagate to a master store; private and secret facts stay isolated. Conflict resolution when multiple agents have different versions of the same fact.

### Phase 5: Package & Publish
Full test suite, documentation, npm package, and listing on [ClawHub](https://clawhub.com). Available for any OpenClaw user to install and configure.

## The Bigger Picture

Memento isn't just a memory plugin. It's an attempt to answer a fundamental question in AI agent design: **how do you build genuine understanding over time?**

Current AI systems are extraordinary at in-context reasoning but terrible at cross-session continuity. They can analyze a complex problem in a single conversation but can't remember what you told them yesterday. This asymmetry is the single biggest barrier to AI agents becoming truly useful long-term companions.

The solution isn't more context window. It's structured, searchable, temporally-aware knowledge that persists independently of any single conversation. Knowledge that evolves — where facts update rather than accumulate, where importance is weighted by recurrence, and where the agent develops something that looks remarkably like intuition.

That's what Memento builds. One conversation at a time.

---

*Memento is an open-source project built for [OpenClaw](https://openclaw.ai). Contributions welcome.*

*Logo: "The Synapse" — a central node radiating neural connections. Because every memory starts with a spark.*


path: openclaw.plugin.json
content: {
  "id": "memento",
  "name": "Memento",
  "description": "Memento — Local persistent memory with temporal intelligence for OpenClaw agents.",
  "configSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "pauseTimeoutMs": {
        "type": "number",
        "minimum": 1000,
        "description": "Idle time (ms) before flushing a conversation segment"
      },
      "maxBufferTurns": {
        "type": "number",
        "minimum": 1,
        "maximum": 500,
        "description": "Maximum message turns before auto-flush"
      },
      "autoCapture": {
        "type": "boolean",
        "description": "Automatically capture all conversation messages"
      },
      "dataDir": {
        "type": "string",
        "description": "Directory for storing conversation data (relative to workspace)"
      },
      "extractionModel": {
        "type": "string",
        "description": "LLM model for fact extraction (Phase 2)"
      },
      "embeddingModel": {
        "type": "string",
        "description": "Local embedding model for semantic search"
      },
      "extraction": {
        "type": "object",
        "properties": {
          "autoExtract": {
            "type": "boolean",
            "description": "Automatically extract facts after each segment"
          },
          "minTurnsForExtraction": {
            "type": "number",
            "minimum": 1,
            "description": "Minimum turns before extraction"
          },
          "maxExtractionsPerMinute": {
            "type": "number",
            "minimum": 1,
            "description": "Rate limit for extractions"
          },
          "includeExistingFactsCount": {
            "type": "number",
            "minimum": 0,
            "description": "(Deprecated) No longer used — dedup is now handled by Phase 2 embedding similarity"
          }
        }
      },
      "recall": {
        "type": "object",
        "properties": {
          "autoRecall": {
            "type": "boolean",
            "description": "Automatically inject recalled facts before each AI turn"
          },
          "maxFacts": {
            "type": "number",
            "minimum": 1,
            "maximum": 100,
            "description": "Maximum facts to inject per turn"
          },
          "maxContextChars": {
            "type": "number",
            "minimum": 500,
            "maximum": 20000,
            "description": "Maximum characters for injected context"
          },
          "minQueryLength": {
            "type": "number",
            "minimum": 1,
            "description": "Minimum message length to trigger recall"
          },
          "crossAgentRecall": {
            "type": "boolean",
            "description": "Include shared facts from other agents (master KB)"
          }
        }
      }
    }
  },
  "uiHints": {
    "pauseTimeoutMs": {
      "label": "Pause Timeout (ms)",
      "help": "Idle time before flushing a conversation segment (default: 300000 = 5 min)",
      "advanced": true,
      "placeholder": "300000"
    },
    "maxBufferTurns": {
      "label": "Max Buffer Turns",
      "help": "Maximum message turns before auto-flush (default: 50)",
      "advanced": true,
      "placeholder": "50"
    },
    "autoCapture": {
      "label": "Auto-Capture",
      "help": "Automatically capture all conversation messages (default: true)"
    },
    "dataDir": {
      "label": "Data Directory",
      "help": "Directory for storing conversation data, relative to workspace (default: .engram)",
      "advanced": true,
      "placeholder": ".engram"
    },
    "extractionModel": {
      "label": "Extraction Model (Phase 2)",
      "help": "LLM model for fact extraction — not used in Phase 1",
      "advanced": true,
      "placeholder": "anthropic/claude-sonnet-4-6"
    },
    "embeddingModel": {
      "label": "Embedding Model",
      "help": "Local embedding model for semantic search (future: BGE-M3)",
      "advanced": true,
      "placeholder": "hf:BAAI/bge-m3-gguf"
    },
    "recall": {
      "label": "Recall Settings",
      "help": "Configure auto-recall behavior (Phase 3)"
    }
  }
}


path: package-lock.json
content: {
  "name": "@openclaw/memento",
  "version": "0.2.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@openclaw/memento",
      "version": "0.2.0",
      "dependencies": {
        "better-sqlite3": "^11.0.0"
      },
      "bin": {
        "memento-token": "dist/cli/ingest-tokens.js"
      },
      "devDependencies": {
        "@types/better-sqlite3": "^7.6.13",
        "@types/node": "^22",
        "typescript": "^5.7.0"
      }
    },
    "node_modules/@types/better-sqlite3": {
      "version": "7.6.13",
      "resolved": "https://registry.npmjs.org/@types/better-sqlite3/-/better-sqlite3-7.6.13.tgz",
      "integrity": [REDACTED_SECRET],
      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@types/node": "*"
      }
    },
    "node_modules/@types/node": {
      "version": "22.19.11",
      "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.11.tgz",
      "integrity": [REDACTED_SECRET],
      "dev": true,
      "license": "MIT",
      "dependencies": {
        "undici-types": "~6.21.0"
      }
    },
    "node_modules/base64-js": {
      "version": "1.5.1",
      "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "MIT"
    },
    "node_modules/better-sqlite3": {
      "version": "11.10.0",
      "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz",
      "integrity": [REDACTED_SECRET],
      "hasInstallScript": true,
      "license": "MIT",
      "dependencies": {
        "bindings": "^1.5.0",
        "prebuild-install": "^7.1.1"
      }
    },
    "node_modules/bindings": {
      "version": "1.5.0",
      "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "file-uri-to-path": "1.0.0"
      }
    },
    "node_modules/bl": {
      "version": "4.1.0",
      "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "buffer": "^5.5.0",
        "inherits": "^2.0.4",
        "readable-stream": "^3.4.0"
      }
    },
    "node_modules/buffer": {
      "version": "5.7.1",
      "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "MIT",
      "dependencies": {
        "base64-js": "^1.3.1",
        "ieee754": "^1.1.13"
      }
    },
    "node_modules/chownr": {
      "version": "1.1.4",
      "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    },
    "node_modules/decompress-response": {
      "version": "6.0.0",
      "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "mimic-response": "^3.1.0"
      },
      "engines": {
        "node": ">=10"
      },
      "funding": {
        "url": "https://github.com/sponsors/sindresorhus"
      }
    },
    "node_modules/deep-extend": {
      "version": "0.6.0",
      "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=4.0.0"
      }
    },
    "node_modules/detect-libc": {
      "version": "2.1.2",
      "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "Apache-2.0",
      "engines": {
        "node": ">=8"
      }
    },
    "node_modules/end-of-stream": {
      "version": "1.4.5",
      "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "once": "^1.4.0"
      }
    },
    "node_modules/expand-template": {
      "version": "2.0.3",
      "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "(MIT OR WTFPL)",
      "engines": {
        "node": ">=6"
      }
    },
    "node_modules/file-uri-to-path": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/fs-constants": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/github-from-package": {
      "version": "0.0.0",
      "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/ieee754": {
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "BSD-3-Clause"
    },
    "node_modules/inherits": {
      "version": "2.0.4",
      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    },
    "node_modules/ini": {
      "version": "1.3.8",
      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    },
    "node_modules/mimic-response": {
      "version": "3.1.0",
      "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=10"
      },
      "funding": {
        "url": "https://github.com/sponsors/sindresorhus"
      }
    },
    "node_modules/minimist": {
      "version": "1.2.8",
      "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/mkdirp-classic": {
      "version": "0.5.3",
      "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/napi-build-utils": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/node-abi": {
      "version": "3.87.0",
      "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.87.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "semver": "^7.3.5"
      },
      "engines": {
        "node": ">=10"
      }
    },
    "node_modules/once": {
      "version": "1.4.0",
      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC",
      "dependencies": {
        "wrappy": "1"
      }
    },
    "node_modules/prebuild-install": {
      "version": "7.1.3",
      "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
      "integrity": [REDACTED_SECRET],
      "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",
      "license": "MIT",
      "dependencies": {
        "detect-libc": "^2.0.0",
        "expand-template": "^2.0.3",
        "github-from-package": "0.0.0",
        "minimist": "^1.2.3",
        "mkdirp-classic": "^0.5.3",
        "napi-build-utils": "^2.0.0",
        "node-abi": "^3.3.0",
        "pump": "^3.0.0",
        "rc": "^1.2.7",
        "simple-get": "^4.0.0",
        "tar-fs": "^2.0.0",
        "tunnel-agent": "^0.6.0"
      },
      "bin": {
        "prebuild-install": "bin.js"
      },
      "engines": {
        "node": ">=10"
      }
    },
    "node_modules/pump": {
      "version": "3.0.3",
      "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.3.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "end-of-stream": "^1.1.0",
        "once": "^1.3.1"
      }
    },
    "node_modules/rc": {
      "version": "1.2.8",
      "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
      "dependencies": {
        "deep-extend": "^0.6.0",
        "ini": "~1.3.0",
        "minimist": "^1.2.0",
        "strip-json-comments": "~2.0.1"
      },
      "bin": {
        "rc": "cli.js"
      }
    },
    "node_modules/readable-stream": {
      "version": "3.6.2",
      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "inherits": "^2.0.3",
        "string_decoder": "^1.1.1",
        "util-deprecate": "^1.0.1"
      },
      "engines": {
        "node": ">= 6"
      }
    },
    "node_modules/safe-buffer": {
      "version": "5.2.1",
      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "MIT"
    },
    "node_modules/semver": {
      "version": "7.7.4",
      "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC",
      "bin": {
        "semver": "bin/semver.js"
      },
      "engines": {
        "node": ">=10"
      }
    },
    "node_modules/simple-concat": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "MIT"
    },
    "node_modules/simple-get": {
      "version": "4.0.1",
      "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ],
      "license": "MIT",
      "dependencies": {
        "decompress-response": "^6.0.0",
        "once": "^1.3.1",
        "simple-concat": "^1.0.0"
      }
    },
    "node_modules/string_decoder": {
      "version": "1.3.0",
      "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "safe-buffer": "~5.2.0"
      }
    },
    "node_modules/strip-json-comments": {
      "version": "2.0.1",
      "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "engines": {
        "node": ">=0.10.0"
      }
    },
    "node_modules/tar-fs": {
      "version": "2.1.4",
      "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "chownr": "^1.1.1",
        "mkdirp-classic": "^0.5.2",
        "pump": "^3.0.0",
        "tar-stream": "^2.1.4"
      }
    },
    "node_modules/tar-stream": {
      "version": "2.2.0",
      "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT",
      "dependencies": {
        "bl": "^4.0.3",
        "end-of-stream": "^1.4.1",
        "fs-constants": "^1.0.0",
        "inherits": "^2.0.3",
        "readable-stream": "^3.1.1"
      },
      "engines": {
        "node": ">=6"
      }
    },
    "node_modules/tunnel-agent": {
      "version": "0.6.0",
      "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "Apache-2.0",
      "dependencies": {
        "safe-buffer": "^5.0.1"
      },
      "engines": {
        "node": "*"
      }
    },
    "node_modules/typescript": {
      "version": "5.9.3",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
      "integrity": [REDACTED_SECRET],
      "dev": true,
      "license": "Apache-2.0",
      "bin": {
        "tsc": "bin/tsc",
        "tsserver": "bin/tsserver"
      },
      "engines": {
        "node": ">=14.17"
      }
    },
    "node_modules/undici-types": {
      "version": "6.21.0",
      "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
      "integrity": [REDACTED_SECRET],
      "dev": true,
      "license": "MIT"
    },
    "node_modules/util-deprecate": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "MIT"
    },
    "node_modules/wrappy": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
      "integrity": [REDACTED_SECRET],
      "license": "ISC"
    }
  }
}


path: package.json
content: {
  "name": "@openclaw/memento",
  "version": "0.6.0",
  "description": "Memento — Local persistent memory for OpenClaw agents. Privacy-first, no cloud, no subscriptions.",
  "type": "module",
  "openclaw": {
    "extensions": [
      "./src/index.ts"
    ]
  },
  "bin": {
    "memento-token": "./dist/cli/ingest-tokens.js",
    "process-raw-sessions": "./dist/cli/process-raw-sessions.js"
  },
  "scripts": {
    "build": "tsc -p tsconfig.build.json",
    "process-raw-sessions": "node dist/cli/process-raw-sessions.js"
  },
  "dependencies": {
    "better-sqlite3": "^11.0.0"
  },
  "devDependencies": {
    "@types/better-sqlite3": "^7.6.13",
    "@types/node": "^22",
    "typescript": "^5.7.0"
  }
}


path: prompts/extraction.md
content: # Memento — Extraction Prompt

This file defines the LLM prompt used during fact extraction.
Edit this file to tune extraction quality without touching compiled code.
The placeholder `{{conversation}}` is replaced at runtime with the conversation segment.

---

## Prompt

You are a personal memory curator. Your job is to extract **durable, high-value facts** from a conversation.

**The golden rule:** Only extract a fact if it would still be useful 6 months from now, in a future conversation that has no other context. If in doubt, skip it.

For each fact, output:
- `category`: one of `[preference, decision, person, action_item, correction, technical, routine, emotional, secret]`
- `content`: full fact, self-contained and specific (no pronouns like "he", "it" — name things explicitly)
- `summary`: one-line summary, max 100 chars
- `visibility`: one of `[shared, private, secret]`
  - `shared`: about the user's life, preferences, relationships, durable facts (default)
  - `private`: agent-specific instructions or operational context
  - `secret`: credentials, sensitive medical/financial details
- `confidence`: 0.0–1.0 — how certain you are this is accurate and worth keeping

**What TO extract:**
- Personal facts: identity, family, relationships, home, vehicles, health history
- Stable preferences: communication style, tools, formats, workflows
- Explicit decisions and agreements (what was decided, not the discussion leading to it)
- Standing action items with clear owner/deadline (things that will matter next week)
- Architectural/system facts that won't change soon (infrastructure, integrations, credentials location)
- Recurring patterns: training schedule, work routine, financial habits

**What NOT to extract:**
- Meeting notes, status updates, progress reports — transient by nature
- Facts that are only relevant today ("it's raining", "server is down right now")
- Completed or cancelled action items ("DONE:", "RESOLVED:", "CLOSED:")
- Implementation details of a one-time task (how you fixed a bug, what command you ran)
- Things the user said they'll handle immediately — not worth storing
- Filler, greetings, acknowledgements ("ok", "thanks", "sounds good")
- Anything already implied by other stored facts (don't restate what's obvious from context)

**Category guidance:**
- `technical`: durable architectural facts only — system design, credentials location, integration patterns, recurring technical constraints. NOT implementation logs, NOT "we ran X command today"
- `action_item`: only open items with real urgency or a known deadline. If it sounds like something that might already be done, skip it.
- `decision`: the outcome only, not the reasoning. "Ben chose X over Y" — not the debate.
- `person`: who they are and their relationship to the user. Stable attributes only.
- `routine`: recurring scheduled behaviors (training days, weekly meetings, medication timing)
- `secret`: treat carefully — credentials, sensitive health info, financial account details

**Confidence guidelines:**
- 0.9–1.0: explicitly stated, unambiguous
- 0.7–0.9: clearly implied by context
- 0.6–0.7: uncertain but plausible — borderline, use sparingly
- < 0.6: skip entirely

**Output format:**
Return a JSON array of fact objects. If nothing worth storing was discussed, return `[]`.
Respond with ONLY the JSON array — no markdown, no explanation, no wrapper object.

---

## Conversation

{{conversation}}


path: prompts/relation-classify.md
content: # Relation Classification Prompt

Given two facts from a knowledge base, classify the semantic relationship between them.

## Relation Types

- `caused_by` — Fact A is a consequence or result of Fact B
- `precondition_of` — Fact A must be true for Fact B to happen
- `part_of` — Fact A is a component or sub-element of Fact B
- `related_to` — Fact A and Fact B share a common topic or context (general)
- `contradicts` — Fact A and Fact B cannot both be true simultaneously
- `superseded_by` — Fact A has been replaced or updated by Fact B

## Instructions

You will receive a JSON array of candidate pairs. For each pair, output a JSON object with:
- `pair_index`: the index from the input (0-based)
- `relation_type`: one of the types above
- `strength`: 0.5–1.0 (confidence in this classification)
- `rationale`: one short sentence explaining why

Only classify pairs where a meaningful relationship exists. If two facts are only weakly related, use `related_to` with strength 0.5–0.6. If no meaningful relationship exists, omit the pair from the output.

Respond with ONLY a JSON array, no markdown, no explanation.

## Pairs

{{pairs}}


path: src/capture/buffer.ts
content: import type { PluginConfig } from "../config.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

export type BufferEntry = {
  role: "user" | "assistant";
  content: string;
  timestamp: number;
  messageId?: string;
  /** Channel ID this message arrived on / was sent via */
  channel: string;
};

export type SessionBuffer = {
  sessionKey: string;
  entries: BufferEntry[];
  /** Timer handle for pause-timeout flush */
  timer: ReturnType<typeof setTimeout> | null;
  startedAt: number;
  lastActivityAt: number;
};

export type FlushHandler = (
  sessionKey: string,
  buf: SessionBuffer,
) => Promise<void>;

// ---------------------------------------------------------------------------
// ConversationBuffer
// ---------------------------------------------------------------------------

/**
 * In-memory buffer that groups messages by session key.
 * Auto-flushes when:
 *   • Pause timeout exceeded (no new messages for `pauseTimeoutMs`)
 *   • Max buffer turns reached (`maxBufferTurns`)
 *   • Caller requests an explicit flush (session end / reset)
 */
export class ConversationBuffer {
  private readonly buffers = new Map<string, SessionBuffer>();

  constructor(
    private readonly cfg: PluginConfig,
    private readonly onFlush: FlushHandler,
    private readonly logger: PluginLogger,
  ) {}

  // ---- Public API ---------------------------------------------------------

  /** Add a message to the buffer for the given session key. */
  async addMessage(sessionKey: string, entry: BufferEntry): Promise<void> {
    let buf = this.buffers.get(sessionKey);

    if (!buf) {
      buf = {
        sessionKey,
        entries: [],
        timer: null,
        startedAt: entry.timestamp,
        lastActivityAt: entry.timestamp,
      };
      this.buffers.set(sessionKey, buf);
    }

    // Reset the pause timer each time a message arrives
    this.resetTimer(buf);

    buf.entries.push(entry);
    buf.lastActivityAt = entry.timestamp;

    // Auto-flush when max turns is reached
    if (buf.entries.length >= this.cfg.maxBufferTurns) {
      await this.flushSession(sessionKey);
    }
  }

  /** Flush a specific session's buffer. No-op if empty or unknown. */
  async flushSession(sessionKey: string): Promise<void> {
    const buf = this.buffers.get(sessionKey);
    if (!buf) return;

    // Cancel pending timer so it does not double-fire
    this.clearTimer(buf);

    // Remove from map before flushing so concurrent flushes are not duped
    this.buffers.delete(sessionKey);

    if (buf.entries.length === 0) return;

    try {
      await this.onFlush(sessionKey, buf);
    } catch (err) {
      this.logger.warn(
        `memento: flush error for session "${sessionKey}": ${String(err)}`,
      );
    }
  }

  /** Flush all active session buffers in parallel (sessions are independent). */
  async flushAll(): Promise<void> {
    const keys = Array.from(this.buffers.keys());
    await Promise.all(keys.map((key) => this.flushSession(key)));
  }

  /** Number of sessions currently buffered. */
  get size(): number {
    return this.buffers.size;
  }

  // ---- Internals ----------------------------------------------------------

  private resetTimer(buf: SessionBuffer): void {
    this.clearTimer(buf);
    buf.timer = setTimeout(() => {
      this.flushSession(buf.sessionKey).catch((err) => {
        this.logger.warn(
          `memento: timer flush error for session "${buf.sessionKey}": ${String(err)}`,
        );
      });
    }, this.cfg.pauseTimeoutMs);
  }

  private clearTimer(buf: SessionBuffer): void {
    if (buf.timer !== null) {
      clearTimeout(buf.timer);
      buf.timer = null;
    }
  }
}


path: src/capture/writer.ts
content: import { randomUUID } from "node:crypto";
import { existsSync, mkdirSync, appendFileSync } from "node:fs";
import { join } from "node:path";
import type { PluginLogger } from "../types.js";
import { ConversationDB } from "../storage/db.js";
import type { ConversationRow, MessageRow } from "../storage/schema.js";
import type { SessionBuffer } from "./buffer.js";

// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------

/**
 * Extract agentId from a session key of the form "agent:<agentId>:<sessionName>".
 * Falls back to the raw session key if it does not match the expected format.
 */
function parseAgentId(sessionKey: string): string {
  const parts = sessionKey.split(":");
  if (parts[0] === "agent" && parts.length >= 2 && parts[1]) {
    return parts[1];
  }
  return sessionKey;
}

/**
 * Format a Date as "YYYY-MM-DD-HHMM" for use in filenames.
 * All components are zero-padded to fixed width.
 */
function formatDateForFilename(d: Date): string {
  const pad = (n: number, w = 2) => String(n).padStart(w, "0");
  return [
    d.getFullYear(),
    "-",
    pad(d.getMonth() + 1),
    "-",
    pad(d.getDate()),
    "-",
    pad(d.getHours()),
    pad(d.getMinutes()),
  ].join("");
}

// ---------------------------------------------------------------------------
// SegmentWriter
// ---------------------------------------------------------------------------

/**
 * Persists a flushed conversation buffer as:
 *   • A row + child rows in SQLite (`<dataDir>/conversations.sqlite`)
 *   • A JSONL line appended to `<dataDir>/conversations/YYYY-MM-DD-HHMM.jsonl`
 *
 * Both writes are best-effort: failures are logged but never propagated so
 * that a write error never disrupts the agent.
 *
 * `writeSegment` returns the ConversationRow that was written (or null if the
 * buffer was empty), so the caller can pass it to the extraction trigger.
 */
export class SegmentWriter {
  private _db: ConversationDB | null = null;

  constructor(
    /** Resolved absolute path to the data directory (e.g. /home/user/.engram/) */
    private readonly dataDir: string,
    private readonly logger: PluginLogger,
  ) {}

  // ---- Public API ---------------------------------------------------------

  /**
   * Persist a conversation segment to SQLite + JSONL.
   * Returns the written ConversationRow, or null if the buffer was empty.
   */
  writeSegment(sessionKey: string, buf: SessionBuffer): ConversationRow | null {
    if (buf.entries.length === 0) return null;

    const segmentId = randomUUID();
    const agentId = parseAgentId(sessionKey);
    const channel = buf.entries[0]?.channel ?? "unknown";

    const rawText = buf.entries
      .map((e) => `[${e.role}] ${e.content}`)
      .join("\n\n---\n\n");

    const conversationRow: ConversationRow = {
      id: segmentId,
      agent_id: agentId,
      session_key: sessionKey,
      channel,
      started_at: buf.startedAt,
      ended_at: buf.lastActivityAt,
      turn_count: buf.entries.length,
      raw_text: rawText,
      metadata: null,
    };

    const messageRows: MessageRow[] = buf.entries.map((entry) => ({
      id: randomUUID(),
      conversation_id: segmentId,
      role: entry.role,
      content: entry.content,
      timestamp: entry.timestamp,
      message_id: entry.messageId ?? null,
      metadata: null,
    }));

    // Write to SQLite (synchronous better-sqlite3 under the hood)
    try {
      this.getDb().insertConversationWithMessages(conversationRow, messageRows);
    } catch (err) {
      this.logger.warn(
        `memento: SQLite write failed for segment ${segmentId}: ${String(err)}`,
      );
    }

    // Write human-readable JSONL backup
    try {
      this.writeJsonlBackup(segmentId, sessionKey, conversationRow, messageRows, buf);
    } catch (err) {
      this.logger.warn(
        `memento: JSONL write failed for segment ${segmentId}: ${String(err)}`,
      );
    }

    this.logger.info(
      `memento: captured segment ${segmentId} ` +
        `(${buf.entries.length} turns, ${buf.lastActivityAt - buf.startedAt}ms, ` +
        `session: ${sessionKey}, channel: ${channel})`,
    );

    return conversationRow;
  }

  /**
   * Expose the underlying DB so other components (e.g. ExtractionTrigger)
   * can share the same connection without opening a second file handle.
   */
  getDb(): ConversationDB {
    if (!this._db) {
      const dbPath = join(this.dataDir, "conversations.sqlite");
      this._db = new ConversationDB(dbPath);
    }
    return this._db;
  }

  close(): void {
    this._db?.close();
    this._db = null;
  }

  // ---- Internals ----------------------------------------------------------

  private writeJsonlBackup(
    segmentId: string,
    sessionKey: string,
    conversation: ConversationRow,
    messages: MessageRow[],
    buf: SessionBuffer,
  ): void {
    const dir = join(this.dataDir, "conversations");
    if (!existsSync(dir)) {
      mkdirSync(dir, { recursive: true });
    }

    const dateStr = formatDateForFilename(new Date(buf.startedAt));
    const filePath = join(dir, `${dateStr}.jsonl`);

    // Write one JSON object per message, each on its own line
    const lines = messages.map((msg) =>
      JSON.stringify({
        segmentId,
        sessionKey,
        agentId: conversation.agent_id,
        channel: conversation.channel,
        turnCount: conversation.turn_count,
        segmentStartedAt: conversation.started_at,
        segmentEndedAt: conversation.ended_at,
        messageId: msg.id,
        role: msg.role,
        content: msg.content,
        timestamp: msg.timestamp,
        providerMessageId: msg.message_id,
      }),
    );

    appendFileSync(filePath, lines.join("\n") + "\n", "utf8");
  }
}


path: src/cli/deep-consolidate.ts
content: #!/usr/bin/env npx tsx
/**
 * CLI script to run deep consolidation ("sleep pass") for all Memento agents.
 *
 * Usage:
 *   npx tsx src/cli/deep-consolidate.ts [--agent <id>]
 *
 * Without --agent, runs for all agents found in the DB.
 * Designed to be called from an OpenClaw cron job.
 */

import { resolve, join } from "node:path";
import { existsSync, readdirSync } from "node:fs";
import { homedir } from "node:os";
import {
  deepConsolidate,
  applyConfidenceDecay,
  DEFAULT_DEEP_CONFIG,
  type DeepConsolidationResult,
} from "../consolidation/deep-consolidator.js";
import { ConversationDB } from "../storage/db.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Auto-discover agent databases
// ---------------------------------------------------------------------------

/**
 * Discovers Memento databases for all agents by scanning common OpenClaw
 * workspace locations. Falls back to MEMENTO_AGENT_PATHS env var if set.
 *
 * Env var format: MEMENTO_AGENT_PATHS="main:/path/to/db,drjones:/path/to/db"
 */
function discoverAgentDBs(): Record<string, string> {
  // Allow explicit override via environment variable
  const envPaths = process.env.MEMENTO_AGENT_PATHS;
  if (envPaths) {
    const result: Record<string, string> = {};
    for (const entry of envPaths.split(",")) {
      const [agentId, dbPath] = entry.split(":");
      if (agentId && dbPath) result[agentId.trim()] = resolve(dbPath.trim());
    }
    return result;
  }

  // Auto-discover from standard OpenClaw locations
  const home = homedir();
  const agents: Record<string, string> = {};

  // Main agent: ~/.engram/conversations.sqlite
  const mainDb = join(home, ".engram", "conversations.sqlite");
  if (existsSync(mainDb)) agents["main"] = mainDb;

  // Other agents: scan ~/.openclaw/workspace-*/  and ~/*/
  const openclawDir = join(home, ".openclaw");
  if (existsSync(openclawDir)) {
    try {
      for (const entry of readdirSync(openclawDir, { withFileTypes: true })) {
        if (entry.isDirectory() && entry.name.startsWith("workspace-")) {
          const agentId = entry.name.replace("workspace-", "");
          const dbPath = join(openclawDir, entry.name, ".engram", "conversations.sqlite");
          if (existsSync(dbPath)) agents[agentId] = dbPath;
        }
      }
    } catch { /* permission errors are OK */ }
  }

  // Also scan home directories for agent workspaces (e.g., ~/bob/.engram/)
  try {
    for (const entry of readdirSync(home, { withFileTypes: true })) {
      if (entry.isDirectory() && !entry.name.startsWith(".")) {
        const dbPath = join(home, entry.name, ".engram", "conversations.sqlite");
        if (existsSync(dbPath) && !agents[entry.name]) {
          agents[entry.name] = dbPath;
        }
      }
    }
  } catch { /* permission errors are OK */ }

  return agents;
}

const AGENT_DB_PATHS = discoverAgentDBs();

// ---------------------------------------------------------------------------
// Simple logger
// ---------------------------------------------------------------------------

const logger: PluginLogger = {
  info: (msg: string) => console.log(`[INFO] ${msg}`),
  warn: (msg: string) => console.warn(`[WARN] ${msg}`),
  error: (msg: string) => console.error(`[ERROR] ${msg}`),
  debug: (msg: string) => {
    if (process.env.DEBUG) console.log(`[DEBUG] ${msg}`);
  },
};

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------

function main(): void {
  const args = process.argv.slice(2);
  const agentIdx = args.indexOf("--agent");
  const targetAgent = agentIdx !== -1 ? args[agentIdx + 1] : null;

  const agents = targetAgent
    ? { [targetAgent]: AGENT_DB_PATHS[targetAgent] }
    : AGENT_DB_PATHS;

  if (targetAgent && !AGENT_DB_PATHS[targetAgent]) {
    console.error(`Unknown agent: ${targetAgent}`);
    console.error(`Known agents: ${Object.keys(AGENT_DB_PATHS).join(", ")}`);
    process.exit(1);
  }

  const allResults: Record<string, DeepConsolidationResult> = {};
  let totalDecayed = 0;
  let totalCreated = 0;
  let totalUpdated = 0;
  let totalMerged = 0;
  let totalErrors = 0;

  for (const [agentId, dbPath] of Object.entries(agents)) {
    if (!existsSync(dbPath)) {
      logger.warn(`Skipping ${agentId} — DB not found at ${dbPath}`);
      continue;
    }

    logger.info(`Running deep consolidation for agent: ${agentId}`);

    try {
      const db = new ConversationDB(dbPath);

      // Step 1: Apply confidence decay
      const decayed = applyConfidenceDecay(
        db,
        agentId,
        DEFAULT_DEEP_CONFIG,
        logger,
      );
      totalDecayed += decayed;

      // Step 2: Run deep consolidation (cluster refresh, merging)
      const result = deepConsolidate(db, agentId, DEFAULT_DEEP_CONFIG, logger);
      allResults[agentId] = result;

      totalCreated += result.clustersCreated;
      totalUpdated += result.clustersUpdated;
      totalMerged += result.clustersMerged;
      totalErrors += result.errors.length;

      // Close DB
      (db as any).close?.();
    } catch (err) {
      logger.error(`Failed for ${agentId}: ${String(err)}`);
      totalErrors++;
    }
  }

  // Summary output (consumed by cron announce)
  const agentCount = Object.keys(allResults).length;
  const parts: string[] = [];

  if (totalDecayed > 0) parts.push(`${totalDecayed} facts decayed`);
  if (totalCreated > 0) parts.push(`${totalCreated} clusters created`);
  if (totalUpdated > 0) parts.push(`${totalUpdated} clusters updated`);
  if (totalMerged > 0) parts.push(`${totalMerged} clusters merged`);
  if (totalErrors > 0) parts.push(`${totalErrors} errors`);

  if (parts.length === 0) {
    console.log(
      `🧠 Deep consolidation complete — ${agentCount} agent(s), nothing to do.`,
    );
  } else {
    console.log(
      `🧠 Deep consolidation complete — ${agentCount} agent(s): ${parts.join(", ")}.`,
    );
  }

  if (totalErrors > 0) {
    for (const [agentId, result] of Object.entries(allResults)) {
      for (const err of result.errors) {
        console.error(`  [${agentId}] ${err}`);
      }
    }
  }
}

main();


path: src/cli/ingest-tokens.ts
content: #!/usr/bin/env node
/**
 * CLI for managing Memento ingest tokens.
 *
 * Usage:
 *   memento-token create --name "mac-mini-m2"
 *   memento-token list
 *   memento-token revoke <id>
 */

import { randomBytes, createHash, randomUUID } from "node:crypto";
import { existsSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
import { createRequire } from "node:module";

// ---------------------------------------------------------------------------
// Load better-sqlite3 (CJS module, load via createRequire for ESM compat)
// ---------------------------------------------------------------------------

import type BetterSQLite3 from "better-sqlite3";
type Database = BetterSQLite3.Database;
type DatabaseConstructor = typeof BetterSQLite3;

const _require = createRequire(import.meta.url);
const DatabaseCtor = _require("better-sqlite3") as DatabaseConstructor;

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

type TokenRow = {
  id: string;
  name: string;
  token_hash: string;
  created_at: number;
  last_used_at: number | null;
  is_active: number;
};

// ---------------------------------------------------------------------------
// DB helpers
// ---------------------------------------------------------------------------

const DB_PATH = join(homedir(), ".engram", "conversations.sqlite");

function openDB(): Database {
  if (!existsSync(DB_PATH)) {
    console.error(`✗ Database not found at ${DB_PATH}`);
    console.error("  Run OpenClaw at least once to initialize the database.");
    process.exit(1);
  }

  const db = new DatabaseCtor(DB_PATH);
  db.pragma("journal_mode = WAL");

  // Ensure the ingest_tokens table exists (idempotent)
  db.exec(`
    CREATE TABLE IF NOT EXISTS ingest_tokens (
      id          TEXT PRIMARY KEY,
      name        TEXT NOT NULL,
      token_hash  TEXT NOT NULL UNIQUE,
      created_at  INTEGER NOT NULL,
      last_used_at INTEGER,
      is_active   INTEGER DEFAULT 1
    );
  `);

  return db;
}

// ---------------------------------------------------------------------------
// Commands
// ---------------------------------------------------------------------------

function cmdCreate(args: string[]): void {
  const nameIdx = args.indexOf("--name");
  const name = nameIdx !== -1 ? args[nameIdx + 1] : undefined;

  if (!name || name.startsWith("--")) {
    console.error("Usage: memento-token create --name <name>");
    console.error("  Example: memento-token create --name mac-mini-m2");
    process.exit(1);
  }

  const rawToken = randomBytes(32).toString("hex");
  const tokenHash = createHash("sha256").update(rawToken).digest("hex");
  const id = randomUUID();

  const db = openDB();
  db.prepare(
    "INSERT INTO ingest_tokens (id, name, token_hash, created_at) VALUES (?, ?, ?, ?)"
  ).run(id, name, tokenHash, Date.now());
  db.close();

  console.log(`✓ Token created for "${name}"`);
  console.log(`  ID:    ${id}`);
  console.log(`  [REDACTED_SECRET]`);
  console.log("");
  console.log("  ⚠  Save this token now — it will NOT be shown again.");
}

function cmdList(): void {
  const db = openDB();
  const rows = db
    .prepare(
      "SELECT id, name, created_at, last_used_at, is_active FROM ingest_tokens ORDER BY created_at DESC"
    )
    .all() as TokenRow[];
  db.close();

  if (rows.length === 0) {
    console.log("No ingest tokens found.");
    return;
  }

  console.log("Ingest tokens:\n");
  for (const row of rows) {
    const status = row.is_active ? "✓ active " : "✗ revoked";
    const created = new Date(row.created_at).toISOString().slice(0, 19).replace("T", " ");
    const lastUsed = row.last_used_at
      ? new Date(row.last_used_at).toISOString().slice(0, 19).replace("T", " ")
      : "never";
    console.log(`  ${status}  ${row.id}`);
    console.log(`            name: ${row.name}`);
    console.log(`         created: ${created}`);
    console.log(`       last used: ${lastUsed}`);
    console.log("");
  }
}

function cmdRevoke(args: string[]): void {
  const id = args[0];

  if (!id || id.startsWith("--")) {
    console.error("Usage: memento-token revoke <id>");
    process.exit(1);
  }

  const db = openDB();
  const result = db
    .prepare("UPDATE ingest_tokens SET is_active = 0 WHERE id = ?")
    .run(id);
  db.close();

  if (result.changes === 0) {
    console.error(`✗ Token not found: ${id}`);
    process.exit(1);
  }

  console.log(`✓ Token revoked: ${id}`);
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------

const [, , command, ...rest] = process.argv;

switch (command) {
  case "create":
    cmdCreate(rest);
    break;
  case "list":
    cmdList();
    break;
  case "revoke":
    cmdRevoke(rest);
    break;
  default:
    console.error("Memento ingest token manager\n");
    console.error("Usage: memento-token <command> [options]\n");
    console.error("Commands:");
    console.error("  create --name <name>   Create a new ingest token");
    console.error("  list                   List all tokens");
    console.error("  revoke <id>            Revoke a token by ID");
    process.exit(command ? 1 : 0);
}


path: src/cli/process-raw-sessions.ts
content: #!/usr/bin/env node
/**
 * CLI to process raw Claude session JSONL files from ~/.engram/raw-sessions/.
 * Extracts facts via LLM and saves them to the Memento DB.
 *
 * Usage:
 *   node dist/cli/process-raw-sessions.js [--dry-run] [--limit N] [--agent <id>]
 *
 * Options:
 *   --dry-run     Parse and show what would be extracted, don't save or move files.
 *   --limit N     Process only N files (useful for testing).
 *   --agent <id>  Which agent_id to attribute facts to (default: "main").
 */

import {
  readdirSync,
  readFileSync,
  mkdirSync,
  renameSync,
  existsSync,
} from "node:fs";
import { join, basename } from "node:path";
import { homedir } from "node:os";
import { randomUUID } from "node:crypto";
import { createRequire } from "node:module";
import { ConversationDB } from "../storage/db.js";
import { extractFacts } from "../extraction/extractor.js";
import { processExtractedFacts } from "../extraction/deduplicator.js";
import type { PluginLogger } from "../types.js";
import type { ConversationRow } from "../storage/schema.js";

// ---------------------------------------------------------------------------
// Config
// ---------------------------------------------------------------------------

const RAW_SESSIONS_DIR = join(homedir(), ".engram", "raw-sessions");
const PROCESSED_DIR = join(RAW_SESSIONS_DIR, ".processed");
const DB_PATH = join(homedir(), ".engram", "conversations.sqlite");
const DEFAULT_MODEL = "anthropic/claude-sonnet-4-6";

// ---------------------------------------------------------------------------
// Load better-sqlite3 for reading schema_meta (model config)
// ---------------------------------------------------------------------------

import type BetterSQLite3 from "better-sqlite3";
type Database = BetterSQLite3.Database;
type DatabaseConstructor = typeof BetterSQLite3;

const _require = createRequire(import.meta.url);
const DatabaseCtor = _require("better-sqlite3") as DatabaseConstructor;

// ---------------------------------------------------------------------------
// Simple logger
// ---------------------------------------------------------------------------

const logger: PluginLogger = {
  info: (msg: string) => console.log(`[INFO] ${msg}`),
  warn: (msg: string) => console.warn(`[WARN] ${msg}`),
  error: (msg: string) => console.error(`[ERROR] ${msg}`),
  debug: (msg: string) => {
    if (process.env.DEBUG) console.log(`[DEBUG] ${msg}`);
  },
};

// ---------------------------------------------------------------------------
// JSONL parsing types
// ---------------------------------------------------------------------------

type ContentBlock = {
  type?: string;
  text?: string;
};

type SessionLine = {
  type?: string;
  message?: {
    role?: string;
    content?: string | ContentBlock[];
  };
};

// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------

function extractTextContent(content: string | ContentBlock[]): string {
  if (typeof content === "string") return content.trim();
  return content
    .filter((b) => b.type === "text" && typeof b.text === "string")
    .map((b) => (b.text as string).trim())
    .join("\n")
    .trim();
}

/**
 * Parse a raw Claude session JSONL file into a flat conversation string.
 * Returns the conversation text and message count.
 */
function parseSessionFile(
  filePath: string,
): { text: string; turnCount: number } {
  const raw = readFileSync(filePath, "utf-8");
  const lines = raw.split("\n");
  const parts: string[] = [];
  let turnCount = 0;

  for (const line of lines) {
    const trimmed = line.trim();
    if (!trimmed) continue;

    let obj: SessionLine;
    try {
      obj = JSON.parse(trimmed) as SessionLine;
    } catch {
      logger.warn(`  Skipping malformed line in ${basename(filePath)}`);
      continue;
    }

    const type = obj?.type;
    if (type !== "user" && type !== "assistant") continue;

    const content = obj?.message?.content;
    if (content === undefined || content === null) continue;

    const text = extractTextContent(
      content as string | ContentBlock[],
    );
    if (!text) continue;

    const role = type === "user" ? "User" : "Assistant";
    parts.push(`[${role}]: ${text}`);
    turnCount++;
  }

  return { text: parts.join("\n\n"), turnCount };
}

/**
 * Try to read the extraction model from schema_meta table.
 * Falls back to DEFAULT_MODEL if not found or DB unavailable.
 */
function readExtractionModel(): string {
  if (!existsSync(DB_PATH)) return DEFAULT_MODEL;
  let db: Database | null = null;
  try {
    db = new DatabaseCtor(DB_PATH, { readonly: true });
    const row = db
      .prepare("SELECT value FROM schema_meta WHERE key = 'extractionModel'")
      .get() as { value: string } | undefined;
    return row?.value ?? DEFAULT_MODEL;
  } catch {
    return DEFAULT_MODEL;
  } finally {
    try { db?.close(); } catch { /* ignore */ }
  }
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------

async function main(): Promise<void> {
  // Parse CLI args
  const args = process.argv.slice(2);
  const dryRun = args.includes("--dry-run");
  const limitIdx = args.indexOf("--limit");
  const limit = limitIdx !== -1 ? parseInt(args[limitIdx + 1]!, 10) : Infinity;
  const agentIdx = args.indexOf("--agent");
  const agentId = agentIdx !== -1 ? (args[agentIdx + 1] ?? "main") : "main";

  if (dryRun) {
    console.log("DRY RUN — no facts will be saved, no files will be moved.");
  }

  // Validate raw-sessions directory
  if (!existsSync(RAW_SESSIONS_DIR)) {
    console.error(`Raw sessions directory does not exist: ${RAW_SESSIONS_DIR}`);
    process.exit(1);
  }

  // Scan for JSONL files (skip .processed/ subdirectory)
  let files = readdirSync(RAW_SESSIONS_DIR)
    .filter((f) => f.endsWith(".jsonl"))
    .sort()
    .map((f) => join(RAW_SESSIONS_DIR, f));

  if (files.length === 0) {
    console.log(`No JSONL files found in ${RAW_SESSIONS_DIR}`);
    return;
  }

  const totalFiles = files.length;
  if (isFinite(limit) && limit < files.length) {
    files = files.slice(0, limit);
  }

  console.log(
    `Found ${totalFiles} JSONL file(s) in raw-sessions/. Processing ${files.length}...`,
  );

  // Read extraction model config
  const extractionModel = readExtractionModel();
  console.log(`Extraction model: ${extractionModel}`);
  console.log(`Agent ID: ${agentId}`);
  console.log("");

  // Open Memento DB
  const db = new ConversationDB(DB_PATH);

  // Ensure .processed dir exists
  if (!dryRun) {
    mkdirSync(PROCESSED_DIR, { recursive: true });
  }

  let totalExtracted = 0;
  let totalUpdated = 0;
  let totalDeduplicated = 0;
  let totalErrors = 0;
  let totalSkipped = 0;
  let processedCount = 0;

  for (let i = 0; i < files.length; i++) {
    const filePath = files[i]!;
    const fileName = basename(filePath);
    console.log(`Processing file ${i + 1}/${files.length}: ${fileName}...`);

    // Parse the raw session JSONL
    let sessionData: { text: string; turnCount: number };
    try {
      sessionData = parseSessionFile(filePath);
    } catch (err) {
      logger.error(`  Failed to read/parse ${fileName}: ${String(err)}`);
      totalErrors++;
      if (!dryRun) {
        renameSync(filePath, join(PROCESSED_DIR, fileName));
      }
      continue;
    }

    const { text, turnCount } = sessionData;

    // Skip files with fewer than 3 meaningful exchanges (< 6 message parts)
    if (turnCount < 6) {
      logger.warn(
        `  Skipping ${fileName} — only ${turnCount} message parts (min: 6)`,
      );
      totalSkipped++;
      if (!dryRun) {
        renameSync(filePath, join(PROCESSED_DIR, fileName));
      }
      continue;
    }

    // Truncate very large conversations to last 8000 chars
    const conversationText =
      text.length > 8000 ? text.slice(-8000) : text;

    if (dryRun) {
      console.log(
        `  → Would extract from ${turnCount} messages (${conversationText.length} chars)`,
      );
      console.log(
        `  Preview: ${conversationText.slice(0, 200).replace(/\n/g, " ")}...`,
      );
      totalSkipped++;
      continue;
    }

    // Build a synthetic ConversationRow for the extractor
    const now = Date.now();
    const conversationId = randomUUID();
    const convRow: ConversationRow = {
      id: conversationId,
      agent_id: agentId,
      session_key: `raw-session:${fileName}`,
      channel: "raw-ingest",
      started_at: now,
      ended_at: now,
      turn_count: Math.floor(turnCount / 2),
      raw_text: conversationText,
      metadata: null,
    };

    // Insert conversation row so FK references in fact_occurrences / extraction_log work
    db.insertConversationWithMessages(convRow, []);

    // Fetch existing facts for dedup context (pass up to 50)
    const existingFacts = db.getRelevantFacts(agentId, 50);

    // Call LLM extractor
    let extractionResult: Awaited<ReturnType<typeof extractFacts>>;
    try {
      extractionResult = await extractFacts(
        convRow,
        existingFacts,
        extractionModel,
        logger,
      );
    } catch (err) {
      logger.error(`  Extraction threw for ${fileName}: ${String(err)}`);
      totalErrors++;
      renameSync(filePath, join(PROCESSED_DIR, fileName));
      // Rate limit: pause before next call
      if (i < files.length - 1) await sleep(1000);
      continue;
    }

    if (extractionResult.error) {
      logger.error(
        `  Extraction failed for ${fileName}: ${extractionResult.error}`,
      );
      totalErrors++;
      renameSync(filePath, join(PROCESSED_DIR, fileName));
      if (i < files.length - 1) await sleep(1000);
      continue;
    }

    // Save extracted facts via deduplicator
    const dedup = await processExtractedFacts(
      extractionResult.facts,
      conversationId,
      agentId,
      db,
      null, // embeddingEngine — not available in CLI mode
      logger,
    );

    console.log(
      `  → extracted ${dedup.factsExtracted} new, ` +
        `${dedup.factsUpdated} updated, ` +
        `${dedup.factsDeduplicated} deduplicated`,
    );

    totalExtracted += dedup.factsExtracted;
    totalUpdated += dedup.factsUpdated;
    totalDeduplicated += dedup.factsDeduplicated;
    processedCount++;

    // Move processed file to .processed/
    renameSync(filePath, join(PROCESSED_DIR, fileName));

    // Rate limit: 1-second delay between LLM calls
    if (i < files.length - 1) {
      await sleep(1000);
    }
  }

  // Summary
  console.log("\n--- Summary ---");
  console.log(`Files processed:    ${processedCount}`);
  console.log(`Files skipped:      ${totalSkipped}`);
  console.log(`Errors:             ${totalErrors}`);
  console.log(`Facts extracted:    ${totalExtracted}`);
  console.log(`Facts updated:      ${totalUpdated}`);
  console.log(`Facts deduplicated: ${totalDeduplicated}`);
}

main().catch((err: unknown) => {
  console.error("Fatal error:", err);
  process.exit(1);
});


path: src/config.ts
content: // ---------------------------------------------------------------------------
// Recall configuration (Phase 3)
// ---------------------------------------------------------------------------

export type RecallConfig = {
  /** Whether to automatically inject recalled facts before each AI turn. Default: true */
  autoRecall: boolean;
  /** Maximum number of facts to inject per turn. Default: 20 */
  maxFacts: number;
  /** Maximum characters for the injected context block. Default: 4000 */
  maxContextChars: number;
  /** Minimum message length (chars) to trigger recall search. Default: 5 */
  minQueryLength: number;
  /** Whether to include shared facts from other agents (master KB). Default: true */
  crossAgentRecall: boolean;
  /**
   * Whether to use LLM-based query planning before retrieval (AMA-Agent inspired).
   * When enabled, calls the LLM to expand the query with synonyms/entities for
   * better FTS matching. Adds a small LLM call overhead per recall. Default: false
   */
  autoQueryPlanning: boolean;
};

export const DEFAULT_RECALL_CONFIG: RecallConfig = {
  autoRecall: true,
  maxFacts: 20,
  maxContextChars: 4000,
  minQueryLength: 5,
  crossAgentRecall: true,
  autoQueryPlanning: false,
};

// ---------------------------------------------------------------------------
// Extraction configuration (Phase 2)
// ---------------------------------------------------------------------------

export type ExtractionConfig = {
  /** Minimum number of turns in a segment before extraction is attempted. Default: 3 */
  minTurnsForExtraction: number;
  /** Max LLM extraction calls per minute (rate limiting). Default: 10 */
  maxExtractionsPerMinute: number;
  /** How many existing facts to include in the extraction prompt for dedup context. Default: 50 */
  includeExistingFactsCount: number;
  /** Whether to automatically extract facts after each segment capture. Default: true */
  autoExtract: boolean;
};

export const DEFAULT_EXTRACTION_CONFIG: ExtractionConfig = {
  minTurnsForExtraction: 3,
  maxExtractionsPerMinute: 10,
  includeExistingFactsCount: 50,
  autoExtract: false, // Opt-in: enabling this sends conversation text to your configured LLM provider
};

// ---------------------------------------------------------------------------
// Top-level plugin configuration
// ---------------------------------------------------------------------------

export type PluginConfig = {
  /** Idle time (ms) before flushing a conversation segment. Default: 5 min */
  pauseTimeoutMs: number;
  /** Maximum message turns before auto-flush. Default: 50 */
  maxBufferTurns: number;
  /** Whether to automatically capture all conversation messages. Default: true */
  autoCapture: boolean;
  /** Directory for storing data, relative to workspace. Default: .engram */
  dataDir: string;
  /**
   * LLM model for fact extraction. Default: "anthropic/claude-sonnet-4-6"
   *
   * Supported provider prefixes — use whichever you have an API key for:
   *   - "anthropic/<model>"  e.g. "anthropic/claude-sonnet-4-6"
   *   - "openai/<model>"     e.g. "openai/gpt-4o"
   *   - "mistral/<model>"    e.g. "mistral/mistral-large-latest"
   *   - "ollama/<model>"     e.g. "ollama/llama3" (local, no key required)
   *   - Any other string is tried as an OpenAI-compatible endpoint (fallback).
   *
   * Credentials are resolved from environment variables:
   *   ANTHROPIC_API_KEY | OPENAI_API_KEY | MISTRAL_API_KEY | MEMENTO_API_KEY
   */
  extractionModel: string;
  /** Local embedding model for semantic search (Phase 5) */
  embeddingModel: string;
  /**
   * Human-readable display names for agents shown in cross-agent recall context.
   * Map of agentId → display name. Defaults to showing the raw agentId.
   * Example: { "main": "Assistant", "medbot": "Medical Assistant" }
   */
  agentDisplay: Record<string, string>;
  /** Extraction engine configuration (Phase 2) */
  extraction: ExtractionConfig;
  /** Recall / auto-inject configuration (Phase 3) */
  recall: RecallConfig;
};

export const DEFAULT_CONFIG: PluginConfig = {
  pauseTimeoutMs: 300_000,
  maxBufferTurns: 50,
  autoCapture: true,
  dataDir: ".engram", // Default data directory; existing deployments use this path.
  extractionModel: "anthropic/claude-sonnet-4-6",
  embeddingModel: "hf:BAAI/bge-m3-gguf",
  agentDisplay: {}, // populated via plugin config: { "main": "Main Agent", "medbot": "Medical Bot" }
  extraction: DEFAULT_EXTRACTION_CONFIG,
  recall: DEFAULT_RECALL_CONFIG,
};

export function resolveConfig(raw?: Record<string, unknown>): PluginConfig {
  if (!raw) return { ...DEFAULT_CONFIG, extraction: { ...DEFAULT_EXTRACTION_CONFIG }, recall: { ...DEFAULT_RECALL_CONFIG } };

  const rawExtraction =
    raw.extraction && typeof raw.extraction === "object"
      ? (raw.extraction as Record<string, unknown>)
      : {};

  const rawRecall =
    raw.recall && typeof raw.recall === "object"
      ? (raw.recall as Record<string, unknown>)
      : {};

  return {
    pauseTimeoutMs:
      typeof raw.pauseTimeoutMs === "number"
        ? raw.pauseTimeoutMs
        : DEFAULT_CONFIG.pauseTimeoutMs,
    maxBufferTurns:
      typeof raw.maxBufferTurns === "number"
        ? raw.maxBufferTurns
        : DEFAULT_CONFIG.maxBufferTurns,
    autoCapture:
      typeof raw.autoCapture === "boolean"
        ? raw.autoCapture
        : DEFAULT_CONFIG.autoCapture,
    dataDir:
      typeof raw.dataDir === "string" ? raw.dataDir : DEFAULT_CONFIG.dataDir,
    extractionModel:
      typeof raw.extractionModel === "string"
        ? raw.extractionModel
        : DEFAULT_CONFIG.extractionModel,
    embeddingModel:
      typeof raw.embeddingModel === "string"
        ? raw.embeddingModel
        : DEFAULT_CONFIG.embeddingModel,
    agentDisplay:
      raw.agentDisplay && typeof raw.agentDisplay === "object" && !Array.isArray(raw.agentDisplay)
        ? (raw.agentDisplay as Record<string, string>)
        : DEFAULT_CONFIG.agentDisplay,
    extraction: {
      minTurnsForExtraction:
        typeof rawExtraction.minTurnsForExtraction === "number"
          ? rawExtraction.minTurnsForExtraction
          : DEFAULT_EXTRACTION_CONFIG.minTurnsForExtraction,
      maxExtractionsPerMinute:
        typeof rawExtraction.maxExtractionsPerMinute === "number"
          ? rawExtraction.maxExtractionsPerMinute
          : DEFAULT_EXTRACTION_CONFIG.maxExtractionsPerMinute,
      includeExistingFactsCount:
        typeof rawExtraction.includeExistingFactsCount === "number"
          ? rawExtraction.includeExistingFactsCount
          : DEFAULT_EXTRACTION_CONFIG.includeExistingFactsCount,
      autoExtract:
        typeof rawExtraction.autoExtract === "boolean"
          ? rawExtraction.autoExtract
          : DEFAULT_EXTRACTION_CONFIG.autoExtract,
    },
    recall: {
      autoRecall:
        typeof rawRecall.autoRecall === "boolean"
          ? rawRecall.autoRecall
          : DEFAULT_RECALL_CONFIG.autoRecall,
      maxFacts:
        typeof rawRecall.maxFacts === "number"
          ? rawRecall.maxFacts
          : DEFAULT_RECALL_CONFIG.maxFacts,
      maxContextChars:
        typeof rawRecall.maxContextChars === "number"
          ? rawRecall.maxContextChars
          : DEFAULT_RECALL_CONFIG.maxContextChars,
      minQueryLength:
        typeof rawRecall.minQueryLength === "number"
          ? rawRecall.minQueryLength
          : DEFAULT_RECALL_CONFIG.minQueryLength,
      crossAgentRecall:
        typeof rawRecall.crossAgentRecall === "boolean"
          ? rawRecall.crossAgentRecall
          : DEFAULT_RECALL_CONFIG.crossAgentRecall,
      autoQueryPlanning:
        typeof rawRecall.autoQueryPlanning === "boolean"
          ? rawRecall.autoQueryPlanning
          : DEFAULT_RECALL_CONFIG.autoQueryPlanning,
    },
  };
}


path: src/consolidation/consolidator.ts
content: /**
 * Consolidation Engine — Multi-Layer Memory
 *
 * Groups related facts into clusters (Layer 2), and clusters into
 * higher-level clusters (Layer 3+). Operates in two modes:
 *
 *   1. **Incremental** — called after each extraction to quickly assign
 *      new facts to existing clusters or create small new ones based on
 *      category proximity and graph edges. Cheap, fast, real-time.
 *
 *   2. **Deep ("sleep")** — periodic batch job that reviews all unclustered
 *      facts, re-evaluates cluster boundaries, merges related clusters,
 *      generates richer summaries, and builds higher-layer abstractions.
 *      Uses the LLM for semantic grouping. Expensive, thorough.
 *
 * Visibility rule: a cluster inherits the most restrictive visibility
 * of any of its members (shared < private < secret).
 */

import { randomUUID } from "node:crypto";
import type { ConversationDB } from "../storage/db.js";
import type { FactRow, FactClusterRow, ClusterMemberRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Visibility helpers
// ---------------------------------------------------------------------------

function visibilityLevel(vis: string): number {
  switch (vis) {
    case "shared": return 0;
    case "private": return 1;
    case "secret": return 2;
    default: return 0;
  }
}

function visibilityFromLevel(level: number): "shared" | "private" | "secret" {
  switch (level) {
    case 2: return "secret";
    case 1: return "private";
    default: return "shared";
  }
}

/**
 * Determine the most restrictive visibility across a set of facts.
 */
function inheritVisibility(facts: FactRow[]): "shared" | "private" | "secret" {
  let maxLevel = 0;
  for (const f of facts) {
    maxLevel = Math.max(maxLevel, visibilityLevel(f.visibility));
  }
  return visibilityFromLevel(maxLevel);
}

// ---------------------------------------------------------------------------
// Incremental consolidation
// ---------------------------------------------------------------------------

export type IncrementalConsolidationResult = {
  /** Number of facts assigned to existing clusters */
  assignedToExisting: number;
  /** Number of new clusters created */
  newClusters: number;
  /** IDs of affected clusters (new or updated) */
  affectedClusterIds: string[];
};

/**
 * Incremental consolidation — run after each extraction.
 *
 * Strategy:
 *   1. Get unclustered facts for this agent
 *   2. For each unclustered fact, check if it has graph edges to any
 *      already-clustered facts → assign to that cluster
 *   3. Group remaining unclustered facts by category
 *   4. If a category group has ≥ 3 facts, create a new cluster
 *
 * This is intentionally simple and fast — no LLM calls.
 */
export function incrementalConsolidate(
  db: ConversationDB,
  agentId: string,
  logger: PluginLogger,
): IncrementalConsolidationResult {
  const now = Date.now();
  let assignedToExisting = 0;
  let newClusters = 0;
  const affectedClusterIds: string[] = [];

  const unclustered = db.getUnclusteredFacts(agentId, 200);
  if (unclustered.length === 0) {
    return { assignedToExisting: 0, newClusters: 0, affectedClusterIds: [] };
  }

  logger.debug?.(
    `memento: incremental consolidation — ${unclustered.length} unclustered facts for agent ${agentId}`,
  );

  const remaining: FactRow[] = [];

  // Pass 1: Assign facts to existing clusters via graph edges
  for (const fact of unclustered) {
    const relations = db.getRelationsForFact(fact.id);
    let bestCluster: FactClusterRow | null = null;
    let bestStrength = 0;

    for (const rel of relations) {
      const neighborId = rel.source_id === fact.id ? rel.target_id : rel.source_id;
      const clusters = db.getClustersForFact(neighborId);

      for (const cluster of clusters) {
        // Only join clusters from the same agent
        if (cluster.agent_id !== agentId) continue;
        // Only join Layer 2 clusters (don't skip levels)
        if (cluster.layer !== 2) continue;

        if (rel.strength > bestStrength) {
          bestStrength = rel.strength;
          bestCluster = cluster;
        }
      }
    }

    if (bestCluster && bestStrength >= 0.5) {
      // Assign to existing cluster
      db.addClusterMember({
        cluster_id: bestCluster.id,
        member_id: fact.id,
        member_type: "fact",
        added_at: now,
      });

      // Update cluster visibility (may become more restrictive)
      const clusterFacts = db.getClusterFacts(bestCluster.id);
      const newVis = inheritVisibility([...clusterFacts, fact]);
      if (newVis !== bestCluster.visibility) {
        db.updateCluster(bestCluster.id, { visibility: newVis });
      }

      if (!affectedClusterIds.includes(bestCluster.id)) {
        affectedClusterIds.push(bestCluster.id);
      }
      assignedToExisting++;
      logger.debug?.(
        `memento: assigned fact ${fact.id} to cluster ${bestCluster.id} (strength: ${bestStrength})`,
      );
    } else {
      remaining.push(fact);
    }
  }

  // Pass 2: Group remaining unclustered facts by category
  // Create new clusters when a category has ≥ 3 unclustered facts
  const byCategory = new Map<string, FactRow[]>();
  for (const fact of remaining) {
    const list = byCategory.get(fact.category) ?? [];
    list.push(fact);
    byCategory.set(fact.category, list);
  }

  for (const [category, facts] of byCategory) {
    if (facts.length < 3) continue;

    const clusterId = randomUUID();
    const visibility = inheritVisibility(facts);
    const avgConfidence =
      facts.reduce((sum, f) => sum + f.confidence, 0) / facts.length;

    // Generate a simple summary from the category + fact summaries
    const factSummaries = facts
      .map((f) => f.summary ?? f.content.slice(0, 60))
      .slice(0, 5)
      .join("; ");

    const cluster: FactClusterRow = {
      id: clusterId,
      agent_id: agentId,
      summary: `${categoryLabel(category)} — ${facts.length} facts`,
      description: `Auto-clustered from category "${category}": ${factSummaries}`,
      layer: 2,
      visibility,
      confidence: avgConfidence,
      created_at: now,
      updated_at: now,
      is_active: 1,
      metadata: JSON.stringify({ source: "incremental", category }),
    };

    db.insertCluster(cluster);

    const members: ClusterMemberRow[] = facts.map((f) => ({
      cluster_id: clusterId,
      member_id: f.id,
      member_type: "fact" as const,
      added_at: now,
    }));

    db.addClusterMembersBatch(members);
    affectedClusterIds.push(clusterId);
    newClusters++;

    logger.debug?.(
      `memento: created cluster ${clusterId} for category "${category}" with ${facts.length} facts`,
    );
  }

  if (assignedToExisting > 0 || newClusters > 0) {
    logger.info(
      `memento: incremental consolidation — assigned ${assignedToExisting} to existing, ` +
        `created ${newClusters} new clusters (agent: ${agentId})`,
    );
  }

  return { assignedToExisting, newClusters, affectedClusterIds };
}

// ---------------------------------------------------------------------------
// Category display label (reused from context-builder pattern)
// ---------------------------------------------------------------------------

function categoryLabel(cat: string): string {
  const labels: Record<string, string> = {
    decision: "Decisions",
    correction: "Corrections",
    action_item: "Action Items",
    preference: "Preferences",
    person: "People",
    technical: "Technical",
    emotional: "Emotional Context",
    routine: "Routines",
  };
  return labels[cat] ?? cat.charAt(0).toUpperCase() + cat.slice(1);
}


path: src/consolidation/deep-consolidator.ts
content: /**
 * Deep Consolidation — The "Sleep" Pass
 *
 * Periodic batch job that performs thorough knowledge base maintenance:
 *
 *   1. Reviews all unclustered facts and uses the LLM to propose semantic
 *      groupings that go beyond simple category matching.
 *   2. Re-evaluates existing cluster summaries — regenerates them from
 *      current member facts for accuracy.
 *   3. Merges overlapping clusters when their member sets converge.
 *   4. Builds Layer 3+ clusters from groups of related Layer 2 clusters.
 *   5. Applies confidence decay to stale facts (soft, not deletion).
 *
 * This is the expensive pass — it uses LLM calls for semantic analysis.
 * Designed to run once or twice per day (like human sleep consolidation).
 */

import { randomUUID } from "node:crypto";
import type { ConversationDB } from "../storage/db.js";
import type { FactRow, FactClusterRow, ClusterMemberRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

export type DeepConsolidationConfig = {
  /** LLM model string for semantic grouping. Same format as extractionModel. */
  model: string;
  /** Maximum unclustered facts to process per run */
  maxFactsPerRun: number;
  /** Confidence decay rate per day for facts not seen in 30+ days */
  decayRatePerDay: number;
  /** Minimum confidence floor — facts never decay below this */
  decayFloor: number;
  /** Days of inactivity before decay starts */
  decayGraceDays: number;
};

export const DEFAULT_DEEP_CONFIG: DeepConsolidationConfig = {
  model: "anthropic/claude-sonnet-4-6",
  maxFactsPerRun: 100,
  decayRatePerDay: 0.005, // 0.5% per day after grace period
  decayFloor: 0.3,
  decayGraceDays: 30,
};

export type DeepConsolidationResult = {
  clustersCreated: number;
  clustersUpdated: number;
  clustersMerged: number;
  factsDecayed: number;
  errors: string[];
};

// ---------------------------------------------------------------------------
// Confidence decay
// ---------------------------------------------------------------------------

/**
 * Apply soft confidence decay to facts that haven't been seen recently.
 *
 * Rules:
 *   - Facts not seen in `decayGraceDays` start losing confidence
 *   - Decay is linear: `decayRatePerDay` per day past the grace period
 *   - Confidence never drops below `decayFloor`
 *   - Facts in clusters decay slower (×0.5 rate) — reinforced by grouping
 *   - High-occurrence facts decay slower (×0.7 rate for 3+ occurrences)
 */
export function applyConfidenceDecay(
  db: ConversationDB,
  agentId: string,
  cfg: DeepConsolidationConfig,
  logger: PluginLogger,
): number {
  const now = Date.now();
  const gracePeriodMs = cfg.decayGraceDays * 24 * 60 * 60 * 1000;
  const decayThreshold = now - gracePeriodMs;

  // Get all active facts that haven't been seen since the grace threshold
  const staleFacts = db.getRelevantFacts(agentId, 1000).filter(
    (f) => f.last_seen_at < decayThreshold && f.confidence > cfg.decayFloor,
  );

  let decayed = 0;

  for (const fact of staleFacts) {
    const daysSinceGrace =
      (now - fact.last_seen_at - gracePeriodMs) / (24 * 60 * 60 * 1000);
    if (daysSinceGrace <= 0) continue;

    // Decay modifiers
    let rateMultiplier = 1.0;

    // Facts in clusters decay slower
    const clusters = db.getClustersForFact(fact.id);
    if (clusters.length > 0) rateMultiplier *= 0.5;

    // High-occurrence facts decay slower
    if (fact.occurrence_count >= 3) rateMultiplier *= 0.7;

    const decay = cfg.decayRatePerDay * daysSinceGrace * rateMultiplier;
    const newConfidence = Math.max(cfg.decayFloor, fact.confidence - decay);

    if (newConfidence < fact.confidence) {
      // Use direct DB access to update confidence
      // (We can't easily add a dedicated method without modifying db.ts further,
      // but we can reuse the pattern from the MC feedback handler)
      try {
        // Update via the existing pattern — insertFact won't work, we need a raw update
        // For now, we'll use the getRelatedFacts pattern to get the DB handle
        // Actually, let's just add a simple helper
        db.updateFactConfidence(fact.id, newConfidence);
        decayed++;
      } catch (err) {
        logger.warn(
          `memento: decay error for fact ${fact.id}: ${String(err)}`,
        );
      }
    }
  }

  if (decayed > 0) {
    logger.info(
      `memento: confidence decay — ${decayed} facts decayed (agent: ${agentId})`,
    );
  }

  return decayed;
}

// ---------------------------------------------------------------------------
// LLM-assisted semantic clustering (placeholder for full implementation)
// ---------------------------------------------------------------------------

/**
 * Deep consolidation entry point.
 *
 * Currently implements:
 *   - Confidence decay (no LLM needed)
 *   - Cluster summary refresh (from member facts, no LLM needed)
 *
 * TODO (future iterations):
 *   - LLM-assisted semantic grouping of unclustered facts
 *   - Cluster merging when member overlap exceeds threshold
 *   - Layer 3+ meta-cluster creation
 *   - LLM-generated rich cluster summaries
 */
export function deepConsolidate(
  db: ConversationDB,
  agentId: string,
  cfg: DeepConsolidationConfig,
  logger: PluginLogger,
): DeepConsolidationResult {
  const result: DeepConsolidationResult = {
    clustersCreated: 0,
    clustersUpdated: 0,
    clustersMerged: 0,
    factsDecayed: 0,
    errors: [],
  };

  // ── Step 1: Confidence decay ────────────────────────────────────────
  try {
    result.factsDecayed = applyConfidenceDecay(db, agentId, cfg, logger);
  } catch (err) {
    const msg = `Confidence decay failed: ${String(err)}`;
    result.errors.push(msg);
    logger.warn(`memento: ${msg}`);
  }

  // ── Step 2: Refresh cluster summaries ───────────────────────────────
  try {
    const clusters = db.getClusters(agentId, 2);
    for (const cluster of clusters) {
      const facts = db.getClusterFacts(cluster.id);
      if (facts.length === 0) {
        // Empty cluster — deactivate it
        db.deactivateCluster(cluster.id);
        result.clustersUpdated++;
        continue;
      }

      // Recalculate visibility from current members
      let maxVisLevel = 0;
      for (const f of facts) {
        const vl = f.visibility === "secret" ? 2 : f.visibility === "private" ? 1 : 0;
        maxVisLevel = Math.max(maxVisLevel, vl);
      }
      const newVis = maxVisLevel === 2 ? "secret" : maxVisLevel === 1 ? "private" : "shared";

      // Regenerate summary from member facts
      const factSummaries = facts
        .sort((a, b) => b.confidence - a.confidence)
        .map((f) => f.summary ?? f.content.slice(0, 80))
        .slice(0, 8);

      const newSummary = `${facts.length} facts: ${factSummaries.slice(0, 3).join("; ")}`;
      const newDesc = factSummaries.join("; ");

      // Average confidence from members
      const avgConf = facts.reduce((s, f) => s + f.confidence, 0) / facts.length;

      const needsUpdate =
        newVis !== cluster.visibility ||
        Math.abs(avgConf - cluster.confidence) > 0.01;

      if (needsUpdate) {
        db.updateCluster(cluster.id, {
          summary: newSummary,
          description: newDesc,
          visibility: newVis,
          confidence: avgConf,
        });
        result.clustersUpdated++;
      }
    }
  } catch (err) {
    const msg = `Cluster refresh failed: ${String(err)}`;
    result.errors.push(msg);
    logger.warn(`memento: ${msg}`);
  }

  // ── Step 3: Merge overlapping clusters ──────────────────────────────
  try {
    const clusters = db.getClusters(agentId, 2);
    const clusterMembers = new Map<string, Set<string>>();

    for (const cluster of clusters) {
      const members = db.getClusterMembers(cluster.id);
      clusterMembers.set(
        cluster.id,
        new Set(members.map((m) => m.member_id)),
      );
    }

    const merged = new Set<string>();

    for (let i = 0; i < clusters.length; i++) {
      if (merged.has(clusters[i].id)) continue;
      const setA = clusterMembers.get(clusters[i].id)!;

      for (let j = i + 1; j < clusters.length; j++) {
        if (merged.has(clusters[j].id)) continue;
        const setB = clusterMembers.get(clusters[j].id)!;

        // Calculate Jaccard similarity
        let intersection = 0;
        for (const id of setA) {
          if (setB.has(id)) intersection++;
        }
        const union = setA.size + setB.size - intersection;
        const jaccard = union > 0 ? intersection / union : 0;

        // Merge if >60% overlap
        if (jaccard > 0.6) {
          // Merge B into A
          const now = Date.now();
          for (const memberId of setB) {
            if (!setA.has(memberId)) {
              db.addClusterMember({
                cluster_id: clusters[i].id,
                member_id: memberId,
                member_type: "fact",
                added_at: now,
              });
              setA.add(memberId);
            }
          }
          db.deactivateCluster(clusters[j].id);
          merged.add(clusters[j].id);
          result.clustersMerged++;

          logger.debug?.(
            `memento: merged cluster ${clusters[j].id} into ${clusters[i].id} (jaccard: ${jaccard.toFixed(2)})`,
          );
        }
      }
    }
  } catch (err) {
    const msg = `Cluster merging failed: ${String(err)}`;
    result.errors.push(msg);
    logger.warn(`memento: ${msg}`);
  }

  logger.info(
    `memento: deep consolidation — created ${result.clustersCreated}, ` +
      `updated ${result.clustersUpdated}, merged ${result.clustersMerged}, ` +
      `decayed ${result.factsDecayed} (agent: ${agentId})`,
  );

  return result;
}


path: src/consolidation/relation-sweep.ts
content: /**
 * Relation Sweep — Phase 3 Pipeline
 *
 * Background job that builds the knowledge graph by linking semantically
 * related facts via embedding similarity. Designed to run at 3 AM or
 * on-demand via the Mission Control UI.
 *
 * Algorithm:
 *   1. Load all active facts with embeddings from DB
 *   2. Compute pairwise cosine similarity across categories (cross-category focus)
 *   3. High-similarity same-category pairs → `related_to` edge (no LLM needed)
 *   4. Cross-category pairs above threshold → batched Haiku classification
 *   5. Store typed edges in fact_relations (created_by: "relation-sweep")
 *   6. Skip pairs already linked (idempotent)
 *
 * Thresholds:
 *   SAME_CATEGORY_AUTO  = 0.85  → `related_to` without LLM
 *   CROSS_CATEGORY_LLM  = 0.72  → send to Haiku for classification
 *   CROSS_CATEGORY_AUTO = 0.88  → `related_to` without LLM (very high sim)
 *
 * The sweep is incremental: it only processes pairs involving facts seen
 * since the last run (tracked via `relation_sweep_cursor` in a metadata row).
 */

import { randomUUID } from "node:crypto";
import { readFileSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import type { ConversationDB } from "../storage/db.js";
import type { FactRelationRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

// ---------------------------------------------------------------------------
// Thresholds
// ---------------------------------------------------------------------------

const SAME_CATEGORY_AUTO = 0.85;    // auto-link same-category pairs (no LLM)
const CROSS_CATEGORY_LLM_MIN = 0.72; // min similarity to send to LLM
const CROSS_CATEGORY_AUTO = 0.88;   // auto-link cross-category (skip LLM)
const LLM_BATCH_SIZE = 30;          // max pairs per Haiku call

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

export type RelationSweepResult = {
  factsScanned: number;
  pairsConsidered: number;
  edgesAutoLinked: number;
  edgesLLMClassified: number;
  edgesSkipped: number;  // already linked
  durationMs: number;
};

type FactWithVector = {
  id: string;
  agent_id: string;
  category: string;
  content: string;
  summary: string | null;
  visibility: string;
  embeddingVector: number[];
};

type CandidatePair = {
  a: FactWithVector;
  b: FactWithVector;
  similarity: number;
  sameCat: boolean;
};

type LLMClassification = {
  pair_index: number;
  relation_type: string;
  strength: number;
  rationale: string;
};

// ---------------------------------------------------------------------------
// Cosine similarity
// ---------------------------------------------------------------------------

function cosine(a: number[], b: number[]): number {
  let dot = 0, normA = 0, normB = 0;
  const len = Math.min(a.length, b.length);
  for (let i = 0; i < len; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  const denom = Math.sqrt(normA) * Math.sqrt(normB);
  return denom === 0 ? 0 : dot / denom;
}

// ---------------------------------------------------------------------------
// Prompt loader
// ---------------------------------------------------------------------------

function loadClassifyPrompt(): string {
  const path = join(__dirname, "..", "..", "prompts", "relation-classify.md");
  return readFileSync(path, "utf-8");
}

// ---------------------------------------------------------------------------
// LLM call (direct Anthropic API — same pattern as MC batch extraction)
// ---------------------------------------------------------------------------

async function callHaiku(pairs: CandidatePair[], apiToken: string, logger: PluginLogger): Promise<LLMClassification[]> {
  const promptTemplate = loadClassifyPrompt();

  const pairsJson = pairs.map((p, i) => ({
    pair_index: i,
    fact_a: { id: p.a.id, category: p.a.category, summary: p.a.summary, content: p.a.content.slice(0, 200) },
    fact_b: { id: p.b.id, category: p.b.category, summary: p.b.summary, content: p.b.content.slice(0, 200) },
    similarity: p.similarity.toFixed(3),
  }));

  const prompt = promptTemplate.replace("{{pairs}}", JSON.stringify(pairsJson, null, 2));

  const response = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${apiToken}`,
      "anthropic-version": "2023-06-01",
      "anthropic-beta": "oauth-2025-04-20",
    },
    body: JSON.stringify({
      model: "claude-haiku-4-5",
      max_tokens: 2048,
      messages: [{ role: "user", content: prompt }],
    }),
  });

  if (!response.ok) {
    const err = await response.text();
    throw new Error(`Haiku API error ${response.status}: ${err.slice(0, 200)}`);
  }

  const data = await response.json() as any;
  const text = data.content?.[0]?.text ?? "";

  // Parse JSON (strip markdown fences if present)
  const clean = text.replace(/^```[a-z]*\n?/, "").replace(/\n?```$/, "").trim();
  try {
    const parsed = JSON.parse(clean);
    return Array.isArray(parsed) ? parsed : [];
  } catch {
    logger.warn(`relation-sweep: failed to parse LLM response: ${clean.slice(0, 100)}`);
    return [];
  }
}

// ---------------------------------------------------------------------------
// Get existing edges (for dedup)
// ---------------------------------------------------------------------------

function getExistingEdgeSet(db: ConversationDB): Set<string> {
  const rows = db.getAllRelations?.() ?? [];
  const set = new Set<string>();
  for (const r of rows) {
    set.add(`${r.source_id}:${r.target_id}`);
    set.add(`${r.target_id}:${r.source_id}`); // undirected
  }
  return set;
}

// ---------------------------------------------------------------------------
// Main entry point
// ---------------------------------------------------------------------------

export async function runRelationSweep(
  db: ConversationDB,
  apiToken: string | null,
  logger: PluginLogger,
): Promise<RelationSweepResult> {
  const startMs = Date.now();
  let edgesAutoLinked = 0;
  let edgesLLMClassified = 0;
  let edgesSkipped = 0;

  logger.info("relation-sweep: starting…");

  // Load all active facts with embeddings
  const allAgentIds = db.getDistinctAgentIds?.() ?? ["main"];
  const allFacts: FactWithVector[] = [];

  for (const agentId of allAgentIds) {
    const facts = db.getFactsWithEmbeddings(agentId);
    // Include shared facts for cross-agent linking
    for (const f of facts) {
      if (f.visibility !== "secret") { // never link secret facts
        allFacts.push(f as FactWithVector);
      }
    }
  }

  logger.info(`relation-sweep: loaded ${allFacts.length} facts with embeddings`);

  if (allFacts.length < 2) {
    return { factsScanned: allFacts.length, pairsConsidered: 0, edgesAutoLinked: 0, edgesLLMClassified: 0, edgesSkipped: 0, durationMs: Date.now() - startMs };
  }

  // Get existing edges to avoid re-linking
  const existingEdges = getExistingEdgeSet(db);

  // Find candidate pairs
  const autoPairs: Array<{ pair: CandidatePair; relationType: string; strength: number }> = [];
  const llmPairs: CandidatePair[] = [];
  let pairsConsidered = 0;

  for (let i = 0; i < allFacts.length; i++) {
    for (let j = i + 1; j < allFacts.length; j++) {
      const a = allFacts[i];
      const b = allFacts[j];

      // Skip same-agent same-fact (shouldn't happen but safety)
      if (a.id === b.id) continue;
      // Skip if already linked
      if (existingEdges.has(`${a.id}:${b.id}`)) { edgesSkipped++; continue; }

      const sim = cosine(a.embeddingVector, b.embeddingVector);
      const sameCat = a.category === b.category;

      if (sameCat && sim >= SAME_CATEGORY_AUTO) {
        pairsConsidered++;
        autoPairs.push({ pair: { a, b, similarity: sim, sameCat }, relationType: "related_to", strength: Math.min(sim, 1.0) });
      } else if (!sameCat && sim >= CROSS_CATEGORY_AUTO) {
        pairsConsidered++;
        autoPairs.push({ pair: { a, b, similarity: sim, sameCat }, relationType: "related_to", strength: Math.min(sim * 0.95, 1.0) });
      } else if (!sameCat && sim >= CROSS_CATEGORY_LLM_MIN && apiToken) {
        pairsConsidered++;
        llmPairs.push({ a, b, similarity: sim, sameCat });
      }
    }
  }

  logger.info(`relation-sweep: ${autoPairs.length} auto pairs, ${llmPairs.length} LLM pairs`);

  const now = Date.now();
  const relationsToInsert: FactRelationRow[] = [];

  // Auto-link pairs
  for (const { pair, relationType, strength } of autoPairs) {
    relationsToInsert.push({
      id: randomUUID(),
      source_id: pair.a.id,
      target_id: pair.b.id,
      relation_type: relationType,
      strength,
      causal_weight: 1.0,
      created_at: now,
      created_by: "relation-sweep",
      metadata: JSON.stringify({ similarity: pair.similarity, method: "embedding-auto" }),
    });
    edgesAutoLinked++;
  }

  // LLM-classified pairs (batched)
  if (llmPairs.length > 0 && apiToken) {
    for (let start = 0; start < llmPairs.length; start += LLM_BATCH_SIZE) {
      const batch = llmPairs.slice(start, start + LLM_BATCH_SIZE);
      try {
        const classifications = await callHaiku(batch, apiToken, logger);
        for (const cls of classifications) {
          if (cls.pair_index < 0 || cls.pair_index >= batch.length) continue;
          const pair = batch[cls.pair_index];
          const validTypes = new Set(["caused_by", "precondition_of", "part_of", "related_to", "contradicts", "superseded_by"]);
          const relType = validTypes.has(cls.relation_type) ? cls.relation_type : "related_to";
          relationsToInsert.push({
            id: randomUUID(),
            source_id: pair.a.id,
            target_id: pair.b.id,
            relation_type: relType,
            strength: Math.min(Math.max(cls.strength ?? 0.7, 0.5), 1.0),
            causal_weight: ["caused_by", "precondition_of"].includes(relType) ? 1.5 : 1.0,
            created_at: now,
            created_by: "relation-sweep",
            metadata: JSON.stringify({ similarity: pair.similarity, method: "llm-haiku", rationale: (cls.rationale ?? "").slice(0, 200) }),
          });
          edgesLLMClassified++;
        }
        logger.info(`relation-sweep: batch ${Math.floor(start / LLM_BATCH_SIZE) + 1} — ${classifications.length} relations from ${batch.length} pairs`);
      } catch (err) {
        logger.warn(`relation-sweep: LLM batch failed: ${String(err)}`);
      }
    }
  }

  // Batch-insert all relations
  if (relationsToInsert.length > 0) {
    try {
      db.insertRelationsBatch(relationsToInsert);
      logger.info(`relation-sweep: inserted ${relationsToInsert.length} new edges`);
    } catch (err) {
      logger.warn(`relation-sweep: failed to insert relations: ${String(err)}`);
    }
  }

  const result: RelationSweepResult = {
    factsScanned: allFacts.length,
    pairsConsidered,
    edgesAutoLinked,
    edgesLLMClassified,
    edgesSkipped,
    durationMs: Date.now() - startMs,
  };

  logger.info(`relation-sweep: done in ${result.durationMs}ms — ${edgesAutoLinked} auto, ${edgesLLMClassified} LLM, ${edgesSkipped} skipped`);
  return result;
}


path: src/extraction/classifier.ts
content: /**
 * Visibility Classifier
 *
 * Determines whether a fact should be private, shared, or secret.
 * The LLM suggestion is trusted but overridden by hard keyword/category rules.
 *
 * Visibility levels:
 *   shared  — user preferences, family info, general knowledge (default)
 *   private — agent-specific operational details
 *   secret  — credentials, medical info, financial details (never propagated)
 */

// ---------------------------------------------------------------------------
// Secret rules (hard overrides — never trust LLM with these)
// ---------------------------------------------------------------------------

const SECRET_CATEGORIES = new Set([
  "credentials",
  "medical",
  "financial",
]);

const SECRET_KEYWORDS = [
  "password",
  "passwd",
  "passphrase",
  "secret",
  "api key",
  "apikey",
  "api_key",
  "access token",
  "access_token",
  "auth token",
  "auth_token",
  "private key",
  "private_key",
  "ssh key",
  "ssh_key",
  "credential",
  "secret key",
  "client secret",
  "diagnosis",
  "medical",
  "prescription",
  "medication",
  "symptom",
  "disease",
  "illness",
  "ssn",
  "social security",
  "credit card",
  "card number",
  "bank account",
  "routing number",
  "iban",
  "pin code",
  "pin number",
];

// ---------------------------------------------------------------------------
// Private rules
// ---------------------------------------------------------------------------

const PRIVATE_CATEGORIES = new Set([
  "operational",
  "workflow",
  "internal",
]);

const PRIVATE_KEYWORDS = [
  "workflow",
  "operational",
  "internal process",
  "internal procedure",
  "pipeline step",
  "agent instruction",
  "system prompt",
];

// ---------------------------------------------------------------------------
// Classifier
// ---------------------------------------------------------------------------

/**
 * Classify the visibility of a fact.
 *
 * @param category   - The LLM-assigned category (e.g. "preference", "medical")
 * @param content    - The full fact text
 * @param llmSuggested - The visibility level the LLM proposed
 * @returns "private" | "shared" | "secret"
 */
export function classifyVisibility(
  category: string,
  content: string,
  llmSuggested: string,
): "private" | "shared" | "secret" {
  const lowerContent = content.toLowerCase();
  const lowerCategory = category.toLowerCase();

  // Secret: hard category match
  if (SECRET_CATEGORIES.has(lowerCategory)) return "secret";

  // [REDACTED_SECRET] match in content
  if (SECRET_KEYWORDS.some((kw) => lowerContent.includes(kw))) return "secret";

  // Private: hard category match
  if (PRIVATE_CATEGORIES.has(lowerCategory)) return "private";

  // Private: keyword match
  if (PRIVATE_KEYWORDS.some((kw) => lowerContent.includes(kw))) return "private";

  // Trust LLM suggestion if it's a valid value
  if (
    llmSuggested === "private" ||
    llmSuggested === "shared" ||
    llmSuggested === "secret"
  ) {
    return llmSuggested;
  }

  // Default
  return "shared";
}


path: src/extraction/deduplicator.ts
content: /**
 * Deduplication Engine — Phase 2: Embedding-Based
 *
 * Processes an array of ExtractedFacts from the LLM, applying one of three
 * outcomes for each fact based on cosine similarity against existing embeddings:
 *
 *   1. Duplicate   (sim >= DUPLICATE_THRESHOLD)  — fact already exists; increment occurrence_count
 *   2. Update      (sim >= UPDATE_THRESHOLD)      — fact supersedes an older one; mark old inactive, insert new
 *   3. New         (sim <  UPDATE_THRESHOLD)      — genuinely new fact; insert + log first occurrence
 *
 * The LLM no longer provides `duplicate_of` / `supersedes` hints — those were
 * unreliable due to the windowed context. Cosine similarity over BGE-M3 embeddings
 * is more consistent and doesn't depend on history quality.
 *
 * Thresholds (tunable via config):
 *   DUPLICATE_THRESHOLD = 0.97  — near-identical wording, same meaning
 *   UPDATE_THRESHOLD    = 0.82  — same topic, different value (e.g. FTP changed)
 *
 * Facts without embeddings (model unavailable) fall back to new-fact insertion.
 */

import { randomUUID } from "node:crypto";
import type { ConversationDB } from "../storage/db.js";
import type { EmbeddingEngine } from "../storage/embeddings.js";
import type { ExtractedFact } from "./extractor.js";
import type { FactRow, FactRelationRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Thresholds
// ---------------------------------------------------------------------------

/** Cosine similarity >= this → treat as duplicate (same fact seen again) */
const DUPLICATE_THRESHOLD = 0.97;

/** Cosine similarity >= this (but < DUPLICATE_THRESHOLD) → treat as update (supersede) */
const UPDATE_THRESHOLD = 0.82;

/** Only compare against facts in the same category (reduces false positives) */
const SAME_CATEGORY_ONLY = true;

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

export type DeduplicationResult = {
  factsExtracted: number;      // genuinely new facts inserted
  factsUpdated: number;        // facts that superseded an older fact
  factsDeduplicated: number;   // facts identified as duplicates (occurrence incremented)
};

type FactWithVector = FactRow & { embeddingVector: number[] };

// ---------------------------------------------------------------------------
// Cosine similarity (inline — no dep on EmbeddingEngine internals)
// ---------------------------------------------------------------------------

function cosine(a: number[], b: number[]): number {
  let dot = 0, normA = 0, normB = 0;
  const len = Math.min(a.length, b.length);
  for (let i = 0; i < len; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  const denom = Math.sqrt(normA) * Math.sqrt(normB);
  return denom === 0 ? 0 : dot / denom;
}

// ---------------------------------------------------------------------------
// Main entry point
// ---------------------------------------------------------------------------

/**
 * Process extracted facts against the database, applying dedup/supersede/insert logic
 * using embedding cosine similarity (Phase 2 pipeline).
 *
 * @param facts           - Facts extracted by the LLM (history-agnostic, no duplicate_of hints)
 * @param conversationId  - Current conversation ID (for occurrence logging)
 * @param agentId         - Agent owning these facts
 * @param db              - ConversationDB instance
 * @param embeddingEngine - Optional: if provided, new facts get embedded immediately
 * @param logger          - Plugin logger
 */
export async function processExtractedFacts(
  facts: ExtractedFact[],
  conversationId: string,
  agentId: string,
  db: ConversationDB,
  embeddingEngine: EmbeddingEngine | null,
  logger: PluginLogger,
): Promise<DeduplicationResult> {
  const now = Date.now();
  let factsExtracted = 0;
  let factsUpdated = 0;
  let factsDeduplicated = 0;

  // Load existing facts with embeddings for this agent (once, for all comparisons)
  const existingFacts: FactWithVector[] = db.getFactsWithEmbeddings(agentId);

  logger.debug?.(
    `memento: deduplicator loaded ${existingFacts.length} existing facts with embeddings`,
  );

  for (const fact of facts) {
    try {
      let persistedFactId: string | null = null;

      // ── Step 1: Embed the incoming fact ──────────────────────────────────
      let incomingVector: number[] | null = null;
      if (embeddingEngine) {
        const text = `${fact.summary ? fact.summary + ". " : ""}${fact.content}`.slice(0, 2000);
        incomingVector = await embeddingEngine.embed(text);
      }

      // ── Step 2: Find best match via cosine similarity ─────────────────────
      let bestSim = 0;
      let bestMatch: FactWithVector | null = null;

      if (incomingVector && existingFacts.length > 0) {
        const candidates = SAME_CATEGORY_ONLY
          ? existingFacts.filter(f => f.category === fact.category)
          : existingFacts;

        for (const existing of candidates) {
          const sim = cosine(incomingVector, existing.embeddingVector);
          if (sim > bestSim) {
            bestSim = sim;
            bestMatch = existing;
          }
        }
      }

      logger.debug?.(
        `memento: best match sim=${bestSim.toFixed(4)} for "${fact.content.slice(0, 50)}"` +
        (bestMatch ? ` → "${bestMatch.summary?.slice(0, 50)}"` : " (no match)"),
      );

      // ── Step 3: Apply outcome based on similarity ─────────────────────────

      if (bestMatch && bestSim >= DUPLICATE_THRESHOLD) {
        // DUPLICATE — increment occurrence, update last_seen_at
        db.updateFactOccurrence(
          bestMatch.id,
          conversationId,
          fact.content,
          fact.sentiment ?? "neutral",
          now,
        );
        persistedFactId = bestMatch.id;
        factsDeduplicated++;
        logger.debug?.(
          `memento: duplicate (sim=${bestSim.toFixed(3)}) of ${bestMatch.id}: ${fact.content.slice(0, 60)}`,
        );

      } else if (bestMatch && bestSim >= UPDATE_THRESHOLD) {
        // UPDATE — supersede the existing fact with the new one
        const newFact: FactRow = buildFactRow(fact, agentId, now);
        db.supersedeFact(bestMatch.id, newFact);
        db.updateFactOccurrence(
          newFact.id,
          conversationId,
          fact.content,
          fact.sentiment ?? "update",
          now,
        );
        // Store embedding for the new fact
        if (incomingVector) {
          db.setFactEmbedding(newFact.id, incomingVector);
          // Update the in-memory list so subsequent facts can match against this new one
          existingFacts.push({ ...newFact, embeddingVector: incomingVector });
        }
        persistedFactId = newFact.id;
        factsUpdated++;
        logger.debug?.(
          `memento: superseded (sim=${bestSim.toFixed(3)}) ${bestMatch.id} → ${newFact.id}: ${fact.content.slice(0, 60)}`,
        );

      } else {
        // NEW FACT — insert and embed
        const newFact: FactRow = buildFactRow(fact, agentId, now);
        db.insertFact(newFact);
        db.updateFactOccurrence(
          newFact.id,
          conversationId,
          fact.content,
          fact.sentiment ?? "neutral",
          now,
        );
        if (incomingVector) {
          db.setFactEmbedding(newFact.id, incomingVector);
          existingFacts.push({ ...newFact, embeddingVector: incomingVector });
        }
        persistedFactId = newFact.id;
        factsExtracted++;
        logger.debug?.(
          `memento: new fact ${newFact.id} [${fact.category}]: ${fact.content.slice(0, 60)}`,
        );
      }

      // ── Step 4: Store graph relations (if any provided by LLM) ───────────
      // Relations are still accepted from the LLM if present, but no longer required.
      // Phase 3 (async background job) will build relations from embedding similarity.
      if (persistedFactId && fact.relations && fact.relations.length > 0) {
        const CAUSAL_TYPES = new Set(["caused_by", "precondition_of"]);
        const relations: FactRelationRow[] = fact.relations
          .filter((r) => r.target_id !== persistedFactId)
          .map((r) => ({
            id: randomUUID(),
            source_id: persistedFactId!,
            target_id: r.target_id,
            relation_type: r.relation_type,
            strength: r.strength ?? 0.8,
            causal_weight: CAUSAL_TYPES.has(r.relation_type) ? 1.5 : 1.0,
            created_at: now,
            created_by: "extraction",
            metadata: null,
          }));

        if (relations.length > 0) {
          try {
            db.insertRelationsBatch(relations);
          } catch (relErr) {
            logger.warn(`memento: failed to store relations for ${persistedFactId}: ${String(relErr)}`);
          }
        }
      }

    } catch (err) {
      logger.warn(
        `memento: deduplicator error processing fact "${fact.content.slice(0, 60)}": ${String(err)}`,
      );
    }
  }

  return { factsExtracted, factsUpdated, factsDeduplicated };
}

// ---------------------------------------------------------------------------
// Internal helpers
// ---------------------------------------------------------------------------

function buildFactRow(fact: ExtractedFact, agentId: string, now: number): FactRow {
  return {
    id: randomUUID(),
    agent_id: agentId,
    category: fact.category,
    content: fact.content,
    summary: fact.summary ?? null,
    visibility: fact.visibility ?? "shared",
    confidence: fact.confidence ?? 1.0,
    first_seen_at: now,
    last_seen_at: now,
    occurrence_count: 0, // updateFactOccurrence will increment to 1
    supersedes: fact.supersedes ?? null,
    is_active: 1,
    metadata: null,
    embedding: null, // set immediately after insert via setFactEmbedding
    previous_value: null, // set by supersedeFact in db.ts
  };
}


path: src/extraction/embedded-runner.ts
content: /**
 * OpenClaw Embedded Runner Integration
 *
 * Delegates LLM calls to OpenClaw's internal model routing system
 * (runEmbeddedPiAgent) instead of making direct HTTP calls.
 *
 * This means Memento automatically inherits:
 *   - The agent's configured model (primary + fallbacks)
 *   - OpenClaw's auth profile handling
 *   - Provider routing and failover
 *
 * Used by ExtractionTrigger when running inside the OpenClaw plugin context.
 * Falls back to direct API calls when running as a standalone CLI tool.
 */

import fs from "node:fs/promises";
import path from "node:path";
import os from "node:os";

type RunEmbeddedPiAgentFn = (params: Record<string, unknown>) => Promise<unknown>;

/**
 * Dynamically load runEmbeddedPiAgent from OpenClaw internals.
 * Returns null if not available (e.g. running standalone CLI).
 */
async function loadRunEmbeddedPiAgent(): Promise<RunEmbeddedPiAgentFn | null> {
  // Try source checkout path first (dev/test)
  const candidates = [
    "openclaw/src/agents/pi-embedded-runner.js",
    "openclaw/dist/agents/pi-embedded-runner.js",
  ];

  for (const candidate of candidates) {
    try {
      const mod = await import(candidate) as any;
      if (typeof mod.runEmbeddedPiAgent === "function") {
        return mod.runEmbeddedPiAgent as RunEmbeddedPiAgentFn;
      }
    } catch {
      // not found — try next
    }
  }

  return null;
}

function collectText(payloads: Array<{ text?: string; isError?: boolean }> | undefined): string {
  return (payloads ?? [])
    .filter((p) => !p.isError && typeof p.text === "string")
    .map((p) => p.text ?? "")
    .join("\n")
    .trim();
}

function stripCodeFences(s: string): string {
  const trimmed = s.trim();
  const m = trimmed.match(/^```(?:json)?\s*([\s\S]*?)\s*```$/i);
  return m ? (m[1] ?? "").trim() : trimmed;
}

export type OpenClawConfig = Record<string, unknown>;

export type EmbeddedRunnerResult = {
  text: string;
  provider: string;
  model: string;
};

/**
 * Run a one-shot LLM prompt via OpenClaw's model routing.
 *
 * Resolves provider/model from api.config (agent defaults),
 * with optional override via the extractionModel config.
 */
export async function runViaOpenClaw(
  prompt: string,
  config: OpenClawConfig,
  extractionModel?: string,
): Promise<EmbeddedRunnerResult | null> {
  const runEmbeddedPiAgent = await loadRunEmbeddedPiAgent();
  if (!runEmbeddedPiAgent) {
    return null; // Not inside OpenClaw — caller falls back to direct API
  }

  // Resolve provider/model from config (same logic as llm-task plugin)
  const defaultsModel = (config as any)?.agents?.defaults?.model;
  const primary =
    typeof defaultsModel === "string"
      ? defaultsModel.trim()
      : (defaultsModel?.primary?.trim() ?? undefined);

  // Allow extractionModel config to override the agent default
  const modelOverride = extractionModel?.trim();
  const resolved = modelOverride ?? primary;

  if (!resolved) {
    return null; // Can't resolve model — fall back to direct API
  }

  const slashIdx = resolved.indexOf("/");
  const provider = slashIdx !== -1 ? resolved.slice(0, slashIdx) : resolved;
  const model = slashIdx !== -1 ? resolved.slice(slashIdx + 1) : resolved;

  const workspaceDir = (config as any)?.agents?.defaults?.workspace ?? process.cwd();

  const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "memento-extract-"));
  try {
    const sessionId = `memento-extract-${Date.now()}`;
    const sessionFile = path.join(tmpDir, "session.json");

    const result = await runEmbeddedPiAgent({
      sessionId,
      sessionFile,
      workspaceDir,
      config,
      prompt,
      timeoutMs: 60_000,
      runId: `memento-extract-${Date.now()}`,
      provider,
      model,
      authProfileIdSource: "auto",
      disableTools: true,
    }) as any;

    const text = collectText(result?.payloads);
    if (!text) return null;

    return { text: stripCodeFences(text), provider, model };
  } finally {
    await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
  }
}


path: src/extraction/extractor.ts
content: /**
 * Fact Extractor — Provider-Agnostic
 *
 * Calls the configured LLM provider (Anthropic, OpenAI, Mistral, Ollama, or any
 * OpenAI-compatible endpoint) with a structured extraction prompt, then parses the
 * JSON response into typed ExtractedFact objects.
 *
 * The LLM is instructed to:
 *   - Return facts as a JSON array
 *   - Flag duplicates (duplicate_of + increment_occurrence)
 *   - Flag updates to existing facts (supersedes)
 *   - Assign visibility and sentiment
 *
 * Supported provider prefixes in the model string:
 *   - anthropic/<model>  → https://api.anthropic.com/v1/messages   (Anthropic format)
 *   - openai/<model>     → https://api.openai.com/v1/chat/completions (OpenAI format)
 *   - mistral/<model>    → https://api.mistral.ai/v1/chat/completions (OpenAI format)
 *   - ollama/<model>     → http://localhost:11434/api/chat           (Ollama format)
 *   - anything else      → OpenAI-compatible format (fallback)
 *
 * Credential resolution (in order):
 *   1. ANTHROPIC_API_KEY / OPENAI_API_KEY / MISTRAL_API_KEY / OLLAMA_API_KEY env vars
 *   2. Generic MEMENTO_API_KEY env var (for custom endpoints)
 *   Ollama requires no key (local model server).
 *
 * All API calls are best-effort: errors are returned, never thrown.
 */

import { readFileSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { classifyVisibility } from "./classifier.js";
import { runViaOpenClaw } from "./embedded-runner.js";
import type { OpenClawConfig } from "./embedded-runner.js";
import type { FactRow, ConversationRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";

// ---------------------------------------------------------------------------
// Public types
// ---------------------------------------------------------------------------

export type ExtractedFactRelation = {
  /** ID of the related existing fact */
  target_id: string;
  /** Type of relation */
  relation_type: string; // 'related_to' | 'elaborates' | 'contradicts' | 'supports' | 'caused_by' | 'part_of' | 'precondition_of'
  /** Edge strength 0.0–1.0 (default 0.8) */
  strength: number;
};

export type ExtractedFact = {
  category: string;
  content: string;
  summary: string;
  visibility: string;
  confidence: number;
  sentiment: string;
  /** Set if this fact supersedes an existing one (its ID) */
  supersedes?: string;
  /** Set if this is a duplicate of an existing fact (its ID) */
  duplicate_of?: string;
  /** true when duplicate_of is set and occurrence should be incremented */
  increment_occurrence?: boolean;
  /** Relations to existing facts identified during extraction */
  relations?: ExtractedFactRelation[];
};

export type ExtractionResult = {
  facts: ExtractedFact[];
  modelUsed: string;
  error?: string;
};

// ---------------------------------------------------------------------------
// Extraction prompt — loaded from prompts/extraction.md at runtime
// Edit that file to tune extraction quality without recompiling.
// ---------------------------------------------------------------------------

const __filename = fileURLToPath(import.meta.url)
const __dirname = dirname(__filename)

function loadExtractionPrompt(): string {
  const promptPath = join(__dirname, '..', '..', 'prompts', 'extraction.md')
  try {
    const raw = readFileSync(promptPath, 'utf-8')
    // Extract the content between "## Prompt" and the trailing "## Conversation" section
    const match = raw.match(/## Prompt\n\n([\s\S]+?)\n---\n\n## Conversation/m)
    if (match) return match[1].trim() + '\n\nCONVERSATION:\n{{conversation}}'
    return raw
  } catch {
    throw new Error(`Extraction prompt not found at ${promptPath}`)
  }
}

const EXTRACTION_PROMPT = loadExtractionPrompt()


// ---------------------------------------------------------------------------
// Token budget helpers
// ---------------------------------------------------------------------------

/** Rough token estimate: ~4 chars per token (good enough for budget checks). */
function estimateTokens(text: string): number {
  return Math.ceil(text.length / 4);
}

/**
 * Truncate a string to approximately `maxTokens` worth of content.
 * Truncates at a newline boundary to avoid cutting mid-sentence.
 */
function truncateToTokenBudget(text: string, maxTokens: number): string {
  const maxChars = maxTokens * 4;
  if (text.length <= maxChars) return text;
  const truncated = text.slice(0, maxChars);
  const lastNewline = truncated.lastIndexOf("\n");
  const cut = lastNewline > maxChars * 0.8 ? lastNewline : maxChars;
  return text.slice(0, cut) + "\n\n[... truncated for token budget ...]";
}

// ---------------------------------------------------------------------------
// Provider detection
// ---------------------------------------------------------------------------

type Provider = "anthropic" | "openai" | "mistral" | "ollama" | "openai-compat";

interface ProviderInfo {
  provider: Provider;
  /** Model name without the provider prefix */
  modelName: string;
  /** API endpoint URL */
  endpoint: string;
  /** ENV var name to look up for the API key */
  apiKeyEnv: string;
}

function detectProvider(model: string): ProviderInfo {
  const slashIdx = model.indexOf("/");
  const prefix = slashIdx !== -1 ? model.slice(0, slashIdx).toLowerCase() : "";
  const modelName = slashIdx !== -1 ? model.slice(slashIdx + 1) : model;

  switch (prefix) {
    case "anthropic":
      return {
        provider: "anthropic",
        modelName,
        endpoint: "https://api.anthropic.com/v1/messages",
        apiKeyEnv: "CLAUDE_CODE_OAUTH_TOKEN",
      };
    case "openai":
      return {
        provider: "openai",
        modelName,
        endpoint: "https://api.openai.com/v1/chat/completions",
        apiKeyEnv: "OPENAI_API_KEY",
      };
    case "mistral":
      return {
        provider: "mistral",
        modelName,
        endpoint: "https://api.mistral.ai/v1/chat/completions",
        apiKeyEnv: "MISTRAL_API_KEY",
      };
    case "ollama":
      return {
        provider: "ollama",
        modelName,
        // Ollama runs locally; no API key required
        endpoint: process.env.OLLAMA_BASE_URL ?? "http://localhost:11434/api/chat",
        apiKeyEnv: "OLLAMA_API_KEY", // optional; Ollama doesn't require one
      };
    default:
      // Unknown prefix — try OpenAI-compatible format as fallback
      return {
        provider: "openai-compat",
        modelName: model, // keep full string — user may know what endpoint expects
        endpoint: process.env.OPENAI_BASE_URL ?? "https://api.openai.com/v1/chat/completions",
        apiKeyEnv: "OPENAI_API_KEY",
      };
  }
}

// ---------------------------------------------------------------------------
// Credential resolution (no filesystem snooping)
// ---------------------------------------------------------------------------

/**
 * Resolve an API key for the given provider by checking environment variables.
 *
 * Priority:
 *   1. Provider-specific env var (e.g. ANTHROPIC_API_KEY)
 *   2. Generic MEMENTO_API_KEY fallback
 *   3. null — let the caller decide (Ollama doesn't need a key)
 */
function resolveApiKey(apiKeyEnv: string): string | null {
  return (
    process.env[apiKeyEnv] ||
    process.env.MEMENTO_API_KEY ||
    null
  );
}

// ---------------------------------------------------------------------------
// Provider-specific API callers
// ---------------------------------------------------------------------------

/** Call the Anthropic Messages API (native format). */
async function callAnthropic(
  info: ProviderInfo,
  [REDACTED_SECRET],
  prompt: string,
): Promise<string> {
  const isOAuth = apiKey.includes("sk-ant-oat");
  const authHeaders: Record<string, string> = isOAuth
    ? {
        "Authorization": `Bearer ${apiKey}`,
        "anthropic-beta": "claude-code-20250219,oauth-2025-04-20",
      }
    : {
        "x-api-key": apiKey,
      };

  const response = await fetch(info.endpoint, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "anthropic-version": "2023-06-01",
      ...authHeaders,
    },
    body: JSON.stringify({
      model: info.modelName,
      max_tokens: 4096,
      messages: [{ role: "user", content: prompt }],
    }),
  });

  if (!response.ok) {
    const errText = await response.text().catch(() => "(unreadable)");
    throw new Error(`Anthropic API error ${response.status}: ${errText.slice(0, 300)}`);
  }

  const data = (await response.json()) as any;
  // Anthropic format: { content: [{ type: "text", text: "..." }] }
  return data?.content?.[0]?.text ?? "";
}

/** Call an OpenAI-compatible API (OpenAI, Mistral, openai-compat). */
async function callOpenAICompat(
  info: ProviderInfo,
  [REDACTED_SECRET] | null,
  prompt: string,
): Promise<string> {
  const headers: Record<string, string> = {
    "Content-Type": "application/json",
  };
  if (apiKey) {
    headers["Authorization"] = `Bearer ${apiKey}`;
  }

  const response = await fetch(info.endpoint, {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: info.modelName,
      max_tokens: 4096,
      messages: [{ role: "user", content: prompt }],
    }),
  });

  if (!response.ok) {
    const errText = await response.text().catch(() => "(unreadable)");
    throw new Error(`${info.provider} API error ${response.status}: ${errText.slice(0, 300)}`);
  }

  const data = (await response.json()) as any;
  // OpenAI format: { choices: [{ message: { content: "..." } }] }
  return data?.choices?.[0]?.message?.content ?? "";
}

/** Call Ollama's native chat API. */
async function callOllama(
  info: ProviderInfo,
  prompt: string,
): Promise<string> {
  const response = await fetch(info.endpoint, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: info.modelName,
      messages: [{ role: "user", content: prompt }],
      stream: false,
    }),
  });

  if (!response.ok) {
    const errText = await response.text().catch(() => "(unreadable)");
    throw new Error(`Ollama API error ${response.status}: ${errText.slice(0, 300)}`);
  }

  const data = (await response.json()) as any;
  // Ollama format: { message: { content: "..." } }
  return data?.message?.content ?? "";
}

// ---------------------------------------------------------------------------
// Main extractor
// ---------------------------------------------------------------------------

/**
 * Call the configured LLM provider to extract facts from a conversation segment.
 *
 * @param conversation   - The captured conversation row
 * @param existingFacts  - Active facts for the agent (kept for API compatibility; Phase 2 dedup uses embeddings)
 * @param model          - Full model string, e.g. "anthropic/claude-sonnet-4-6"
 * @param logger
 */
export async function extractFacts(
  conversation: ConversationRow,
  _existingFacts: FactRow[], // Phase 2: dedup now handled by embedding similarity, not LLM
  model: string,
  logger: PluginLogger,
  openClawConfig?: OpenClawConfig,
): Promise<ExtractionResult> {

  // Model context window budget (leave headroom for response)
  const MODEL_CONTEXT_TOKENS = 180_000;
  const RESPONSE_HEADROOM_TOKENS = 4_096;
  const PROMPT_TEMPLATE_TOKENS = estimateTokens(EXTRACTION_PROMPT);
  const availableTokens = MODEL_CONTEXT_TOKENS - RESPONSE_HEADROOM_TOKENS - PROMPT_TEMPLATE_TOKENS;

  // Phase 1: history-agnostic extraction.
  // Existing facts are NO LONGER passed to the LLM.
  // Dedup/superseding is handled by Phase 2 (embedding similarity) in deduplicator.ts.
  // This reduces token cost, removes history-quality dependency, and improves extraction focus.
  const conversationText = truncateToTokenBudget(conversation.raw_text, availableTokens);
  const prompt = EXTRACTION_PROMPT.replace("{{conversation}}", conversationText);

  // ── Dispatch: try OpenClaw model routing first, fall back to direct API ──
  let responseText: string;
  let modelUsed = model;
  try {
    // Strategy 1: Use OpenClaw's embedded runner (preferred — inherits agent model config)
    if (openClawConfig) {
      const ocResult = await runViaOpenClaw(prompt, openClawConfig, model);
      if (ocResult) {
        responseText = ocResult.text;
        modelUsed = `${ocResult.provider}/${ocResult.model}`;
        logger.debug?.(`memento: extraction via OpenClaw model routing (${modelUsed})`);
      } else {
        // OpenClaw runner not available — fall through to direct API
        logger.debug?.("memento: OpenClaw runner unavailable, falling back to direct API");
        const info = detectProvider(model);
        const [REDACTED_SECRET]);
        if (!apiKey && info.provider !== "ollama") {
          return {
            facts: [],
            modelUsed: model,
            error: `No API key found for provider "${info.provider}". Set the ${info.apiKeyEnv} environment variable (or MEMENTO_API_KEY as a generic fallback).`,
          };
        }
        switch (info.provider) {
          case "anthropic":
            responseText = await callAnthropic(info, apiKey!, prompt);
            break;
          case "ollama":
            responseText = await callOllama(info, prompt);
            break;
          default:
            responseText = await callOpenAICompat(info, apiKey, prompt);
            break;
        }
      }
    } else {
      // Strategy 2: Direct API (CLI mode or no OpenClaw config available)
      const info = detectProvider(model);
      const [REDACTED_SECRET]);
      if (!apiKey && info.provider !== "ollama") {
        return {
          facts: [],
          modelUsed: model,
          error: `No API key found for provider "${info.provider}". Set the ${info.apiKeyEnv} environment variable (or MEMENTO_API_KEY as a generic fallback).`,
        };
      }
      switch (info.provider) {
        case "anthropic":
          responseText = await callAnthropic(info, apiKey!, prompt);
          break;
        case "ollama":
          responseText = await callOllama(info, prompt);
          break;
        default:
          responseText = await callOpenAICompat(info, apiKey, prompt);
          break;
      }
    }
  } catch (err) {
    return {
      facts: [],
      modelUsed,
      error: `API call failed: ${String(err)}`,
    };
  }

  // Parse JSON array from the model response
  let rawFacts: any[];
  try {
    const trimmed = responseText.trim();
    // Handle case where model wraps the array in markdown code fences
    const jsonStr = trimmed.startsWith("```")
      ? trimmed.replace(/^```[a-z]*\n?/, "").replace(/\n?```$/, "").trim()
      : trimmed;

    // Find the outermost JSON array
    const arrayStart = jsonStr.indexOf("[");
    const arrayEnd = jsonStr.lastIndexOf("]");
    if (arrayStart === -1 || arrayEnd === -1) {
      return {
        facts: [],
        modelUsed: model,
        error: `No JSON array found in response: ${responseText.slice(0, 200)}`,
      };
    }

    rawFacts = JSON.parse(jsonStr.slice(arrayStart, arrayEnd + 1));
  } catch (parseErr) {
    return {
      facts: [],
      modelUsed: model,
      error: `JSON parse error: ${String(parseErr)} | response: ${responseText.slice(0, 200)}`,
    };
  }

  // Validate, normalize, and apply classifier overrides
  const facts: ExtractedFact[] = [];
  for (const raw of rawFacts) {
    if (typeof raw !== "object" || raw === null) continue;
    if (typeof raw.content !== "string" || !raw.content.trim()) continue;

    const category = typeof raw.category === "string" ? raw.category : "technical";
    const llmVisibility =
      typeof raw.visibility === "string" ? raw.visibility : "shared";

    const fact: ExtractedFact = {
      category,
      content: raw.content.trim(),
      summary:
        typeof raw.summary === "string"
          ? raw.summary.slice(0, 100).trim()
          : raw.content.slice(0, 100).trim(),
      visibility: classifyVisibility(category, raw.content, llmVisibility),
      confidence:
        typeof raw.confidence === "number"
          ? Math.min(1, Math.max(0, raw.confidence))
          : 1.0,
      sentiment:
        typeof raw.sentiment === "string" ? raw.sentiment : "neutral",
    };

    if (typeof raw.supersedes === "string" && raw.supersedes.trim()) {
      fact.supersedes = raw.supersedes.trim();
    }
    if (typeof raw.duplicate_of === "string" && raw.duplicate_of.trim()) {
      fact.duplicate_of = raw.duplicate_of.trim();
      fact.increment_occurrence = raw.increment_occurrence === true;
    }

    // Parse relations to existing facts
    if (Array.isArray(raw.relations) && raw.relations.length > 0) {
      fact.relations = raw.relations
        .filter(
          (r: any) =>
            typeof r === "object" &&
            r !== null &&
            typeof r.target_id === "string" &&
            r.target_id.trim() &&
            typeof r.relation_type === "string" &&
            r.relation_type.trim(),
        )
        .map((r: any) => ({
          target_id: r.target_id.trim(),
          relation_type: r.relation_type.trim(),
          strength:
            typeof r.strength === "number"
              ? Math.min(1, Math.max(0, r.strength))
              : 0.8,
        }));
    }

    // Skip low-confidence inferences (threshold matches prompt instruction of 0.6)
    if (fact.confidence < 0.6) {
      logger.debug?.(
        `memento: skipping low-confidence fact (${fact.confidence}): ${fact.content.slice(0, 60)}`,
      );
      continue;
    }

    facts.push(fact);
  }

  logger.debug?.(
    `memento: extractor parsed ${facts.length} facts from ${rawFacts.length} raw`,
  );

  return { facts, modelUsed: model };
}


path: src/extraction/migrate.ts
content: #!/usr/bin/env node
/**
 * Memento Migration Script
 *
 * Bootstraps the Memento knowledge base from existing agent memory files.
 * Uses the same extraction pipeline as live capture (extractor + deduplicator).
 * Tracks processed files in extraction_log so re-runs are safe (incremental).
 *
 * Usage:
 *   npx tsx src/extraction/migrate.ts --all
 *   npx tsx src/extraction/migrate.ts --agent main --dry-run
 *   npx tsx src/extraction/migrate.ts --agent drjones --force
 *   npx tsx src/extraction/migrate.ts --all --model anthropic/claude-haiku-4-5-20251001
 *   npx tsx src/extraction/migrate.ts --agent main --db /custom/path/conversations.sqlite
 */

import { readFileSync, existsSync, readdirSync, statSync } from "node:fs";
import { join, resolve, basename } from "node:path";
import { homedir } from "node:os";
import { createHash, randomUUID } from "node:crypto";

import { ConversationDB } from "../storage/db.js";
import { extractFacts } from "./extractor.js";
import { processExtractedFacts } from "./deduplicator.js";
import type { ConversationRow } from "../storage/schema.js";

// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------

/** Files to always skip — system/operational, not knowledge */
const SYSTEM_FILES = new Set([
  "SOUL.md",
  "AGENTS.md",
  "USER.md",
  "IDENTITY.md",
  "HEARTBEAT.md",
  "TOOLS.md",
  "BOOTSTRAP.md",
]);

const FILE_CHUNK_THRESHOLD_BYTES = 50 * 1024; // 50 KB: split files larger than this
const CHUNK_SIZE_BYTES = 30 * 1024;            // 30 KB per chunk
const EXTRACTION_DELAY_MS = 2_000;             // 2 s between API calls
const DEFAULT_MODEL = "anthropic/claude-sonnet-4-6";
const DEFAULT_DB_PATH = join(homedir(), ".engram", "conversations.sqlite");
const EXISTING_FACTS_LIMIT = 50;

// ---------------------------------------------------------------------------
// Agent inventory
// ---------------------------------------------------------------------------

interface AgentConfig {
  agentId: string;
  workspace: string;
  /** Glob patterns relative to workspace */
  paths: string[];
}

/**
 * Load agent configs from ~/.engram/migration-config.json if it exists,
 * otherwise fall back to env vars, otherwise return an empty list and ask
 * the user to configure.
 *
 * Config file format:
 * {
 *   "agents": [
 *     { "agentId": "main", "workspace": "/path/to/workspace", "paths": ["MEMORY.md", "memory/*.md"] },
 *     ...
 *   ]
 * }
 */
function loadAgentConfigs(): AgentConfig[] {
  const home = homedir();

  // 1. External config file (preferred — nothing leaks into source)
  const configPath = join(home, ".engram", "migration-config.json");
  if (existsSync(configPath)) {
    try {
      const raw = JSON.parse(readFileSync(configPath, "utf8"));
      if (Array.isArray(raw.agents) && raw.agents.length > 0) {
        return raw.agents as AgentConfig[];
      }
    } catch (err) {
      console.warn(`[migrate] Could not parse ${configPath}: ${String(err)}`);
    }
  }

  // 2. Environment variables for common agents
  const fromEnv: AgentConfig[] = [];
  const envMappings: Array<{ envVar: string; agentId: string; defaultPaths: string[] }> = [
    { envVar: "MEMENTO_WORKSPACE_MAIN",   agentId: "main",    defaultPaths: ["MEMORY.md", "memory/*.md"] },
    { envVar: "MEMENTO_WORKSPACE_DRJONES",agentId: "drjones", defaultPaths: ["MEMORY.md", "memory/*.md", "prescriptions/*.md", "medical_records/*.md"] },
  ];
  for (const { envVar, agentId, defaultPaths } of envMappings) {
    const ws = process.env[envVar];
    if (ws) {
      fromEnv.push({ agentId, workspace: ws, paths: defaultPaths });
    }
  }
  if (fromEnv.length > 0) return fromEnv;

  // 3. No config found — return empty list; parseArgs() will surface a helpful error
  return [];
}

let _defaultAgentsCache: AgentConfig[] | null = null;

function getDefaultAgents(): AgentConfig[] {
  if (_defaultAgentsCache !== null) return _defaultAgentsCache;
  _defaultAgentsCache = loadAgentConfigs();
  return _defaultAgentsCache;
}

// ---------------------------------------------------------------------------
// CLI parsing
// ---------------------------------------------------------------------------

interface CliOptions {
  agents: AgentConfig[];
  dryRun: boolean;
  force: boolean;
  model: string;
  dbPath: string;
}

function parseArgs(): CliOptions {
  const args = process.argv.slice(2);
  let agentFilter: string | null = null;
  let all = false;
  let dryRun = false;
  let force = false;
  let model = DEFAULT_MODEL;
  let dbPath = DEFAULT_DB_PATH;

  for (let i = 0; i < args.length; i++) {
    const arg = args[i];
    if (arg === "--all") {
      all = true;
    } else if (arg === "--dry-run") {
      dryRun = true;
    } else if (arg === "--force") {
      force = true;
    } else if (arg === "--agent" && args[i + 1]) {
      agentFilter = args[++i];
    } else if (arg === "--model" && args[i + 1]) {
      model = args[++i];
    } else if (arg === "--db" && args[i + 1]) {
      dbPath = resolve(args[++i]);
    }
  }

  const defaultAgents = getDefaultAgents();

  if (defaultAgents.length === 0) {
    console.error(
      "\nNo agent configuration found. Please create ~/.engram/migration-config.json:\n" +
      JSON.stringify({
        agents: [
          { agentId: "main", workspace: join(homedir(), "your-workspace"), paths: ["MEMORY.md", "memory/*.md"] },
        ],
      }, null, 2) +
      "\n\nOr set MEMENTO_WORKSPACE_MAIN environment variable to your workspace path.\n",
    );
    process.exit(1);
  }

  if (!all && !agentFilter) {
    console.error(
      "Usage: migrate.ts --all | --agent <id> [--dry-run] [--force] [--model <model>] [--db <path>]\n" +
        `Valid agents: ${defaultAgents.map((a) => a.agentId).join(", ")}`,
    );
    process.exit(1);
  }

  let agents: AgentConfig[];
  if (all) {
    agents = defaultAgents;
  } else {
    const found = defaultAgents.find((a) => a.agentId === agentFilter);
    if (!found) {
      console.error(
        `Unknown agent: ${agentFilter}. Valid agents: ${defaultAgents.map((a) => a.agentId).join(", ")}`,
      );
      process.exit(1);
    }
    agents = [found];
  }

  return { agents, dryRun, force, model, dbPath };
}

// ---------------------------------------------------------------------------
// Simple glob expander (no deps — handles "dir/*.md" and "file.md")
// ---------------------------------------------------------------------------

function expandGlob(baseDir: string, pattern: string): string[] {
  const slashIdx = pattern.lastIndexOf("/");
  const dir = slashIdx === -1 ? baseDir : join(baseDir, pattern.slice(0, slashIdx));
  const filePattern = slashIdx === -1 ? pattern : pattern.slice(slashIdx + 1);

  if (!existsSync(dir)) return [];

  if (!filePattern.includes("*")) {
    // Literal filename
    const full = join(dir, filePattern);
    return existsSync(full) && statSync(full).isFile() ? [full] : [];
  }

  // Wildcard — convert glob to regex and match against dir entries
  const reStr = "^" + filePattern.replace(/\./g, "\\.").replace(/\*/g, ".*") + "$";
  const re = new RegExp(reStr);
  try {
    return readdirSync(dir)
      .filter((name) => re.test(name))
      .map((name) => join(dir, name))
      .filter((full) => statSync(full).isFile());
  } catch {
    return [];
  }
}

// ---------------------------------------------------------------------------
// Deterministic conversation ID from file path (UUID-shaped hex slice)
// ---------------------------------------------------------------------------

function makeConversationId(agentId: string, filePath: string, chunkIndex?: number): string {
  const key =
    chunkIndex !== undefined
      ? `migration:${agentId}:${filePath}:chunk:${chunkIndex}`
      : `migration:${agentId}:${filePath}`;
  const hex = createHash("sha256").update(key).digest("hex");
  // Format as UUID to satisfy any UUID validators downstream
  return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}

// ---------------------------------------------------------------------------
// Text chunker — splits at newline boundaries
// ---------------------------------------------------------------------------

function chunkText(text: string, maxBytes: number): string[] {
  if (Buffer.byteLength(text, "utf8") <= maxBytes) return [text];

  const chunks: string[] = [];
  const lines = text.split("\n");
  let current = "";

  for (const line of lines) {
    const next = current ? current + "\n" + line : line;
    if (Buffer.byteLength(next, "utf8") > maxBytes && current) {
      chunks.push(current);
      current = line;
    } else {
      current = next;
    }
  }
  if (current) chunks.push(current);
  return chunks;
}

// ---------------------------------------------------------------------------
// Logger (stdout only)
// ---------------------------------------------------------------------------

const logger = {
  info: (...args: any[]) => console.log("[migrate]", ...args),
  warn: (...args: any[]) => console.warn("[migrate:WARN]", ...args),
  error: (...args: any[]) => console.error("[migrate:ERROR]", ...args),
  // Suppress debug noise from extractor/deduplicator
  debug: (_msg: string) => {},
};

function log(...args: any[]) {
  console.log("[migrate]", ...args);
}

// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------

function sleep(ms: number): Promise<void> {
  return new Promise((r) => setTimeout(r, ms));
}

function isFileAlreadyExtracted(db: ConversationDB, agentId: string, filePath: string): boolean {
  // For small files, the ID has no chunk index. For chunked files, check chunk 0.
  // We check both to cover all cases regardless of which scheme was used last run.
  return (
    db.isExtracted(makeConversationId(agentId, filePath)) ||
    db.isExtracted(makeConversationId(agentId, filePath, 0))
  );
}

// ---------------------------------------------------------------------------
// Per-file processing
// ---------------------------------------------------------------------------

type FileResult = {
  skipped: boolean;
  reason?: string;
  chunksProcessed: number;
  factsExtracted: number;
  factsUpdated: number;
  factsDeduplicated: number;
  error?: string;
};

async function processFile(
  db: ConversationDB,
  agentId: string,
  filePath: string,
  model: string,
  dryRun: boolean,
  force: boolean,
): Promise<FileResult> {
  const name = basename(filePath);

  // Skip system files
  if (SYSTEM_FILES.has(name)) {
    log(`Skip [system file]: ${name}`);
    return { skipped: true, reason: "system file", chunksProcessed: 0, factsExtracted: 0, factsUpdated: 0, factsDeduplicated: 0 };
  }

  // Skip already-processed (unless --force)
  if (!force && isFileAlreadyExtracted(db, agentId, filePath)) {
    log(`Skip [already done]: ${name}`);
    return { skipped: true, reason: "already processed", chunksProcessed: 0, factsExtracted: 0, factsUpdated: 0, factsDeduplicated: 0 };
  }

  // Read file content
  let content: string;
  try {
    content = readFileSync(filePath, "utf8");
  } catch (err) {
    logger.warn(`Cannot read ${filePath}: ${String(err)}`);
    return { skipped: false, chunksProcessed: 0, factsExtracted: 0, factsUpdated: 0, factsDeduplicated: 0, error: String(err) };
  }

  if (!content.trim()) {
    log(`Skip [empty]: ${name}`);
    return { skipped: true, reason: "empty file", chunksProcessed: 0, factsExtracted: 0, factsUpdated: 0, factsDeduplicated: 0 };
  }

  const fileSizeKb = Math.round(Buffer.byteLength(content, "utf8") / 1024);
  const chunks = chunkText(content, CHUNK_SIZE_BYTES);
  const isLarge = Buffer.byteLength(content, "utf8") > FILE_CHUNK_THRESHOLD_BYTES;

  if (dryRun) {
    log(
      `[DRY RUN] Would process: ${name}` +
        (isLarge ? ` (${fileSizeKb}KB → ${chunks.length} chunks)` : ` (${fileSizeKb}KB)`),
    );
    return { skipped: false, chunksProcessed: chunks.length, factsExtracted: 0, factsUpdated: 0, factsDeduplicated: 0 };
  }

  log(`Processing: ${name} (${fileSizeKb}KB${chunks.length > 1 ? `, ${chunks.length} chunks` : ""})`);

  let totalExtracted = 0;
  let totalUpdated = 0;
  let totalDeduped = 0;
  let lastError: string | undefined;

  for (let i = 0; i < chunks.length; i++) {
    const chunk = chunks[i];
    const conversationId = chunks.length > 1
      ? makeConversationId(agentId, filePath, i)
      : makeConversationId(agentId, filePath);
    const chunkLabel = chunks.length > 1 ? ` [chunk ${i + 1}/${chunks.length}]` : "";

    const now = Date.now();
    const conversation: ConversationRow = {
      id: conversationId,
      agent_id: agentId,
      session_key: `migration:${agentId}:${filePath}`,
      channel: "migration",
      started_at: now,
      ended_at: now,
      // Large value so ExtractionTrigger's minTurns check is irrelevant if called
      turn_count: 999,
      raw_text: chunks.length > 1
        ? `[MEMORY FILE: ${name}${chunkLabel}]\n\n${chunk}`
        : `[MEMORY FILE: ${name}]\n\n${chunk}`,
      metadata: JSON.stringify({
        source: "migration",
        filePath,
        chunkIndex: i,
        totalChunks: chunks.length,
      }),
    };

    // Insert conversation row (idempotent — OR IGNORE)
    try {
      db.insertConversationWithMessages(conversation, []);
    } catch (err) {
      logger.warn(`DB insert failed for ${name}${chunkLabel}: ${String(err)}`);
    }

    // Fetch existing facts for dedup context
    const existingFacts = db.getRelevantFacts(agentId, EXISTING_FACTS_LIMIT);

    // Call extraction LLM
    log(`  Extracting${chunkLabel}...`);
    const extractionResult = await extractFacts(conversation, existingFacts, model, logger);

    if (extractionResult.error) {
      logger.warn(`  Extraction error${chunkLabel}: ${extractionResult.error}`);
      db.logExtraction(conversationId, {
        modelUsed: extractionResult.modelUsed,
        factsExtracted: 0,
        factsUpdated: 0,
        factsDeduplicated: 0,
        error: extractionResult.error,
      });
      lastError = extractionResult.error;
    } else if (extractionResult.facts.length === 0) {
      log(`  No facts extracted${chunkLabel}`);
      db.logExtraction(conversationId, {
        modelUsed: extractionResult.modelUsed,
        factsExtracted: 0,
        factsUpdated: 0,
        factsDeduplicated: 0,
        error: null,
      });
    } else {
      const dedup = await processExtractedFacts(
        extractionResult.facts,
        conversationId,
        agentId,
        db,
        null, // embeddingEngine — not available in batch migration
        logger,
      );
      totalExtracted += dedup.factsExtracted;
      totalUpdated += dedup.factsUpdated;
      totalDeduped += dedup.factsDeduplicated;

      db.logExtraction(conversationId, {
        modelUsed: extractionResult.modelUsed,
        factsExtracted: dedup.factsExtracted,
        factsUpdated: dedup.factsUpdated,
        factsDeduplicated: dedup.factsDeduplicated,
        error: null,
      });

      log(
        `  Done${chunkLabel}: ${dedup.factsExtracted} new, ` +
          `${dedup.factsUpdated} updated, ${dedup.factsDeduplicated} dupes`,
      );
    }

    // Delay between chunks to respect API rate limits
    if (i < chunks.length - 1) {
      await sleep(EXTRACTION_DELAY_MS);
    }
  }

  return {
    skipped: false,
    chunksProcessed: chunks.length,
    factsExtracted: totalExtracted,
    factsUpdated: totalUpdated,
    factsDeduplicated: totalDeduped,
    error: lastError,
  };
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------

export async function runMigration(): Promise<void> {
  const opts = parseArgs();

  // Credentials are resolved by the extractor from environment variables:
  //   ANTHROPIC_API_KEY | OPENAI_API_KEY | MISTRAL_API_KEY | MEMENTO_API_KEY
  // Set the appropriate variable before running this script.

  console.log("=".repeat(60));
  console.log("Memento Migration");
  console.log(`DB:     ${opts.dbPath}`);
  console.log(`Model:  ${opts.model}`);
  console.log(`Agents: ${opts.agents.map((a) => a.agentId).join(", ")}`);
  if (opts.dryRun) console.log("Mode:   DRY RUN (no writes)");
  if (opts.force) console.log("Force:  yes (reprocess already-done files)");
  console.log("=".repeat(60));

  const db = new ConversationDB(opts.dbPath);

  type AgentStats = {
    total: number;
    skipped: number;
    processed: number;
    factsExtracted: number;
    factsUpdated: number;
    factsDeduplicated: number;
    errors: number;
  };

  const summary: Record<string, AgentStats> = {};

  for (const agent of opts.agents) {
    console.log(`\n${"─".repeat(60)}`);
    console.log(`Agent: ${agent.agentId}  (${agent.workspace})`);
    console.log("─".repeat(60));

    if (!existsSync(agent.workspace)) {
      console.warn(`  Workspace not found — skipping: ${agent.workspace}`);
      continue;
    }

    // Expand all glob patterns, deduplicate
    const fileSet = new Set<string>();
    for (const pattern of agent.paths) {
      for (const f of expandGlob(agent.workspace, pattern)) {
        fileSet.add(f);
      }
    }
    const files = [...fileSet].sort();

    log(`${files.length} file(s) found`);

    const stats: AgentStats = {
      total: files.length,
      skipped: 0,
      processed: 0,
      factsExtracted: 0,
      factsUpdated: 0,
      factsDeduplicated: 0,
      errors: 0,
    };

    for (let i = 0; i < files.length; i++) {
      const filePath = files[i];
      const result = await processFile(
        db,
        agent.agentId,
        filePath,
        opts.model,
        opts.dryRun,
        opts.force,
      );

      if (result.skipped) {
        stats.skipped++;
      } else {
        stats.processed++;
        stats.factsExtracted += result.factsExtracted;
        stats.factsUpdated += result.factsUpdated;
        stats.factsDeduplicated += result.factsDeduplicated;
        if (result.error) stats.errors++;

        // Delay between files (not after the last one)
        if (i < files.length - 1) {
          await sleep(EXTRACTION_DELAY_MS);
        }
      }
    }

    summary[agent.agentId] = stats;

    console.log(`\n  Summary for ${agent.agentId}:`);
    console.log(`    Files found:     ${stats.total}`);
    console.log(`    Skipped:         ${stats.skipped}`);
    console.log(`    Processed:       ${stats.processed}`);
    if (!opts.dryRun) {
      console.log(`    Facts new:       ${stats.factsExtracted}`);
      console.log(`    Facts updated:   ${stats.factsUpdated}`);
      console.log(`    Facts deduped:   ${stats.factsDeduplicated}`);
      if (stats.errors > 0) {
        console.log(`    Errors:          ${stats.errors}`);
      }
    }
  }

  // Overall totals
  console.log(`\n${"=".repeat(60)}`);
  console.log("Migration complete!");
  if (opts.dryRun) {
    console.log("(DRY RUN — no changes written to DB)");
  } else {
    let totalNew = 0;
    let totalUp = 0;
    let totalDe = 0;
    for (const s of Object.values(summary)) {
      totalNew += s.factsExtracted;
      totalUp += s.factsUpdated;
      totalDe += s.factsDeduplicated;
    }
    console.log(`Total facts new:     ${totalNew}`);
    console.log(`Total facts updated: ${totalUp}`);
    console.log(`Total facts deduped: ${totalDe}`);
  }
  console.log("=".repeat(60));

  db.close();
}

// Run when invoked directly (npx tsx src/extraction/migrate.ts ...)
runMigration().catch((err) => {
  console.error("Fatal:", err);
  process.exit(1);
});


path: src/extraction/trigger.ts
content: /**
 * Extraction Trigger
 *
 * Called by the segment writer after a conversation segment is saved to SQLite.
 * Runs fact extraction asynchronously so it never blocks the message flow.
 *
 * Responsibilities:
 *   - Skip segments that are too short (configurable minTurnsForExtraction)
 *   - Rate-limit extraction calls (configurable maxExtractionsPerMinute)
 *   - Fetch existing facts for dedup context
 *   - Call the LLM extractor
 *   - Run deduplication
 *   - Log the result (success or error) to extraction_log
 *
 * All errors are caught and logged — this code must never crash the plugin.
 */

import type { ConversationDB, ExtractionLogResult } from "../storage/db.js";
import type { ExtractionConfig } from "../config.js";
import type { ConversationRow } from "../storage/schema.js";
import type { PluginLogger } from "../types.js";
import { extractFacts } from "./extractor.js";
import type { OpenClawConfig } from "./embedded-runner.js";
import { processExtractedFacts } from "./deduplicator.js";
import { EmbeddingEngine } from "../storage/embeddings.js";
import { incrementalConsolidate } from "../consolidation/consolidator.js";

export class ExtractionTrigger {
  /** How many extractions have fired in the current 60-second window */
  private extractionsThisMinute = 0;
  /** Timestamp when the current rate-limit window started */
  private minuteWindowStart = Date.now();

  constructor(
    private readonly db: ConversationDB,
    private readonly cfg: ExtractionConfig,
    /** Full model string, e.g. "anthropic/claude-sonnet-4-6" */
    private readonly extractionModel: string,
    private readonly logger: PluginLogger,
    /** OpenClaw config passed from plugin api — enables model routing */
    private readonly openClawConfig?: OpenClawConfig,
    /** Embedding engine for Phase 2 dedup — passed if available */
    private readonly embeddingEngine: EmbeddingEngine | null = null,
  ) {}

  /**
   * Schedule extraction for a captured conversation segment.
   * Returns immediately — extraction runs in the next event-loop tick.
   */
  triggerAsync(conversation: ConversationRow): void {
    setImmediate(() => {
      this.runExtraction(conversation).catch((err) => {
        this.logger.warn(
          `memento: unhandled extraction error for ${conversation.id}: ${String(err)}`,
        );
      });
    });
  }

  // ---- Private ------------------------------------------------------------

  private async runExtraction(conversation: ConversationRow): Promise<void> {
    // Skip segments that are too short to yield useful facts
    if (conversation.turn_count < this.cfg.minTurnsForExtraction) {
      this.logger.debug?.(
        `memento: skipping extraction for ${conversation.id} — ` +
          `only ${conversation.turn_count} turns (min: ${this.cfg.minTurnsForExtraction})`,
      );
      return;
    }

    // Reset rate-limit window every 60 seconds
    const now = Date.now();
    if (now - this.minuteWindowStart >= 60_000) {
      this.extractionsThisMinute = 0;
      this.minuteWindowStart = now;
    }

    if (this.extractionsThisMinute >= this.cfg.maxExtractionsPerMinute) {
      this.logger.warn(
        `memento: extraction rate limit reached (${this.cfg.maxExtractionsPerMinute}/min), ` +
          `skipping segment ${conversation.id}`,
      );
      return;
    }
    this.extractionsThisMinute++;

    this.logger.debug?.(
      `memento: starting extraction for ${conversation.id} ` +
        `(${conversation.turn_count} turns, agent: ${conversation.agent_id})`,
    );

    try {
      // Fetch existing active facts for dedup context
      // Call the LLM
      const extractionResult = await extractFacts(
        conversation,
        [],  // Phase 1: history-agnostic — no existing facts passed to LLM
        this.extractionModel,
        this.logger,
        this.openClawConfig,
      );

      if (extractionResult.error) {
        this.logger.warn(
          `memento: extraction failed for ${conversation.id}: ${extractionResult.error}`,
        );
        this.logResult(conversation.id, {
          modelUsed: extractionResult.modelUsed,
          factsExtracted: 0,
          factsUpdated: 0,
          factsDeduplicated: 0,
          error: extractionResult.error,
        });
        return;
      }

      if (extractionResult.facts.length === 0) {
        this.logger.debug?.(
          `memento: no facts extracted from ${conversation.id}`,
        );
        this.logResult(conversation.id, {
          modelUsed: extractionResult.modelUsed,
          factsExtracted: 0,
          factsUpdated: 0,
          factsDeduplicated: 0,
          error: null,
        });
        return;
      }

      // Deduplicate and persist
      const dedup = await processExtractedFacts(
        extractionResult.facts,
        conversation.id,
        conversation.agent_id,
        this.db,
        this.embeddingEngine,
        this.logger,
      );

      this.logResult(conversation.id, {
        modelUsed: extractionResult.modelUsed,
        ...dedup,
        error: null,
      });

      this.logger.info(
        `memento: [${conversation.id}] extracted ` +
          `${dedup.factsExtracted} new, ` +
          `${dedup.factsUpdated} updated, ` +
          `${dedup.factsDeduplicated} deduplicated ` +
          `(agent: ${conversation.agent_id})`,
      );

      // Run incremental consolidation after extraction (cheap, no LLM)
      try {
        incrementalConsolidate(this.db, conversation.agent_id, this.logger);
      } catch (consolErr) {
        this.logger.warn(
          `memento: incremental consolidation error after ${conversation.id}: ${String(consolErr)}`,
        );
      }
    } catch (err) {
      const errMsg = String(err);
      this.logger.warn(
        `memento: extraction threw for ${conversation.id}: ${errMsg}`,
      );
      this.logResult(conversation.id, {
        modelUsed: this.extractionModel,
        factsExtracted: 0,
        factsUpdated: 0,
        factsDeduplicated: 0,
        error: errMsg,
      });
    }
  }

  private logResult(
    conversationId: string,
    result: ExtractionLogResult,
  ): void {
    try {
      this.db.logExtraction(conversationId, result);
    } catch (err) {
      this.logger.warn(
        `memento: failed to log extraction result for ${conversationId}: ${String(err)}`,
      );
    }
  }
}


path: src/index.ts
content: /**
 * Memento — Local Persistent Memory for OpenClaw Agents
 *
 * Phase 1 (Capture): Hooks into message events to buffer all conversations
 * per session, then flushes complete segments to SQLite + JSONL when a pause
 * is detected, the buffer is full, or the session ends.
 *
 * Phase 2 (Extraction): After each segment is captured, asynchronously
 * extracts structured facts using the configured LLM. Facts are deduplicated,
 * classified by visibility, and stored in SQLite with temporal tracking.
 *
 * Phase 3 (Recall): Before each AI turn, searches the knowledge base for
 * facts relevant to the current user message and injects them as context
 * via the `before_prompt_build` hook. Uses FTS5 keyword search + recency/
 * frequency scoring. Semantic embedding search planned for Phase 5.
 *
 * Phase 4 (Master KB): Cross-agent knowledge sharing. Each agent's recall
 * searches its own facts (all visibility levels) plus shared facts from
 * other agents. Private and secret facts never cross agent boundaries.
 * Cross-agent facts are scored lower than own-agent facts and displayed
 * with provenance tags (e.g., "[via agent-id]" or a configured display name).
 */

import { resolveConfig } from "./config.js";
import { ConversationBuffer } from "./capture/buffer.js";
import { SegmentWriter } from "./capture/writer.js";
import { ExtractionTrigger } from "./extraction/trigger.js";
import { searchRelevantFacts } from "./recall/search.js";
import { buildRecallContext } from "./recall/context-builder.js";
import { EmbeddingEngine } from "./storage/embeddings.js";

// ---------------------------------------------------------------------------
// Plugin definition
// ---------------------------------------------------------------------------

const mementoPlugin = {
  id: "memento",
  name: "Memento",
  description:
    "Memento — Local persistent memory with temporal intelligence for OpenClaw agents.",

  register(api: any): void {
    const cfg = resolveConfig(api.pluginConfig);

    if (!cfg.autoCapture) {
      api.logger.info(
        "memento: autoCapture is disabled — hooks not registered",
      );
      return;
    }

    // Resolve the data directory from the plugin config.
    // resolvePath expands ~ and resolves relative paths against workspace.
    const dataDir = api.resolvePath(cfg.dataDir);

    const writer = new SegmentWriter(dataDir, api.logger);

    // ── Embedding engine (lazy-loaded on first recall) ─────────────────────
    const embeddingEngine = new EmbeddingEngine(
      cfg.embeddingModel,
      api.logger,
    );

    // ── Phase 2: extraction trigger ─────────────────────────────────────────
    // Created unconditionally (the trigger checks cfg.extraction.autoExtract).
    // Shares the same DB connection as the writer to avoid opening a second handle.
    let trigger: ExtractionTrigger | null = null;

    if (cfg.extraction.autoExtract) {
      trigger = new ExtractionTrigger(
        writer.getDb(),
        cfg.extraction,
        cfg.extractionModel,
        api.logger,
        api.config,  // Pass OpenClaw config for model routing
        embeddingEngine, // Phase 2: embedding-based dedup
      );

      api.logger.info(
        `memento: extraction enabled (model: ${cfg.extractionModel}, ` +
          `minTurns: ${cfg.extraction.minTurnsForExtraction}, ` +
          `maxPerMin: ${cfg.extraction.maxExtractionsPerMinute})`,
      );
    } else {
      api.logger.info("memento: extraction disabled (autoExtract=false)");
    }

    const buffer = new ConversationBuffer(
      cfg,
      // flush handler — called by buffer whenever a segment is complete
      async (sessionKey, buf) => {
        const conversation = writer.writeSegment(sessionKey, buf);
        // Trigger extraction async — never awaited, never blocks capture
        if (conversation && trigger) {
          trigger.triggerAsync(conversation);
        }
      },
      api.logger,
    );

    api.logger.info(
      `memento: registered (dataDir: ${dataDir}, ` +
        `pauseTimeout: ${cfg.pauseTimeoutMs}ms, maxTurns: ${cfg.maxBufferTurns})`,
    );

    // -----------------------------------------------------------------------
    // Hook: message:received  — user message arriving via a channel
    // -----------------------------------------------------------------------
    api.on("message_received", async (event: any, ctx: any) => {
      try {
        // Key the buffer by conversationId when available; fall back to channelId.
        // conversationId typically maps to a Telegram chat / Discord thread / etc.
        const sessionKey = ctx.conversationId ?? ctx.channelId;

        await buffer.addMessage(sessionKey, {
          role: "user",
          content: event.content,
          timestamp: event.timestamp ?? Date.now(),
          messageId: undefined,
          channel: ctx.channelId,
        });
      } catch (err) {
        // Never propagate — capture failures must not disrupt the agent
        api.logger.warn(
          `memento: message_received error: ${String(err)}`,
        );
      }
    });

    // -----------------------------------------------------------------------
    // Hook: message:sent  — assistant response delivered to the channel
    // -----------------------------------------------------------------------
    api.on("message_sent", async (event: any, ctx: any) => {
      // Skip failed deliveries — nothing useful to capture
      if (!event.success) return;

      try {
        const sessionKey = ctx.conversationId ?? ctx.channelId;

        await buffer.addMessage(sessionKey, {
          role: "assistant",
          content: event.content,
          timestamp: Date.now(),
          messageId: undefined,
          channel: ctx.channelId,
        });
      } catch (err) {
        api.logger.warn(
          `memento: message_sent error: ${String(err)}`,
        );
      }
    });

    // -----------------------------------------------------------------------
    // Hook: before_reset  — user typed /new (or equivalent session reset)
    // Flush all active buffers so nothing is lost when the session clears.
    // -----------------------------------------------------------------------
    api.on("before_reset", async (_event: any, _ctx: any) => {
      try {
        await buffer.flushAll();
      } catch (err) {
        api.logger.warn(
          `memento: before_reset flush error: ${String(err)}`,
        );
      }
    });

    // -----------------------------------------------------------------------
    // Hook: session_end  — belt-and-suspenders flush when a session closes
    // -----------------------------------------------------------------------
    api.on("session_end", async (_event: any, _ctx: any) => {
      try {
        await buffer.flushAll();
      } catch (err) {
        api.logger.warn(
          `memento: session_end flush error: ${String(err)}`,
        );
      }
    });

    // -----------------------------------------------------------------------
    // Hook: before_prompt_build — Phase 3: Auto-Recall + Semantic Search
    // Injects relevant facts from the knowledge base before each AI turn.
    // Uses FTS5 keyword search + embedding cosine similarity when available.
    // -----------------------------------------------------------------------
    if (cfg.recall.autoRecall) {
      api.on("before_prompt_build", async (event: any, ctx: any) => {
        try {
          // Extract the latest user message from the conversation messages
          const messages: any[] = event.messages ?? [];
          const lastUserMsg = [...messages]
            .reverse()
            .find((m: any) => m.role === "user");

          if (!lastUserMsg) return {};

          // Get the text content from the message
          let queryText = "";
          if (typeof lastUserMsg.content === "string") {
            queryText = lastUserMsg.content;
          } else if (Array.isArray(lastUserMsg.content)) {
            // Multi-part content (text + images, etc.)
            queryText = lastUserMsg.content
              .filter((p: any) => p.type === "text")
              .map((p: any) => p.text)
              .join(" ");
          }

          if (queryText.length < cfg.recall.minQueryLength) return {};

          // Resolve agent ID from hook context
          const agentId = ctx.agentId ?? "unknown";

          // Generate query embedding (async, may return null if model not loaded).
          // ensureInit is lazy — first call loads the model; subsequent calls are instant.
          const queryEmbedding: number[] | null = await embeddingEngine.embed(queryText);

          // Search for relevant facts (FTS5 + semantic when embedding available)
          const scoredFacts = await searchRelevantFacts(
            writer.getDb(),
            agentId,
            queryText,
            cfg.recall,
            embeddingEngine.isReady ? embeddingEngine : null,
            queryEmbedding,
            api.config, // OpenClaw config for query planning (autoQueryPlanning)
          );

          if (scoredFacts.length === 0) return {};

          // Build the context block
          const contextBlock = buildRecallContext(
            scoredFacts,
            cfg.recall.maxContextChars,
            cfg.agentDisplay,
          );

          if (!contextBlock) return {};

          api.logger.debug?.(
            `memento: recall injecting ${scoredFacts.length} facts ` +
              `(${contextBlock.length} chars, semantic: ${queryEmbedding ? "yes" : "no"}) ` +
              `for agent ${agentId}`,
          );

          return { prependContext: contextBlock };
        } catch (err) {
          // Recall failures must never block the AI turn
          api.logger.warn(
            `memento: before_prompt_build recall error: ${String(err)}`,
          );
          return {};
        }
      });

      api.logger.info(
        `memento: recall enabled (maxFacts: ${cfg.recall.maxFacts}, ` +
          `maxChars: ${cfg.recall.maxContextChars}, ` +
          `embeddings: ${cfg.embeddingModel})`,
      );
    } else {
      api.logger.info("memento: recall disabled (autoRecall=false)");
    }

    // -----------------------------------------------------------------------
    // Service lifecycle — flush + close DB cleanly on gateway shutdown
    // -----------------------------------------------------------------------
    api.registerService({
      id: "memento",

      start: (ctx: any) => {
        api.logger.info(
          `memento: service started` +
            (ctx.workspaceDir ? ` (workspace: ${ctx.workspaceDir})` : ""),
        );
      },

      stop: async () => {
        try {
          if (buffer.size > 0) {
            api.logger.info(
              `memento: flushing ${buffer.size} active session(s) before shutdown`,
            );
            await buffer.flushAll();
          }
          await embeddingEngine.dispose();
          writer.close();
        } catch (err) {
          api.logger.warn(
            `memento: stop flush error: ${String(err)}`,
          );
        }
        api.logger.info("memento: service stopped");
      },
    });
  },
};

export default mementoPlugin;


path: src/recall/context-builder.ts
content: /**
 * Recall Context Builder
 *
 * Transforms scored facts from the search engine into a formatted
 * context block suitable for injection via `prependContext` in the
 * `before_prompt_build` hook.
 *
 * Design goals:
 *   - Compact but readable — every token costs money
 *   - Temporal awareness — include when facts were last seen
 *   - Pattern signals — flag recurring topics
 *   - Structured grouping — decisions first, then preferences, etc.
 */

import type { ScoredFact } from "./search.js";

// ---------------------------------------------------------------------------
// Category display order — most actionable first
// ---------------------------------------------------------------------------

const CATEGORY_ORDER: string[] = [
  "decision",
  "correction",
  "action_item",
  "preference",
  "person",
  "technical",
  "emotional",
  "routine",
];

// ---------------------------------------------------------------------------
// Formatting helpers
// ---------------------------------------------------------------------------

function formatRelativeTime(timestampMs: number): string {
  const diffMs = Date.now() - timestampMs;
  const mins = Math.floor(diffMs / 60_000);
  if (mins < 60) return `${mins}m ago`;
  const hours = Math.floor(mins / 60);
  if (hours < 24) return `${hours}h ago`;
  const days = Math.floor(hours / 24);
  if (days < 7) return `${days}d ago`;
  const weeks = Math.floor(days / 7);
  if (weeks < 5) return `${weeks}w ago`;
  const months = Math.floor(days / 30);
  return `${months}mo ago`;
}

function categoryLabel(cat: string): string {
  const labels: Record<string, string> = {
    decision: "📌 Decision",
    correction: "✏️ Correction",
    action_item: "📋 Action Item",
    preference: "⭐ Preference",
    person: "👤 Person",
    technical: "🔧 Technical",
    emotional: "💭 Emotional",
    routine: "📎 Routine",
  };
  return labels[cat] ?? `📎 ${cat}`;
}

/**
 * Recurring topic marker — flags facts that keep coming up.
 */
function recurrenceTag(occurrences: number): string {
  if (occurrences >= 5) return " 🔴 (recurring ×" + occurrences + ")";
  if (occurrences >= 3) return " 🟡 (mentioned ×" + occurrences + ")";
  return "";
}

/**
 * Resolve a human-readable label for an agent ID.
 * Falls back to the raw agentId if no display name is configured.
 */
function agentLabel(agentId: string, agentDisplay: Record<string, string>): string {
  return agentDisplay[agentId] ?? agentId;
}

// ---------------------------------------------------------------------------
// Builder
// ---------------------------------------------------------------------------

/**
 * Build the context block to inject before the AI turn.
 *
 * Returns an empty string if there are no facts to inject (the hook
 * should return `undefined` for prependContext in that case).
 */
export function buildRecallContext(
  facts: ScoredFact[],
  maxChars: number = 4000,
  agentDisplay: Record<string, string> = {},
): string {
  if (facts.length === 0) return "";

  // Group by category
  const grouped = new Map<string, ScoredFact[]>();
  for (const fact of facts) {
    const list = grouped.get(fact.category) ?? [];
    list.push(fact);
    grouped.set(fact.category, list);
  }

  const hasCrossAgent = facts.some((f) => f.isCrossAgent);

  const lines: string[] = [
    "## 🧠 Memento — Recalled Memory",
    hasCrossAgent
      ? "_Relevant facts from your knowledge base + shared knowledge (auto-injected):_"
      : "_Relevant facts from your knowledge base (auto-injected):_",
    "",
  ];

  let totalChars = lines.join("\n").length;

  // Iterate in priority order
  for (const cat of CATEGORY_ORDER) {
    const catFacts = grouped.get(cat);
    if (!catFacts || catFacts.length === 0) continue;

    const header = `### ${categoryLabel(cat)}`;
    if (totalChars + header.length + 2 > maxChars) break;
    lines.push(header);
    totalChars += header.length + 1;

    for (const fact of catFacts) {
      const when = formatRelativeTime(fact.last_seen_at);
      const recurrence = recurrenceTag(fact.occurrence_count);
      const source = fact.isCrossAgent ? ` [via ${agentLabel(fact.agent_id, agentDisplay)}]` : "";
      const graphTag = fact.matchSource === "graph" && fact.graphParentSummary
        ? ` 🔗 _(related to: ${fact.graphParentSummary.slice(0, 60)})_`
        : "";
      const previousTag = fact.previous_value
        ? ` _(previously: ${fact.previous_value.slice(0, 60)}${fact.previous_value.length > 60 ? "..." : ""})_`
        : "";
      const line = `- ${fact.summary ?? fact.content.slice(0, 120)} _(${when})_${recurrence}${source}${graphTag}${previousTag}`;

      if (totalChars + line.length + 1 > maxChars) break;
      lines.push(line);
      totalChars += line.length + 1;
    }

    lines.push("");
    totalChars += 1;
  }

  // Handle any categories not in CATEGORY_ORDER
  for (const [cat, catFacts] of grouped) {
    if (CATEGORY_ORDER.includes(cat)) continue;
    if (catFacts.length === 0) continue;

    const header = `### ${categoryLabel(cat)}`;
    if (totalChars + header.length + 2 > maxChars) break;
    lines.push(header);
    totalChars += header.length + 1;

    for (const fact of catFacts) {
      const when = formatRelativeTime(fact.last_seen_at);
      const recurrence = recurrenceTag(fact.occurrence_count);
      const source = fact.isCrossAgent ? ` [via ${agentLabel(fact.agent_id, agentDisplay)}]` : "";
      const graphTag = fact.matchSource === "graph" && fact.graphParentSummary
        ? ` 🔗 _(related to: ${fact.graphParentSummary.slice(0, 60)})_`
        : "";
      const previousTag = fact.previous_value
        ? ` _(previously: ${fact.previous_value.slice(0, 60)}${fact.previous_value.length > 60 ? "..." : ""})_`
        : "";
      const line = `- ${fact.summary ?? fact.content.slice(0, 120)} _(${when})_${recurrence}${source}${graphTag}${previousTag}`;

      if (totalChars + line.length + 1 > maxChars) break;
      lines.push(line);
      totalChars += line.length + 1;
    }

    lines.push("");
    totalChars += 1;
  }

  const result = lines.join("\n").trim();
  return result.length > 80 ? result : ""; // Don't inject just headers
}


path: src/types.ts
content: /**
 * Shared types used across Memento plugin modules.
 */

/**
 * Minimal logger interface compatible with OpenClaw's plugin API logger.
 * Defined once here to avoid duplication across the codebase.
 */
export type PluginLogger = {
  info: (...args: any[]) => void;
  warn: (...args: any[]) => void;
  error: (...args: any[]) => void;
  debug?: (...args: any[]) => void;
};


path: tsconfig.build.json
content: {
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "rootDir": "src",
    "outDir": "dist",
    "strict": true,
    "skipLibCheck": true,
    "declaration": false
  },
  "include": ["src/cli/ingest-tokens.ts", "src/cli/process-raw-sessions.ts"]
}


path: tsconfig.json
content: {
  "compilerOptions": {
    "target": "ES2022",
    "module": "ES2022",
    "moduleResolution": "bundler",
    "allowImportingTsExtensions": true,
    "strict": true,
    "noEmit": true,
    "skipLibCheck": true
  },
  "include": ["src/**/*.ts"]
}


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)