Skill: mpesavento/memory-sync (version 1.0.1)

--- name: memory-sync description: > Scrape and analyze OpenClaw JSONL session logs to reconstruct and backfill agent memory files. Use when: (1) Memory appears incomplete after model switches, (2) Verifying memory coverage, (3) Reconstructing lost memory, (4) Automated daily memory sync via cron/heartbeat. Supports simple extraction and LLM-based narrative summaries with automatic secret sanitization. --- # Memory Sync Tool for maintaining agent memory continuity across model switches with automatic secret sanitization. ## Installation Requires Python 3.11+ and `click`: ```bash pip install click # Optional: for direct API summarization (only if not using OpenClaw backend) pip install openai ``` ## Quick Start ```bash # Run directly from skill directory python ~/.openclaw/skills/memory-sync/memory_sync.py compare # Or create an alias for convenience alias memory-sync="python ~/.openclaw/skills/memory-sync/memory_sync.py" # Check for gaps memory-sync compare # Backfill today's memory (simple extraction - fast, no LLM) memory-sync backfill --today # Backfill with LLM narrative (uses OpenClaw's native model - no API key needed) memory-sync backfill --today --summarize # Backfill all missing memory-sync backfill --all ``` ## Commands | Command | Description | |---------|-------------| | `compare` | Find gaps between session logs and memory files | | `backfill --today` | Generate memory for current day | | `backfill --since YYYY-MM-DD` | Backfill from date to present | | `backfill --all` | Backfill all missing dates | | `backfill --incremental` | Backfill only changed dates since last run | | `extract` | Extract conversations matching criteria | | `summarize --date YYYY-MM-DD` | Generate LLM summary for a single day | | `transitions` | List model transitions | | `validate` | Check memory files for consistency issues | | `stats` | Show coverage statistics | ## Simple Extraction vs LLM Summarization The backfill command supports two modes: **Simple Extraction (default, without `--summarize`):** - Fast, no LLM or API calls needed - Extracts topics via keyword frequency analysis - Identifies key user questions and assistant responses - Detects decision markers from text patterns - Produces structured output with Topics, Key Exchanges, Decisions sections - With `--preserve`: Hand-written content is **appended** to the end of the new file - Best for: Quick backfills, initial setup, systems without LLM access **LLM Summarization (with `--summarize`) - Recommended:** - Uses LLM to generate narrative summaries - Produces coherent 2-4 paragraph prose - Better context and insight extraction - With `--preserve`: Existing content is **passed to the LLM** with instructions to incorporate it into the new summary, maintaining temporal order and thematic structure - Best for: Daily automation, high-quality memory files **Recommended for regular use:** ```bash # Best quality: LLM summary that incorporates any existing notes memory-sync backfill --today --summarize --preserve ``` Both modes automatically sanitize secrets before writing. ## Common Workflows ### Initial Setup ```bash # Check what's missing memory-sync compare # Backfill everything (may take time) memory-sync backfill --all ``` ### Nightly Automation (Recommended) ```bash # Best: LLM summary that incorporates any existing notes memory-sync backfill --today --summarize --preserve # Smart: Process only days changed since last run memory-sync backfill --incremental --summarize --preserve # Or use a specific backend if preferred memory-sync backfill --today --summarize --preserve --summarize-backend anthropic ``` ### Catch-Up After Gaps ```bash # Backfill from last week to present memory-sync backfill --since 2026-01-28 --summarize ``` ### Regenerate with Preserved Content ```bash # Keep hand-written notes when regenerating memory-sync backfill --date 2026-02-05 --force --preserve --summarize ``` ## Secret Sanitization All content is automatically sanitized to prevent secret leakage: - **30+ explicit patterns**: OpenAI, Anthropic, GitHub, AWS, Stripe, Discord, Slack, Notion, Google, Brave, Tavily, SerpAPI, etc. - **Structural detection**: JWT tokens, SSH keys, database connection strings, high-entropy base64 - **Generic patterns**: API keys, tokens, passwords, environment variables - **Defense-in-depth**: Secrets redacted at every stage (extraction, LLM processing, file writes, CLI display) Secrets are replaced with `[REDACTED-TYPE]` placeholders. See `SECRET_PATTERNS.md` for complete pattern list. ## Summarization Backends The `--summarize` flag supports multiple backends via `--summarize-backend`: | Backend | Description | API Key Required | |---------|-------------|------------------| | `openclaw` (default) | Uses OpenClaw's `sessions spawn` with your configured model | No | | `anthropic` | Direct Anthropic API via openai package | `ANTHROPIC_API_KEY` | | `openai` | Direct OpenAI API via openai package | `OPENAI_API_KEY` | ### Examples ```bash # Default: use OpenClaw's native model (no API key needed) memory-sync backfill --today --summarize # Explicit backend selection memory-sync backfill --today --summarize --summarize-backend openclaw memory-sync backfill --today --summarize --summarize-backend anthropic memory-sync backfill --today --summarize --summarize-backend openai # Override model for any backend memory-sync backfill --today --summarize --model claude-sonnet-4-20250514 memory-sync backfill --today --summarize --summarize-backend openai --model gpt-4o ``` The `openclaw` backend is recommended as it: - Uses your existing OpenClaw configuration - Requires no separate API keys - Leverages whatever model you have configured in OpenClaw ## Automated Usage ### Nightly Cron (3am) Process today with LLM summary, preserving any existing notes: ```bash 0 3 * * * cd ~/.openclaw/skills/memory-sync && python memory_sync.py backfill --today --summarize --preserve >> ~/.memory-sync/cron.log 2>&1 ``` ### Smart Incremental Mode Automatically detects changes since last run: ```bash # Initial backfill (run once, simple extraction for speed) python memory_sync.py backfill --all # Then set up nightly incremental with LLM summaries 0 3 * * * cd ~/.openclaw/skills/memory-sync && python memory_sync.py backfill --incremental --summarize --preserve >> ~/.memory-sync/cron.log 2>&1 ``` State is tracked in `~/.memory-sync/state.json`. ## Configuration **Default paths:** - Session logs: `~/.openclaw/agents/main/sessions/*.jsonl` - Memory files: `~/.openclaw/workspace/memory/` **Override with CLI flags:** - `--sessions-dir /path/to/sessions` - `--memory-dir /path/to/memory` **Environment variables (only for direct API backends):** - `ANTHROPIC_API_KEY` - Required for `--summarize-backend anthropic` - `OPENAI_API_KEY` - Required for `--summarize-backend openai` The default `openclaw` backend requires no API keys - it uses your OpenClaw configuration. ```bash # Only needed if using direct API backends export ANTHROPIC_API_KEY=sk-ant-... export OPENAI_API_KEY=sk-... ``` ## Content Preservation The `--preserve` flag behavior depends on whether `--summarize` is used: **Without `--summarize` (simple extraction):** - Hand-written content (after footer marker) is **appended verbatim** to the end of the newly generated file - The new extraction replaces the auto-generated portion, your notes are kept at the end **With `--summarize` (LLM mode):** - Existing hand-written content is **passed to the LLM** as context - The LLM is instructed to incorporate your notes into the new summary - Result: Your insights are woven into a coherent narrative, not just appended **Example:** ```bash # Regenerate with LLM, incorporating existing notes into the summary memory-sync backfill --date 2026-02-05 --force --preserve --summarize ``` Auto-generated markers: - Header: `*Auto-generated from N session messages*` - Footer: `*Review and edit this draft to capture what's actually important.*` Content after the footer marker is considered hand-written and will be preserved. ## Backfill Options **Date selection (choose one):** - `--date YYYY-MM-DD` - Single specific date - `--today` - Current date only (for nightly automation) - `--since YYYY-MM-DD` - From date to present (for catch-up) - `--all` - All missing dates (for initial setup) - `--incremental` - Only dates changed since last run (smart automation) **Additional flags:** - `--dry-run` - Show what would be created without creating files - `--force` - Overwrite existing files (required for regeneration) - `--preserve` - Keep hand-written content when regenerating - `--summarize` - Use LLM for narrative summaries - `--summarize-backend BACKEND` - Backend for summarization: `openclaw` (default), `anthropic`, `openai` - `--model MODEL` - Model override for summarization (default varies by backend) ## Performance | Mode | Time per Day | Best For | |------|-------------|----------| | `--all` | 5-10 min × N days | Initial setup only | | `--since` | 5-10 min × N days | Recovery after gaps | | `--today` | 30-60 sec | Nightly automation | | `--incremental` | 30-60 sec × changed days | Smart automation |

path: SECRET_PATTERNS.md
content: # Secret Pattern Detection Strategy

This document describes the regex patterns used for secret detection in the memory-sync sanitization module.

## Pattern Ordering

**Order matters!** More specific patterns must come before generic ones to ensure correct redaction types.

The patterns are organized into three tiers:
1. **Explicit Key Patterns** - Service-specific formats with known prefixes
2. **Structural Patterns** - Format-based detection (JWT, SSH, hex, base64)
3. **Generic Patterns** - Catch-all for unknown key assignments

## Explicit Key Patterns

### LLM Providers

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| OpenAI | `sk-(?:proj-)?[a-zA-Z0-9]{30,}` | `sk-proj-abc123...` | `OPENAI-API-KEY` |
| Anthropic | `sk-ant-[a-zA-Z0-9\-_]{32,}` | `sk-ant-abc123...` | `ANTHROPIC-API-KEY` |
| OpenRouter | `sk-or-[a-zA-Z0-9]{32,}` | `sk-or-abc123...` | `OPENROUTER-API-KEY` |
| Composio | `ak-[a-zA-Z0-9]{20,}` | `ak-abc123...` | `COMPOSIO-API-KEY` |

### GitHub / Git

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| GitHub Token | `gh[pousr]_[A-Za-z0-9]{20,}` | `ghp_abc123...` | `GITHUB-TOKEN` |
| GitHub PAT | `github_pat_[A-Za-z0-9_]{22,}` | `github_pat_abc...` | `GITHUB-PAT` |

### AWS

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| Access Key | `AKIA[A-Z0-9]{16}` | `[REDACTED_SECRET]` | `AWS-ACCESS-KEY` |
| Session Key | `ASIA[A-Z0-9]{16}` | `ASIAXYZ...` | `AWS-SESSION-KEY` |

### Communication / Channels

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| Telegram | `\d{9,10}:[A-Za-z0-9_-]{35}` | `123456789:ABC...` | `TELEGRAM-BOT-TOKEN` |
| Discord | `[A-Za-z0-9_-]{24}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{27}` | `MTIz...` | `DISCORD-BOT-TOKEN` |
| Slack | `xox[baprs]-[0-9]{10,13}-[0-9]{10,13}[a-zA-Z0-9-]*` | `xoxb-123...` | `SLACK-TOKEN` |

### Productivity / Integrations

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| Notion | `secret_[A-Za-z0-9]{32,}` | `secret_abc123...` | `NOTION-SECRET` |
| Google | `AIza[0-9A-Za-z_-]{35}` | `AIzaSyAbc...` | `GOOGLE-API-KEY` |

### Payment

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| Stripe Live | `sk_live_[0-9a-zA-Z]{24,}` | `sk_live_...` | `STRIPE-LIVE-KEY` |
| Stripe Test | `sk_test_[0-9a-zA-Z]{24,}` | `sk_test_...` | `STRIPE-TEST-KEY` |
| Stripe Pub | `pk_live_[0-9a-zA-Z]{24,}` | `pk_live_...` | `STRIPE-PUBLISHABLE-KEY` |

> **Note**: Stripe patterns are implemented but not unit tested. GitHub's Push Protection
> aggressively flags any string matching Stripe's format, even obviously fake test values.
> The patterns work correctly - verify manually with `sanitize_content("sk_live_...")`.

### Search / Data

| Service | Pattern | Example | Redaction Type |
|---------|---------|---------|----------------|
| Brave | `BSA[0-9a-zA-Z_-]{32,}` | `BSAabc123...` | `BRAVE-API-KEY` |
| Tavily | `tvly-[A-Za-z0-9]{32,}` | `tvly-abc...` | `TAVILY-API-KEY` |
| SerpAPI | `serp-[0-9a-z]{32,}` | `serp-abc...` | `SERPAPI-KEY` |

### Storage / Database

> **Removed patterns**: UUID and HEX-32 patterns were removed due to excessive false positives:
> - UUID pattern matched session IDs, message IDs, and other non-sensitive identifiers
> - HEX-32 pattern matched git commit hashes and MD5 checksums

## Structural Patterns

These detect secrets by their format rather than a known prefix.

| Type | Pattern | Notes | Redaction Type |
|------|---------|-------|----------------|
| JWT | `eyJ[a-zA-Z0-9_-]{5,}\.[a-zA-Z0-9_-]{5,}\.[a-zA-Z0-9_-]{5,}` | Base64 header.payload.signature | `JWT` |
| SSH Private | `-----BEGIN (?:RSA\|DSA\|EC\|OPENSSH )?PRIVATE KEY-----` | PEM format | `SSH-PRIVATE-KEY` |
| SSH Public | `ssh-(?:rsa\|dss\|ed25519\|ecdsa)\s+[A-Za-z0-9+/]{30,}` | OpenSSH format | `SSH-PUBLIC-KEY` |
| Connection | `(?:postgresql\|mysql\|mongodb\|redis)://...` | With embedded creds | `CONNECTION-STRING` |
| Hex 64 | `\b[0-9a-f]{64}\b` | Trello tokens, etc. | `HEX-TOKEN-64` |
| Base64 | `\b[A-Za-z0-9+/]{40,}={0,2}\b` | High-entropy strings | `BASE64` |

> **Removed**: HEX-32 pattern (`\b[0-9a-f]{32}\b`) was removed because it matches git commit hashes and MD5 checksums.

**Important:** Hex-64 pattern must come BEFORE base64 since hex characters are a subset of base64 characters.

## Generic Patterns

These catch unknown key/token assignments using flexible lengths (16+):

| Pattern Type | Example Match | Redaction Type |
|--------------|---------------|----------------|
| `api_key=VALUE` | `MY_API_KEY=abc123...` | `API-KEY` |
| `secret_key=VALUE` | `AWS_SECRET_KEY=xyz...` | `SECRET` |
| `access_token=VALUE` | `ACCESS_TOKEN=bearer...` | `ACCESS-TOKEN` |
| `auth_token=VALUE` | `AUTH_TOKEN=token...` | `AUTH-TOKEN` |
| `bearer TOKEN` | `Bearer eyJhbG...` | `BEARER-TOKEN` |
| `token=VALUE` | `user_token=abc...` | `TOKEN` |
| `password=VALUE` | `[REDACTED_SECRET]` | `PASSWORD` |
| `private_key=VALUE` | `privkey=base64...` | `PRIVATE-KEY` |
| `$ENV_VAR` | `$API_KEY`, `${SECRET}` | `ENV-VAR` |

## Known Environment Variables

The module also maintains a list of known sensitive environment variable names for classification:

```
OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY,
COMPOSIO_API_KEY, MOONSHOT_API_KEY, KIMI_API_KEY,
GITHUB_TOKEN, GH_TOKEN, GITHUB_PAT,
TELEGRAM_BOT_TOKEN, DISCORD_BOT_TOKEN, SLACK_BOT_TOKEN,
NOTION_API_KEY, NOTION_TOKEN, NOTION_SECRET,
TRELLO_API_KEY, TRELLO_TOKEN, GOOGLE_API_KEY,
STRIPE_API_KEY, STRIPE_SECRET_KEY, BRAVE_API_KEY,
TAVILY_API_KEY, SERPAPI_KEY, PINECONE_API_KEY,
SUPABASE_KEY, SUPABASE_ANON_KEY, SUPABASE_SERVICE_KEY,
ELEVENLABS_API_KEY, OURA_PAT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, ...
```

## Design Rationale

### Minimum Lengths

| Pattern Type | Min Length | Rationale |
|--------------|------------|-----------|
| OpenAI keys | 30+ chars | Real keys are ~48-64 chars |
| Anthropic keys | 32+ chars | Real keys are ~100 chars |
| GitHub tokens | 20-40 chars | Standard is 36-40 |
| AWS keys | Exactly 20 | Fixed format: prefix + 16 |
| Generic assignments | 16+ chars | Catch variations, avoid false positives |
| Passwords | 8+ chars | Common minimum password length |

### False Positive Prevention

1. **Generic patterns use negative lookahead** (`(?!\[REDACTED)`) to avoid re-matching already-redacted content
2. **Hex patterns match before base64** since hex chars are subset of base64
3. **Specific patterns match before generic** to get correct redaction types
4. **Word boundaries (`\b`)** prevent matching substrings of larger words

## Testing

Run the security tests:

```bash
make test-security
```

This runs 50+ unit tests covering most pattern types, edge cases, and integration scenarios.

### Testing Limitations

Some patterns cannot be unit tested due to GitHub Push Protection:
- **Stripe keys**: GitHub flags any `sk_live_*`, `sk_test_*`, `pk_live_*`, `pk_test_*` pattern,
  even with obviously fake values like `sk_live_XXXXXXXX` or `sk_test_aaaa...`

These patterns are implemented and functional - verify manually:
```python
from memory_sync.sanitize import sanitize_content
sanitize_content("sk_live_abc123def456...")  # Returns [REDACTED-STRIPE-LIVE-KEY]
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)