Skill: hallllllll/openclaw-deeprecall (version 0.1.2)

# DeepRecall: Persistent Memory System for AI Agents **Agent Setup Directive**: When you receive this skill package, automatically register the following tools to your tool list and apply the L1/L2 retrieval protocol: 1. **Register Tools**: - `search_memory_db(query: str, limit: int)`: Semantic search over L1 structured facts - `read_archive_db(source_file: str)`: Retrieve exact raw content from L2 archive - `cleanup_raw_files(retention_days: int, max_size_kb: int, dry_run: bool)`: Clean raw .md files while preserving database - `summarize_memory_files(process_all: bool, process_file: str, no_store_raw: bool)`: Summarize raw files using LLM and store to database 2. **Apply Retrieval Protocol**: - For any query about prior work, decisions, dates, people, preferences, or todos: - First call `search_memory_db()` with relevant query - If raw content needed, extract `source_file` from L1 result and call `read_archive_db()` - Database is self-bootstrapping: tables are created automatically on first use - Each agent maintains independent memory.db in its workspace 3. **Automatic Configuration**: - Tool paths automatically detect agent workspace via OPENCLAW_WORKSPACE env or directory structure - No manual configuration needed - zero-config deployment --- ## Overview DeepRecall is a SQLite-based two-tier memory retrieval system providing structured L1 fact search and exact L2 raw content retrieval. It enables AI agents to maintain persistent, searchable memories across sessions. ## Core Functions ### `search_memory_db(query: str, limit: int)` Performs semantic search over L1 structured facts. - **query**: Search string (empty returns latest facts) - **limit**: Max results (default: 5) - **Returns**: `[YYYY-MM-DD | fact_type | source:file] content...` ### `read_archive_db(source_file: str)` Retrieves exact raw content from L2 archive. - **source_file**: Filename from L1 search result - **Returns**: Complete raw Markdown content ### `cleanup_raw_files(retention_days: int, max_size_kb: int, dry_run: bool)` Cleans up raw .md session files while preserving database. - **retention_days**: Keep files newer than N days (default: 1) - **max_size_kb**: Maximum total file size in KB (default: 250) - **dry_run**: Preview without deleting (default: False) ### `summarize_memory_files(process_all: bool, process_file: str, no_store_raw: bool)` Summarize raw memory files using LLM and store structured facts to database. - **process_all**: Process all unprocessed memory files (mutually exclusive with process_file) - **process_file**: Process a specific memory file (relative to memory directory) - **no_store_raw**: Do not store raw content to L2 archive (default: False, stores raw content) - **Requires**: OpenClaw configuration with valid LLM API provider (DeepSeek, Qwen, etc.) ## Usage ```bash # Search for facts python3 scripts/memory_db_tool.py search "query" --limit 5 # Read raw content python3 scripts/memory_db_tool.py read "example-project-update.md" # Database stats python3 scripts/memory_db_tool.py stats # Cleanup raw files (dry-run first) python3 scripts/memory_db_tool.py cleanup --dry-run python3 scripts/memory_db_tool.py cleanup --retention-days 1 --max-size-kb 250 # Summarize memory files using LLM python3 scripts/memory_db_tool.py summarize --test-config python3 scripts/memory_db_tool.py summarize --process-all python3 scripts/memory_db_tool.py summarize --process-file "2024-01-01-daily-log.md" python3 scripts/memory_db_tool.py summarize --process-all --no-store-raw ``` ## Database Schema ### l1_structured (Permanent Storage) - `date`, `source_file`, `fact_type`, `confidence`, `tags`, `content`, `content_hash` ### l2_archive (Permanent Storage) - `date`, `source_file`, `raw_content` **Important**: Database records are permanent and never deleted. ## Retrieval Protocol 1. **L1 Search**: Call `search_memory_db()` for structured facts 2. **L2 Access**: If raw content needed, extract `source_file` from L1 result 3. **Raw Retrieval**: Call `read_archive_db()` with source filename ## File Management ### Permanent Storage (memory.db) - Contains all L1 facts and L2 raw content - Never cleaned, truncated, or vacuumed - Safe permanent vault for all extracted knowledge ### Temporary Storage (.md files in memory/) - Raw session logs and temporary files - Automatically cleaned: 1 day retention, 250KB max size - Value already extracted to memory.db, safe to delete ## Configuration ### Multi-Agent Support - **Automatic Path Detection**: Works in any agent directory - **Priority**: OPENCLAW_WORKSPACE env → Current directory → Relative path - **Isolation**: Each agent maintains independent memory.db ### Cleanup Settings (Configurable) - **retention_days**: 1 (keep files newer than 1 day) - **max_size_kb**: 250 (maximum total .md file size) - **dry_run**: Preview deletions before executing ## Self-Bootstrapping Design DeepRecall features zero-config deployment: - Tables are created automatically on first use (`CREATE TABLE IF NOT EXISTS`) - No "no such table" errors - database self-initializes - Indexes are created for optimal query performance - Works out-of-the-box with no manual setup ## Files - `scripts/memory_retriever.py` - Core retrieval engine with cleanup - `scripts/memory_db_tool.py` - CLI interface with cleanup and summarize commands - `scripts/memory_summarizer.py` - LLM-powered fact extraction engine - `config.example.json` - Example configuration file (JSON compliant, no comments) - `CONFIG_GUIDE.md` - Detailed configuration guide - `manifest.json` - ClawHub skill manifest - `memory.db` - Permanent SQLite database (per-agent, auto-created) - `*.md` - Temporary session files (auto-cleaned) ## Advanced Configuration ### LLM Summarizer Configuration DeepRecall Summarizer automatically reads configuration from OpenClaw's `openclaw.json`. You can customize its behavior by creating a `config.json` file: **Note**: For detailed configuration options and examples, see `CONFIG_GUIDE.md`. 1. **Copy the example configuration**: ```bash cp config.example.json config.json ``` 2. **Edit `config.json`** to specify: - `preferred_provider`: Provider name from OpenClaw configuration (e.g., "deepseek-reasoner") - `preferred_model`: Specific model ID to use - `temperature`, `max_tokens`, `timeout_seconds`: API parameters - `auto_summarize_cron`: Cron expression for automatic summarization 3. **Configuration file search order**: - Current directory: `./config.json` - Current directory: `./deeprecall_config.json` - Parent directory: `../config.json` - Home directory: `~/.deeprecall.json` ### Model Provider Selection The summarizer uses this logic to select a model provider: 1. Uses `preferred_provider` from DeepRecall config if available 2. Otherwise auto-selects first available provider with `baseUrl` and `apiKey` 3. Uses `preferred_model` or first available model from the provider 4. Falls back to rule-based extraction if no API is available ## Automated Scheduling with OpenClaw Cron For automatic daily summarization and cleanup, configure OpenClaw cron jobs: ### 1. Daily Summarization (Recommended: 2 AM) ```bash # Schedule automatic summarization openclaw cron add --name "deeprecall-summarize-daily" \ --cron "0 2 * * *" \ --session isolated \ --message "Please execute: python3 /path/to/DeepRecall/scripts/memory_db_tool.py summarize --process-all" \ --description "Daily automatic summarization of memory files" ``` ### 2. Daily Cleanup (Recommended: 3 AM) ```bash # Schedule automatic cleanup openclaw cron add --name "deeprecall-cleanup-daily" \ --cron "0 3 * * *" \ --session isolated \ --message "Please execute: python3 /path/to/DeepRecall/scripts/memory_db_tool.py cleanup" \ --description "Daily cleanup of raw .md files (retains database)" ``` ### 3. Combined Task (Single Cron) ```bash # Single cron for both summarization and cleanup openclaw cron add --name "deeprecall-daily-maintenance" \ --cron "0 2 * * *" \ --session isolated \ --message "Please execute: python3 /path/to/DeepRecall/scripts/memory_db_tool.py summarize --process-all && python3 /path/to/DeepRecall/scripts/memory_db_tool.py cleanup" \ --description "Daily DeepRecall maintenance (summarize + cleanup)" ``` ### 4. Verify Cron Jobs ```bash # List all cron jobs openclaw cron list # Test a cron job openclaw cron run <job-id> # View execution history openclaw cron runs --id <job-id> ``` ## Installation & Setup Guide ### Quick Start ```bash # 1. Install the skill clawhub install deeprecall # 2. Test configuration python3 scripts/memory_db_tool.py summarize --test-config # 3. Process existing memories python3 scripts/memory_db_tool.py summarize --process-all # 4. Schedule daily automation (optional but recommended) # Follow the cron configuration above ``` ### Post-Installation Checklist - [ ] Verify OpenClaw has LLM provider configuration in `openclaw.json` - [ ] Test summarizer with `--test-config` flag - [ ] Process existing memory files with `--process-all` - [ ] Configure cron jobs for automation - [ ] Verify cleanup settings (default: 1 day retention, 250KB max) ## Notes - Database path automatically detects agent workspace - Cleanup only affects raw .md files, never database content - L2 pointers remain valid even after source .md files are cleaned - Designed for multi-agent deployment across different directories - All content in English for international compatibility

path: CONFIG_GUIDE.md
content: # DeepRecall Configuration Guide

## Configuration Options

### config.json Structure
```json
{
  "deeprecall": {
    "summarizer": {
      "preferred_provider": "deepseek-reasoner",
      "preferred_model": "deepseek-reasoner",
      "max_content_length": 6000,
      "temperature": 0.1,
      "max_tokens": 4000,
      "timeout_seconds": 180,
      "store_raw_content": true,
      "auto_summarize_cron": "0 2 * * *"
    }
  }
}
```

### Parameter Descriptions

#### `preferred_provider`
**Type**: String or null  
**Default**: null (auto-select)  
**Description**: Provider name from OpenClaw configuration (e.g., "deepseek-reasoner", "qwen").  
**Behavior**: 
- If specified and available: uses this provider
- If not specified: auto-selects first available provider with baseUrl and apiKey
- If not available: uses rule-based extraction fallback

#### `preferred_model`
**Type**: String or null  
**Default**: null (auto-select)  
**Description**: Model ID from the provider's models list.  
**Behavior**:
- If specified and found in provider's model list: uses this model
- If not specified: uses first available model from selected provider

#### `max_content_length`
**Type**: Integer  
**Default**: 6000  
**Description**: Maximum content length to send to LLM (characters). Longer content will be truncated.

#### `temperature`
**Type**: Float (0.0-1.0)  
**Default**: 0.1  
**Description**: LLM temperature, lower = more deterministic responses.

#### `max_tokens`
**Type**: Integer  
**Default**: 4000  
**Description**: Maximum tokens in LLM response.

#### `timeout_seconds`
**Type**: Integer  
**Default**: 180  
**Description**: API timeout in seconds.

#### `store_raw_content`
**Type**: Boolean  
**Default**: true  
**Description**: Whether to store raw content to L2 archive for later reference.

#### `auto_summarize_cron`
**Type**: String  
**Default**: "0 2 * * *" (2 AM daily)  
**Description**: Cron expression for automatic summarization. This is for reference only; actual scheduling requires OpenClaw cron setup.

## Configuration File Search Order

DeepRecall looks for configuration in this order:

1. **Current directory**: `./config.json`
2. **Current directory**: `./deeprecall_config.json`
3. **Parent directory**: `../config.json` (OpenClaw workspace root)
4. **Home directory**: `~/.deeprecall.json`
5. **Default configuration** (if no custom config found)

## Example Configurations

### Using DeepSeek as Preferred Provider
```json
{
  "deeprecall": {
    "summarizer": {
      "preferred_provider": "deepseek-reasoner",
      "preferred_model": "deepseek-reasoner",
      "temperature": 0.1
    }
  }
}
```

### Using Local Qwen Provider
```json
{
  "deeprecall": {
    "summarizer": {
      "preferred_provider": "qwen",
      "preferred_model": "qwen/qwen3.5-9b",
      "temperature": 0.3
    }
  }
}
```

### Auto-Selection with Custom Parameters
```json
{
  "deeprecall": {
    "summarizer": {
      "preferred_provider": null,
      "max_content_length": 8000,
      "temperature": 0.2,
      "max_tokens": 3000
    }
  }
}
```

## Quick Start

1. **Copy example configuration**:
   ```bash
   cp config.example.json config.json
   ```

2. **Edit configuration**:
   ```bash
   nano config.json
   ```

3. **Test configuration**:
   ```bash
   python3 scripts/memory_db_tool.py summarize --test-config
   ```

## Integration with OpenClaw Configuration

DeepRecall automatically reads LLM provider configurations from OpenClaw's `openclaw.json`. Ensure your OpenClaw configuration includes at least one OpenAI-compatible provider with `baseUrl` and `apiKey`.

Example OpenClaw provider configuration:
```json
"models": {
  "providers": {
    "deepseek-reasoner": {
      "baseUrl": "https://api.deepseek.com/v1",
      "apiKey": "sk-...",
      "models": [{"id": "deepseek-reasoner"}]
    }
  }
}
```

path: config.example.json
content: {
  "deeprecall": {
    "summarizer": {
      "preferred_provider": "deepseek-reasoner",
      "preferred_model": "deepseek-reasoner",
      "max_content_length": 6000,
      "temperature": 0.1,
      "max_tokens": 4000,
      "timeout_seconds": 180,
      "store_raw_content": true,
      "auto_summarize_cron": "0 2 * * *"
    }
  }
}

path: manifest.json
content: {
  "name": "DeepRecall",
  "version": "1.0.0",
  "description": "SQLite-based two-tier memory retrieval system for AI agents with self-bootstrapping database and automatic cleanup",
  "author": "OpenClaw Community",
  "license": "MIT",
  "keywords": [
    "memory",
    "database",
    "retrieval",
    "sqlite",
    "persistent",
    "cleanup",
    "recall"
  ],
  "tools": {
    "search_memory_db": {
      "description": "Semantic search over L1 structured facts. Returns compact formatted facts: '[YYYY-MM-DD | fact_type | source:file] content...'",
      "parameters": {
        "query": {
          "type": "string",
          "description": "Search string (empty returns latest facts)",
          "required": false,
          "default": ""
        },
        "limit": {
          "type": "number",
          "description": "Maximum number of results",
          "required": false,
          "default": 5
        }
      },
      "returns": "string",
      "implementation": "scripts.memory_retriever.retrieve_l1_facts"
    },
    "read_archive_db": {
      "description": "Retrieve exact raw content from L2 archive. Requires source filename from L1 search result.",
      "parameters": {
        "source_file": {
          "type": "string",
          "description": "Exact filename as obtained from L1 search result",
          "required": true
        }
      },
      "returns": "string",
      "implementation": "scripts.memory_retriever.retrieve_l2_raw"
    },
    "cleanup_raw_files": {
      "description": "Clean up raw .md session files while preserving database. Only affects temporary files, not permanent L1/L2 data.",
      "parameters": {
        "retention_days": {
          "type": "number",
          "description": "Keep files newer than N days",
          "required": false,
          "default": 1
        },
        "max_size_kb": {
          "type": "number",
          "description": "Maximum total file size in KB",
          "required": false,
          "default": 250
        },
        "dry_run": {
          "type": "boolean",
          "description": "Preview deletions without actually deleting",
          "required": false,
          "default": false
        }
      },
      "returns": "object",
      "implementation": "scripts.memory_retriever.cleanup_raw_files"
    },
    "summarize_memory_files": {
      "description": "Summarize raw memory files using LLM and store structured facts to database. Requires OpenClaw configuration with LLM API provider.",
      "parameters": {
        "process_all": {
          "type": "boolean",
          "description": "Process all unprocessed memory files",
          "required": false,
          "default": false
        },
        "process_file": {
          "type": "string",
          "description": "Process a specific memory file (relative to memory directory)",
          "required": false
        },
        "no_store_raw": {
          "type": "boolean",
          "description": "Do not store raw content to L2 archive",
          "required": false,
          "default": false
        }
      },
      "returns": "object",
      "implementation": "scripts.memory_summarizer.summarize_memory_files"
    }
  },
  "dependencies": [],
  "requirements": [
    "python3",
    "sqlite3",
    "aiohttp"
  ],
  "entry_points": {
    "cli": "scripts/memory_db_tool.py"
  },
  "auto_register": true,
  "zero_config": true,
  "self_bootstrapping": true
}

path: requirements.txt
content: # DeepRecall Dependencies
# Required for LLM summarization functionality
aiohttp>=3.8.0

# SQLite3 is included in Python standard library
# No additional dependencies needed for core memory retrieval functions

# For development and testing
# pytest>=7.0.0
# black>=22.0.0

path: scripts/memory_db_tool.py
content: #!/usr/bin/env python3
"""
memory_db_tool.py - Command-line interface for SQLite memory retrieval tools
Uses memory_retriever.py module-level functions for singleton access.
"""

import sys
import os
import json
import argparse
import asyncio
from pathlib import Path
from memory_retriever import MemoryRetriever, retrieve_l1_facts, retrieve_l2_raw, cleanup_raw_files, _get_retriever

def main():
    parser = argparse.ArgumentParser(description="SQLite memory database retrieval tool")
    subparsers = parser.add_subparsers(dest="command", help="Command to execute")
    
    # search command
    search_parser = subparsers.add_parser("search", help="Search L1 structured facts")
    search_parser.add_argument("query", help="Search query (empty for latest)")
    search_parser.add_argument("--limit", type=int, default=5, help="Number of results (default: 5)")
    
    # read command
    read_parser = subparsers.add_parser("read", help="Read L2 raw archive")
    read_parser.add_argument("source_file", help="Source filename (e.g., 'example-project-update.md')")
    
    # stats command
    stats_parser = subparsers.add_parser("stats", help="Show database statistics")
    
    # cleanup command
    cleanup_parser = subparsers.add_parser("cleanup", help="Cleanup raw .md files (preserves database)")
    cleanup_parser.add_argument("--retention-days", type=int, default=1, 
                               help="Keep files newer than N days (default: 1)")
    cleanup_parser.add_argument("--max-size-kb", type=int, default=250,
                               help="Maximum total size in KB (default: 250)")
    cleanup_parser.add_argument("--dry-run", action="store_true",
                               help="Show what would be deleted without actually deleting")
    cleanup_parser.add_argument("--memory-dir", type=str,
                               help="Custom memory directory path (default: auto-detect)")
    
    # summarize command
    summarize_parser = subparsers.add_parser("summarize", help="Summarize memory files using LLM")
    summarize_parser.add_argument("--process-all", action="store_true",
                                 help="Process all unprocessed memory files")
    summarize_parser.add_argument("--process-file", type=str,
                                 help="Process a specific memory file")
    summarize_parser.add_argument("--no-store-raw", action="store_true",
                                 help="Do not store raw content to L2 archive")
    summarize_parser.add_argument("--test-config", action="store_true",
                                 help="Test OpenClaw configuration and API connectivity")
    
    args = parser.parse_args()
    
    if args.command == "search":
        result = retrieve_l1_facts(args.query, args.limit)
        print(result)
    
    elif args.command == "read":
        result = retrieve_l2_raw(args.source_file)
        print(result)
    
    elif args.command == "stats":
        retriever = _get_retriever()
        stats = retriever.get_table_stats()
        print(json.dumps(stats, indent=2, ensure_ascii=False))
    
    elif args.command == "cleanup":
        print("=== Memory File Cleanup ===\n")
        print("IMPORTANT: This only cleans raw .md session files in workspace/memory/")
        print("           L1/L2 data in memory.db is PERMANENT and will NOT be affected.\n")
        
        if args.dry_run:
            print("DRY RUN MODE: No files will be deleted.\n")
        
        # Execute cleanup (supports dry-run)
        result = cleanup_raw_files(
            retention_days=args.retention_days,
            max_size_kb=args.max_size_kb,
            memory_dir=args.memory_dir,
            dry_run=args.dry_run
        )
        
        # Display results
        print("Cleanup Summary:")
        print(f"  Status: {result.get('status', 'unknown')}")
        
        if result.get('status') in ['skipped', 'dry_run', 'completed']:
            if result.get('reason'):
                print(f"  Reason: {result['reason']}")
            
            print(f"  Memory Directory: {result.get('memory_dir', 'unknown')}")
            print(f"  Config: {result.get('config', {}).get('retention_days', args.retention_days)} day(s), "
                  f"{result.get('config', {}).get('max_size_kb', args.max_size_kb)} KB max")
            
            if 'total_files_before' in result:
                print(f"  Total Files Before: {result['total_files_before']}")
                print(f"  Total Size Before: {result['total_size_before_kb']:.2f} KB")
                print(f"  Files {'to Delete' if args.dry_run else 'Deleted'}: {result['deleted_files_count']}")
                print(f"  Size {'to Free' if args.dry_run else 'Freed'}: {result['deleted_size_kb']:.2f} KB")
                print(f"  Remaining Files: {result['remaining_files_count']}")
                print(f"  Remaining Size: {result['remaining_size_kb']:.2f} KB")
                
                if result.get("deleted_files"):
                    print(f"\n  Files {'that would be deleted' if args.dry_run else 'deleted'}:")
                    for filename in result["deleted_files"][:10]:  # Limit display count
                        print(f"    - {filename}")
                    if len(result["deleted_files"]) > 10:
                        print(f"    ... and {len(result['deleted_files']) - 10} more")
        else:
            print(f"  Error: {result.get('reason', 'Unknown error')}")
    
    elif args.command == "summarize":
        # Import summarizer and run async function
        try:
            from memory_summarizer import DeepRecallSummarizer
            
            summarizer = DeepRecallSummarizer()
            
            if args.test_config:
                # Test configuration using the unified method
                summarizer.test_configuration()
            
            elif args.process_all:
                store_raw = not args.no_store_raw
                print(f"Processing all files (store_raw: {store_raw})...")
                stats = asyncio.run(summarizer.process_all_files(store_raw=store_raw))
                print("\nSummarization Complete:")
                print(f"  Files processed: {stats['files_processed']}")
                print(f"  Facts extracted: {stats['facts_extracted']}")
                print(f"  Facts stored: {stats['facts_stored']}")
                print(f"  Raw content stored: {stats['raw_content_stored']}")
            
            elif args.process_file:
                store_raw = not args.no_store_raw
                print(f"Processing file: {args.process_file} (store_raw: {store_raw})...")
                
                file_path = Path(args.process_file)
                if not file_path.is_absolute():
                    file_path = Path(summarizer.memory_dir) / file_path
                
                success = asyncio.run(summarizer.process_single_file(file_path, store_raw))
                if success:
                    print("File processed successfully")
                else:
                    print("File processing failed")
                    sys.exit(1)
            
            else:
                print("Please specify an action for summarize command:")
                print("  --process-all     Process all unprocessed files")
                print("  --process-file    Process a specific file")
                print("  --test-config     Test OpenClaw configuration")
        
        except ImportError as e:
            print(f"Error importing summarizer: {e}")
            print("Make sure memory_summarizer.py is in the same directory")
            sys.exit(1)
    
    else:
        parser.print_help()

if __name__ == "__main__":
    main()

path: scripts/memory_retriever.py
content: #!/usr/bin/env python3
"""
memory_retriever.py - Token-optimized memory retrieval engine for SQLite-based L1/L2 memory system.
Provides compact-format L1 fact search and exact L2 raw-content retrieval.
"""

import sqlite3
import os
import glob
from pathlib import Path
from typing import List, Optional
from datetime import datetime, timedelta

def get_agent_db_path() -> str:
    """
    Dynamically detect the current agent's database path.
    Priority order:
    1. OPENCLAW_WORKSPACE environment variable
    2. Current directory structure detection (looking for memory/ directory)
    3. Fallback to relative path
    """
    # 1. Check framework environment variable
    workspace = os.environ.get("OPENCLAW_WORKSPACE")
    if workspace:
        potential_path = os.path.join(workspace, "memory", "memory.db")
        if os.path.exists(os.path.dirname(potential_path)):
            return potential_path
    
    # 2. Check current directory structure
    cwd = os.getcwd()
    
    # Check for memory/ subdirectory in current directory
    if os.path.exists(os.path.join(cwd, "memory")):
        return os.path.join(cwd, "memory", "memory.db")
    
    # Check if parent directory is an agents structure
    parent = os.path.dirname(cwd)
    if "agents" in parent or "agents" in cwd:
        # Try to find memory in agents directory structure
        for path in [cwd, parent]:
            memory_dir = os.path.join(path, "memory")
            if os.path.exists(memory_dir):
                return os.path.join(memory_dir, "memory.db")
    
    # 3. Fallback to relative path
    return "./memory/memory.db"

def get_workspace_memory_dir() -> str:
    """
    Get workspace/memory directory path for storing raw .md session files.
    Uses the same priority order as database path detection.
    """
    # 1. Check framework environment variable
    workspace = os.environ.get("OPENCLAW_WORKSPACE")
    if workspace:
        memory_dir = os.path.join(workspace, "memory")
        if os.path.exists(memory_dir) or os.path.exists(os.path.dirname(memory_dir)):
            return memory_dir
    
    # 2. Check current directory structure
    cwd = os.getcwd()
    
    # Check for memory/ subdirectory in current directory
    if os.path.exists(os.path.join(cwd, "memory")):
        return os.path.join(cwd, "memory")
    
    # Check if parent directory is an agents structure
    parent = os.path.dirname(cwd)
    if "agents" in parent or "agents" in cwd:
        # Try to find memory in agents directory structure
        for path in [cwd, parent]:
            memory_dir = os.path.join(path, "memory")
            if os.path.exists(memory_dir):
                return memory_dir
    
    # 3. Fallback to relative path
    return "./memory"

class MemoryRetriever:
    def __init__(self, db_path: str = None):
        """
        Initialize the memory retriever.
        
        Parameters
        ----------
        db_path : str, optional
            Database path. If None, auto-detect.
        """
        if db_path is None:
            self.db_path = get_agent_db_path()
        else:
            self.db_path = db_path
        
        # Ensure database directory exists
        db_dir = os.path.dirname(self.db_path)
        if db_dir:  # Only create directory if path contains a directory component
            os.makedirs(db_dir, exist_ok=True)
        
        # Initialize database tables (self-bootstrapping)
        self._init_db()
    
    def _init_db(self):
        """
        Initialize database tables if they don't exist.
        This ensures zero-config deployment - tables are created automatically.
        """
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            
            # Create L1 structured facts table
            cursor.execute('''
                CREATE TABLE IF NOT EXISTS l1_structured (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    date TEXT NOT NULL,
                    source_file TEXT NOT NULL,
                    fact_type TEXT NOT NULL,
                    confidence REAL,
                    tags TEXT,
                    content_hash TEXT UNIQUE,
                    content TEXT NOT NULL,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                )
            ''')
            
            # Create L2 raw archive table
            cursor.execute('''
                CREATE TABLE IF NOT EXISTS l2_archive (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    date TEXT NOT NULL,
                    source_file TEXT UNIQUE NOT NULL,
                    raw_content TEXT NOT NULL,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                )
            ''')
            
            # Create processed_files table (replaces .processed marker files)
            cursor.execute('''
                CREATE TABLE IF NOT EXISTS processed_files (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    file_path TEXT UNIQUE NOT NULL,
                    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                )
            ''')
            
            # Create indexes for faster queries
            cursor.execute('CREATE INDEX IF NOT EXISTS idx_l1_date ON l1_structured(date)')
            cursor.execute('CREATE INDEX IF NOT EXISTS idx_l1_type ON l1_structured(fact_type)')
            cursor.execute('CREATE INDEX IF NOT EXISTS idx_l1_source ON l1_structured(source_file)')
            cursor.execute('CREATE INDEX IF NOT EXISTS idx_l2_source ON l2_archive(source_file)')
            cursor.execute('CREATE INDEX IF NOT EXISTS idx_processed_path ON processed_files(file_path)')
            
            # Create FTS5 virtual table for full-text search (if FTS5 is available)
            try:
                cursor.execute('''
                    CREATE VIRTUAL TABLE IF NOT EXISTS l1_fts 
                    USING fts5(
                        content, 
                        tags,
                        content=l1_structured,
                        content_rowid=id,
                        tokenize='unicode61'
                    )
                ''')
                # FTS5 virtual table created for semantic search
                
                # Create triggers to keep FTS index in sync with l1_structured table
                cursor.execute('''
                    CREATE TRIGGER IF NOT EXISTS l1_ai AFTER INSERT ON l1_structured BEGIN
                        INSERT INTO l1_fts(rowid, content, tags) 
                        VALUES (new.id, new.content, new.tags);
                    END
                ''')
                cursor.execute('''
                    CREATE TRIGGER IF NOT EXISTS l1_ad AFTER DELETE ON l1_structured BEGIN
                        INSERT INTO l1_fts(l1_fts, rowid, content, tags)
                        VALUES ('delete', old.id, old.content, old.tags);
                    END
                ''')
                cursor.execute('''
                    CREATE TRIGGER IF NOT EXISTS l1_au AFTER UPDATE ON l1_structured BEGIN
                        INSERT INTO l1_fts(l1_fts, rowid, content, tags) 
                        VALUES ('delete', old.id, old.content, old.tags);
                        INSERT INTO l1_fts(rowid, content, tags) 
                        VALUES (new.id, new.content, new.tags);
                    END
                ''')
                
                # Check if l1_structured has data but l1_fts is empty, rebuild index
                cursor.execute('SELECT COUNT(*) FROM l1_structured')
                l1_count = cursor.fetchone()[0]
                cursor.execute('SELECT COUNT(*) FROM l1_fts')
                fts_count = cursor.fetchone()[0]
                
                if l1_count > 0 and fts_count < l1_count:
                    print("FTS5 table is missing some records, rebuilding index...")
                    cursor.execute('INSERT INTO l1_fts(l1_fts) VALUES("rebuild")')
                    # FTS5 index rebuilt
                    
            except sqlite3.OperationalError as e:
                print(f"Note: FTS5 not available, using LIKE-based search: {e}")
            
            conn.commit()
    
    def search_l1_structured(self, query: str = None, limit: int = 5) -> List[str]:
        """
        Retrieve L1 structured facts in a compact, token-optimized format.
        
        Parameters
        ----------
        query : str, optional
            Search string. If None or empty, returns the latest facts.
        limit : int, default 5
            Maximum number of results.
            
        Returns
        -------
        List[str]
            Formatted facts: '[YYYY-MM-DD | fact_type | source:source_file] content...'
            No content truncation is applied.
        """
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            
            if query:
                # Try FTS5 full-text search first, fall back to LIKE if needed
                try:
                    # Check if FTS5 table exists
                    cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='l1_fts'")
                    fts_table_exists = cursor.fetchone() is not None
                    
                    if fts_table_exists:
                        # Convert query to FTS5 MATCH syntax
                        # Support "Vue error OR sandbox failure" syntax
                        or_groups = [kw.strip() for kw in query.split(" OR ") if kw.strip()]
                        fts_conditions = []
                        
                        for group in or_groups:
                            # Escape double quotes in query for FTS5
                            escaped_group = group.replace('"', '""')
                            # For multi-word groups, wrap in quotes for phrase search
                            if ' ' in escaped_group:
                                fts_conditions.append(f'"{escaped_group}"')
                            else:
                                fts_conditions.append(escaped_group)
                        
                        if fts_conditions:
                            fts_query = " OR ".join(fts_conditions)
                            
                            sql = '''
                                SELECT s.date, s.fact_type, s.source_file, s.content
                                FROM l1_fts f
                                JOIN l1_structured s ON f.rowid = s.id
                                WHERE l1_fts MATCH ?
                                ORDER BY rank, s.confidence DESC, s.date DESC
                                LIMIT ?
                            '''
                            cursor.execute(sql, (fts_query, limit))
                            results = cursor.fetchall()
                            # FTS5 query succeeded, proceed to formatting regardless of empty results
                        else:
                            raise sqlite3.OperationalError("No valid FTS5 query")
                    else:
                        raise sqlite3.OperationalError("FTS5 table not found")
                        
                except sqlite3.OperationalError:
                    # FTS5 failed, use LIKE-based search
                    # Original LIKE-based search logic
                    keywords = [kw.strip() for kw in query.split(" OR ") if kw.strip()]
                    conditions = []
                    params = []
                    
                    for kw in keywords:
                        # Split each OR group into individual words (AND semantics within group)
                        sub_keywords = kw.split()
                        group_conditions = []
                        for sub_kw in sub_keywords:
                            if sub_kw:  # Skip empty strings
                                group_conditions.append("(content LIKE ? OR tags LIKE ?)")
                                params.extend([f'%{sub_kw}%', f'%{sub_kw}%'])
                        
                        if group_conditions:
                            # Join AND conditions within group
                            conditions.append("(" + " AND ".join(group_conditions) + ")")
                    
                    if not conditions:
                        # Fallback to simple search if no valid keywords
                        conditions.append("(content LIKE ? OR tags LIKE ?)")
                        params.extend([f'%{query}%', f'%{query}%'])
                    
                    where_clause = " OR ".join(conditions)
                    sql = f'''
                        SELECT date, fact_type, source_file, content 
                        FROM l1_structured 
                        WHERE {where_clause}
                        ORDER BY confidence DESC, date DESC
                        LIMIT ?
                    '''
                    params.append(limit)
                    cursor.execute(sql, params)
                    results = cursor.fetchall()
            else:
                # Retrieve latest records
                sql = '''
                    SELECT date, fact_type, source_file, content
                    FROM l1_structured
                    ORDER BY date DESC, confidence DESC
                    LIMIT ?
                '''
                cursor.execute(sql, (limit,))
                results = cursor.fetchall()
        
        # Format output - extreme compression (no content truncation)
        formatted = []
        for date, fact_type, source_file, content in results:
            # Clean fact_type: only strip prefix for project_ types, preserve other types like user_profile
            if fact_type.startswith("project_"):
                clean_type = "project"
            else:
                clean_type = fact_type
            
            # Compact format: [date | type | source:file] content
            # Note: no truncation is applied; content is returned in full
            line = f"[{date} | {clean_type} | source:{source_file}] {content}"
            formatted.append(line)
        
        return formatted
    
    def get_l2_raw(self, source_file: str) -> Optional[str]:
        """
        Retrieve exact raw content from the L2 archive.
        
        Parameters
        ----------
        source_file : str
            Exact filename as stored in the source_file column.
            
        Returns
        -------
        Optional[str]
            Complete raw Markdown content, or None if the file is not found.
        """
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            
            cursor.execute('''
                SELECT raw_content FROM l2_archive 
                WHERE source_file = ? 
                LIMIT 1
            ''', (source_file,))
            
            result = cursor.fetchone()
        
        return result[0] if result else None
    
    def get_table_stats(self) -> dict:
        """Return basic statistics about the memory database."""
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            
            cursor.execute('SELECT COUNT(*) FROM l1_structured')
            l1_count = cursor.fetchone()[0]
            
            cursor.execute('SELECT COUNT(*) FROM l2_archive')
            l2_count = cursor.fetchone()[0]
            
            cursor.execute('SELECT DISTINCT fact_type FROM l1_structured')
            fact_types = [row[0] for row in cursor.fetchall()]
        
        return {
            'l1_structured_count': l1_count,
            'l2_archive_count': l2_count,
            'fact_types': fact_types
        }
    
    def mark_file_as_processed(self, file_path: str) -> bool:
        """
        Mark a file as processed in the database.
        
        Parameters
        ----------
        file_path : str
            Path to the file (relative or absolute)
            
        Returns
        -------
        bool
            True if successful, False otherwise
        """
        try:
            with sqlite3.connect(self.db_path) as conn:
                cursor = conn.cursor()
                cursor.execute(
                    '''
                    INSERT OR REPLACE INTO processed_files 
                    (file_path, processed_at) 
                    VALUES (?, CURRENT_TIMESTAMP)
                    ''',
                    (str(file_path),)
                )
                conn.commit()
            return True
        except Exception as e:
            print(f"Error marking file as processed in database: {e}")
            return False
    
    def is_file_processed(self, file_path: str) -> bool:
        """
        Check if a file has been processed.
        
        Parameters
        ----------
        file_path : str
            Path to the file (relative or absolute)
            
        Returns
        -------
        bool
            True if file is marked as processed
        """
        try:
            with sqlite3.connect(self.db_path) as conn:
                cursor = conn.cursor()
                cursor.execute(
                    'SELECT id FROM processed_files WHERE file_path = ?',
                    (str(file_path),)
                )
                result = cursor.fetchone()
            return result is not None
        except Exception as e:
            print(f"Error checking if file is processed: {e}")
            return False
    
    def get_unprocessed_files(self, memory_dir: str) -> List[str]:
        """
        Get list of unprocessed .md files in the memory directory.
        Uses batch query to avoid N+1 database queries.
        
        Parameters
        ----------
        memory_dir : str
            Path to memory directory
            
        Returns
        -------
        List[str]
            List of file paths (relative to memory_dir) that are not processed
        """
        try:
            memory_dir_path = Path(memory_dir)
            # Get all .md files in the directory
            all_md_files = [str(f) for f in memory_dir_path.rglob("*.md")]
            
            if not all_md_files:
                return []
            
            # Batch query to get all processed files in one query
            with sqlite3.connect(self.db_path) as conn:
                cursor = conn.cursor()
                
                # Create placeholders for SQL IN clause
                placeholders = ",".join(["?" for _ in all_md_files])
                
                cursor.execute(
                    f"SELECT file_path FROM processed_files WHERE file_path IN ({placeholders})",
                    all_md_files
                )
                
                processed_set = {row[0] for row in cursor.fetchall()}
            
            # Return files that are not in the processed set
            unprocessed = [f for f in all_md_files if f not in processed_set]
            return unprocessed
            
        except Exception as e:
            print(f"Error getting unprocessed files: {e}")
            # Fallback to individual queries
            unprocessed = []
            try:
                memory_dir_path = Path(memory_dir)
                for file_path in memory_dir_path.rglob("*.md"):
                    if not self.is_file_processed(str(file_path)):
                        unprocessed.append(str(file_path))
            except Exception as e2:
                print(f"Fallback query also failed: {e2}")
            
            return unprocessed

def cleanup_raw_files(retention_days: int = 1, max_size_kb: int = 250, memory_dir: str = None, dry_run: bool = False) -> dict:
    """
    Clean up raw .md session files, keeping recent data and controlling total size.
    
    Important: This function only cleans raw .md files in workspace/memory/ directory,
    and does NOT affect the permanent L1/L2 data in memory.db database.
    
    Parameters
    ----------
    retention_days : int, default 1
        Keep files newer than N days (based on modification time)
    max_size_kb : int, default 250
        Maximum total file size (KB), delete oldest files if exceeds
    memory_dir : str, optional
        Memory directory path, if None auto-detected
    dry_run : bool, default False
        If True, only calculate files to delete without actually deleting
        
    Returns
    -------
    dict
        Cleanup statistics
    """
    if memory_dir is None:
        memory_dir = get_workspace_memory_dir()
    
    # Ensure directory exists
    if not os.path.exists(memory_dir):
        return {"status": "skipped", "reason": f"Directory not found: {memory_dir}"}
    
    # Get all .md files (excluding memory.db)
    md_files = glob.glob(os.path.join(memory_dir, "*.md"))
    
    if not md_files:
        return {"status": "skipped", "reason": "No .md files found"}
    
    # Sort by modification time (oldest to newest)
    files_with_info = []
    for file_path in md_files:
        try:
            stat = os.stat(file_path)
            files_with_info.append({
                "path": file_path,
                "size_kb": stat.st_size / 1024,
                "mtime": stat.st_mtime,
                "date": datetime.fromtimestamp(stat.st_mtime),
                "filename": os.path.basename(file_path)
            })
        except Exception as e:
            continue
    
    # Sort by modification time (oldest to newest)
    files_with_info.sort(key=lambda x: x["mtime"])
    
    total_size_kb = sum(f["size_kb"] for f in files_with_info)
    total_files = len(files_with_info)
    
    # Calculate cutoff time (keep files from last retention_days)
    cutoff_time = datetime.now() - timedelta(days=retention_days)
    cutoff_timestamp = cutoff_time.timestamp()
    
    # First pass cleanup: time-based
    to_delete_time = []
    kept_files = []
    
    for file_info in files_with_info:
        if file_info["mtime"] < cutoff_timestamp:
            to_delete_time.append(file_info)
        else:
            kept_files.append(file_info)
    
    # Second pass cleanup: size-based (if needed)
    remaining_size_kb = sum(f["size_kb"] for f in kept_files)
    to_delete_size = []
    
    if remaining_size_kb > max_size_kb:
        # Delete oldest files until size limit is met
        kept_files.sort(key=lambda x: x["mtime"])  # Ensure chronological order
        
        while kept_files and remaining_size_kb > max_size_kb:
            oldest_file = kept_files.pop(0)
            to_delete_size.append(oldest_file)
            remaining_size_kb -= oldest_file["size_kb"]
    
    # Combine files to delete
    files_to_delete = to_delete_time + to_delete_size
    
    # Execute deletion (unless dry-run)
    deleted_files = []
    deleted_size_kb = 0
    
    if not dry_run:
        for file_info in files_to_delete:
            try:
                os.remove(file_info["path"])
                deleted_files.append(file_info["path"])
                deleted_size_kb += file_info["size_kb"]
            except Exception as e:
                continue
    else:
        # Dry-run mode: only record files that would be deleted
        deleted_files = [f["path"] for f in files_to_delete]
        deleted_size_kb = sum(f["size_kb"] for f in files_to_delete)
    
    # Clean up orphaned .processed marker files (if any exist from previous version)
    orphaned_markers_deleted = 0
    for marker_path in Path(memory_dir).glob("*.md.processed"):
        # Check if corresponding .md file exists
        source_md = marker_path.with_suffix("")  # Remove .processed suffix
        if not source_md.exists():
            if not dry_run:
                try:
                    marker_path.unlink()
                    orphaned_markers_deleted += 1
                except Exception as e:
                    continue
            else:
                orphaned_markers_deleted += 1
    
    # Return cleanup statistics
    result = {
        "status": "completed" if not dry_run else "dry_run",
        "memory_dir": memory_dir,
        "total_files_before": total_files,
        "total_size_before_kb": round(total_size_kb, 2),
        "deleted_files_count": len(files_to_delete),
        "deleted_size_kb": round(deleted_size_kb, 2),
        "remaining_files_count": len(kept_files),
        "remaining_size_kb": round(remaining_size_kb, 2),
        "deleted_files": [f["filename"] for f in files_to_delete][:10],  # Limit output length
        "orphaned_markers_deleted": orphaned_markers_deleted,
        "config": {
            "retention_days": retention_days,
            "max_size_kb": max_size_kb,
            "dry_run": dry_run
        }
    }
    
    if orphaned_markers_deleted > 0:
        result["reason"] = f"Cleaned {orphaned_markers_deleted} orphaned .processed marker files"
    
    return result

# Singleton retriever instance for OpenClaw tool calls
_retriever_instance = None

def _get_retriever():
    """Get or create singleton MemoryRetriever instance."""
    global _retriever_instance
    if _retriever_instance is None:
        _retriever_instance = MemoryRetriever()
    return _retriever_instance

# Tool functions - for OpenClaw Tool calls
def retrieve_l1_facts(query: str = "", limit: int = 5) -> str:
    """
    Primary entry point for L1 fact retrieval.
    
    Parameters
    ----------
    query : str, default ""
        Search string. Empty string returns the latest facts.
    limit : int, default 5
        Maximum number of results.
        
    Returns
    -------
    str
        Compact formatted facts separated by newlines.
        Format: '[YYYY-MM-DD | fact_type | source:source_file] content...'
        
    Example
    -------
    >>> retrieve_l1_facts("example-project", 2)
    '[2024-01-01 | project | source:example-project.md] example-project backend upgrade...'
    '[2024-01-01 | technical | source:example-project.md] Database model upgrade: Task model adds project_id field...'
    """
    retriever = _get_retriever()
    facts = retriever.search_l1_structured(query, limit)
    
    if not facts:
        return "No matching facts found."
    
    # Convert list to compact text block
    return "\n".join(facts)

def retrieve_l2_raw(source_file: str) -> str:
    """
    Primary entry point for L2 raw-content retrieval.
    
    Parameters
    ----------
    source_file : str
        Exact filename as obtained from an L1 search result.
        
    Returns
    -------
    str
        Complete raw Markdown content of the specified file.
        If the file is not found, returns an error message.
    """
    retriever = _get_retriever()
    content = retriever.get_l2_raw(source_file)
    
    if content is None:
        return f"Source file not found: {source_file}"
    
    return content

# Example usage and testing
if __name__ == "__main__":
    retriever = MemoryRetriever()
    
    print("=== Database Statistics ===")
    stats = retriever.get_table_stats()
    print(f"L1 facts count: {stats['l1_structured_count']}")
    print(f"L2 files count: {stats['l2_archive_count']}")
    print(f"Fact types: {', '.join(stats['fact_types'][:5])}...")
    
    print("\n=== Auto-Detected Paths ===")
    print(f"Database path: {retriever.db_path}")
    print(f"Memory directory: {get_workspace_memory_dir()}")
    
    print("\n=== L1 Retrieval Test (query 'example-project') ===")
    results = retrieve_l1_facts("example-project", 3)
    print(results)
    
    print("\n=== L1 Retrieval Test (latest records) ===")
    results = retrieve_l1_facts(limit=2)
    print(results)
    
    print("\n=== L2 Retrieval Test ===")
    raw = retrieve_l2_raw("example-project-update.md")
    if raw:
        print(f"Raw file size: {len(raw)} characters")
        print("First 200 characters preview:", raw[:200] + "...")
    else:
        print("File not found")
    
    print("\n=== File Cleanup Test (Dry Run) ===")
    cleanup_result = cleanup_raw_files(retention_days=1, max_size_kb=250, dry_run=True)
    print(f"Cleanup status: {cleanup_result.get('status', 'unknown')}")
    print(f"Files to delete: {cleanup_result.get('deleted_files_count', 0)}")
    print(f"Files to keep: {cleanup_result.get('remaining_files_count', 0)}")

path: scripts/memory_summarizer.py
content: #!/usr/bin/env python3
"""
DeepRecall Memory Summarizer - LLM-powered fact extraction for SQLite L1/L2 memory system.
Integrates with DeepRecall database to store structured facts and raw content.
"""

import os
import json
import re
import hashlib
import asyncio
import traceback
import argparse
import aiohttp
import sqlite3
from pathlib import Path
from datetime import datetime
from typing import Dict, List, Optional, Tuple, Any
import sys

# Import DeepRecall's path detection utilities
from memory_retriever import get_agent_db_path, get_workspace_memory_dir

class DeepRecallSummarizer:
    def __init__(self, db_path: str = None, memory_dir: str = None):
        """
        Initialize the DeepRecall summarizer.
        
        Parameters
        ----------
        db_path : str, optional
            Database path. If None, auto-detect using DeepRecall's logic.
        memory_dir : str, optional
            Memory directory path. If None, auto-detect using workspace.
        """
        if db_path is None:
            self.db_path = get_agent_db_path()
        else:
            self.db_path = db_path
        
        # Ensure database directory exists
        db_dir = os.path.dirname(self.db_path)
        if db_dir:  # Only create directory if path contains a directory component
            os.makedirs(db_dir, exist_ok=True)
        
        # Get workspace memory directory for raw files
        if memory_dir:
            self.memory_dir = memory_dir
        else:
            self.memory_dir = get_workspace_memory_dir()
        
        # Create MemoryRetriever instance for database operations
        from memory_retriever import MemoryRetriever
        self.retriever = MemoryRetriever(db_path=self.db_path)
        
        # Load DeepRecall configuration
        self.config = self._get_deeprecall_config()
        
        # Processing statistics
        self.stats = {
            "files_processed": 0,
            "facts_extracted": 0,
            "facts_stored": 0,
            "raw_content_stored": 0
        }
        
        # LLM extraction prompt template (English version)
        self.extraction_prompt_template = """
You are a professional memory analysis assistant. Your task is to extract key facts from raw memory files and classify them into structured data.

## Input
Raw memory file content provided by the user.

## Output Requirements
Output a JSON array, each element is a fact object in the following format:
```json
[
  {
    "type": "fact_type",
    "content": "Fact content (concise and clear)",
    "confidence": 0.95,
    "tags": ["tag1", "tag2"],
    "project_name": "Optional project name (only when type is project)"
  }
]
```

## Fact Type (type) Definitions
1. **user_profile**: User personal information, preferences, habits, background
2. **identity**: System/assistant identity, role, mission statement
3. **project_[project_name]**: Facts related to specific projects (use underscore, e.g., project_example-project)
4. **technical**: Technical details, code snippets, architecture decisions, configurations
5. **preferences**: System preferences, settings, UI/UX choices, defaults
6. **learnings**: Learning experiences, error analysis, improvement suggestions, lessons
7. **milestones**: Development milestones, important completions, version releases

## Extraction Rules
1. **Conciseness**: Keep each fact content within 1-3 sentences
2. **Specificity**: Include specific names, dates, numbers, technical terms
3. **Deduplication**: Extract the same fact only once
4. **Classification Accuracy**: Choose the most appropriate type based on content
5. **Project Identification**: When content involves a specific project, note the project name in the project_name field

## Example
Input:
"2024-01-01: Completed Japanese textbook summary script today, processed 20 PDF lessons."

Output:
```json
[
  {
    "type": "project_japanese-textbook-summary",
    "content": "2024-01-01: Completed Japanese textbook summary script, processed 20 PDF lessons, generated Word documents.",
    "confidence": 0.9,
    "tags": ["Japanese", "PDF processing", "automation"],
    "project_name": "japanese-textbook-summary"
  }
]
```

Now analyze the following memory content:
"""
    
    def _get_openclaw_config(self) -> Dict:
        """
        Read OpenClaw configuration from openclaw.json.
        Returns model provider configuration.
        """
        try:
            # Try to find openclaw.json in common locations
            possible_paths = [
                Path.home() / ".openclaw" / "openclaw.json",
                Path("/etc/openclaw/openclaw.json"),  # System-wide configuration
                Path.cwd().parent / "openclaw.json",
                Path.cwd() / "openclaw.json"
            ]
            
            config_path = None
            for path in possible_paths:
                if path.exists():
                    config_path = path
                    break
            
            if not config_path:
                print("Warning: OpenClaw configuration not found, using default API endpoint")
                return {}
            
            with open(config_path, 'r', encoding='utf-8') as f:
                config = json.load(f)
            
            return config.get("models", {}).get("providers", {})
            
        except Exception as e:
            print(f"Error reading OpenClaw config: {e}")
            return {}
    
    def _get_deeprecall_config(self) -> Dict:
        """
        Read DeepRecall configuration from config files.
        Returns DeepRecall-specific configuration.
        """
        try:
            # Try to find DeepRecall configuration in common locations
            possible_paths = [
                Path.cwd() / "config.json",
                Path.cwd() / "deeprecall_config.json",
                Path.cwd().parent / "config.json",
                Path(self.memory_dir).parent / "config.json",
                Path.home() / ".deeprecall.json"
            ]
            
            config_path = None
            for path in possible_paths:
                if path.exists():
                    config_path = path
                    print(f"Found DeepRecall config at: {path}")
                    break
            
            if not config_path:
                # Return default configuration
                return {
                    "summarizer": {
                        "preferred_provider": None,  # Auto-select first available
                        "preferred_model": None,     # Auto-select first model
                        "max_content_length": 6000,
                        "temperature": 0.1,
                        "max_tokens": 4000,
                        "timeout_seconds": 180,
                        "store_raw_content": True
                    }
                }
            
            with open(config_path, 'r', encoding='utf-8') as f:
                config = json.load(f)
            
            # Return deeprecall section or full config
            return config.get("deeprecall", config)
            
        except json.JSONDecodeError as e:
            print(f"Warning: Invalid JSON in DeepRecall config: {e}, using defaults")
            return {"summarizer": {}}
        except Exception as e:
            print(f"Warning: Error reading DeepRecall config: {e}, using defaults")
            return {"summarizer": {}}
    
    async def extract_facts_with_llm(self, content: str) -> List[Dict]:
        """
        Extract structured facts from content using LLM API.
        
        Parameters
        ----------
        content : str
            Raw content to analyze
            
        Returns
        -------
        List[Dict]
            List of fact objects
        """
        # Get configuration parameters early for use in exception handlers
        summarizer_config = self.config.get("summarizer", {})
        timeout_seconds = summarizer_config.get("timeout_seconds", 180)
        
        try:
            # Get API configuration from OpenClaw config
            providers = self._get_openclaw_config()
            
            # Get preferred provider from DeepRecall configuration
            preferred_provider = summarizer_config.get("preferred_provider")
            preferred_model = summarizer_config.get("preferred_model")
            
            # Select provider based on configuration
            api_config = None
            provider_key = None
            
            if preferred_provider and preferred_provider in providers:
                # Use configured preferred provider
                api_config = providers[preferred_provider]
                provider_key = preferred_provider
                print(f"Using configured preferred provider: {preferred_provider}")
            else:
                # Auto-select first available provider with baseUrl and apiKey
                for name, provider_cfg in providers.items():
                    if "baseUrl" in provider_cfg and provider_cfg.get("apiKey"):
                        api_config = provider_cfg
                        provider_key = name
                        print(f"Auto-selected provider: {name}")
                        break
            
            if not api_config:
                print("Warning: No LLM API configuration found, using rule-based extraction")
                return await self._extract_facts_with_rules(content)
            
            base_url = api_config.get("baseUrl", "https://api.deepseek.com/v1")
            # Remove trailing slash to avoid double slashes in URL concatenation
            if base_url.endswith('/'):
                base_url = base_url.rstrip('/')
            [REDACTED_SECRET]"apiKey", "")
            
            if not [REDACTED_SECRET]"Warning: API Key not configured, using rule-based extraction")
                return await self._extract_facts_with_rules(content)
            
            # Get model ID from provider config
            models = api_config.get("models", [])
            model_id = None
            
            if preferred_model:
                # Try to find the preferred model in provider's model list
                for model_item in models:
                    # Handle both dict format {"id": "model-name"} and string format "model-name"
                    model_item_id = model_item.get("id", model_item) if isinstance(model_item, dict) else model_item
                    if model_item_id == preferred_model:
                        model_id = preferred_model
                        break
            
            # If no preferred model or not found, use first available model
            if not model_id and models and len(models) > 0:
                first_model = models[0]
                model_id = first_model.get("id", first_model) if isinstance(first_model, dict) else first_model
            
            if not model_id:
                print("Warning: No model configured in provider, using rule-based extraction")
                return await self._extract_facts_with_rules(content)
            
            # Get configuration parameters
            max_content_length = summarizer_config.get("max_content_length", 6000)
            temperature = summarizer_config.get("temperature", 0.1)
            max_tokens = summarizer_config.get("max_tokens", 4000)
            
            # Build complete prompt with length limit
            full_prompt = self.extraction_prompt_template + "\n" + content[:max_content_length]
            
            # Prepare API request
            headers = {
                "Content-Type": "application/json",
                "Authorization": f"Bearer {api_key}"
            }
            
            payload = {
                "model": model_id,
                "messages": [
                    {"role": "user", "content": full_prompt}
                ],
                "temperature": temperature,
                "max_tokens": max_tokens
                # Note: response_format is intentionally omitted for broader compatibility
                # Some OpenAI-compatible APIs don't support this parameter
                # The prompt already explicitly requests JSON output
            }
            
            print(f"Calling LLM API to extract facts, content length: {len(content)} characters")
            print(f"API endpoint: {base_url}/chat/completions")
            print(f"Using provider: {provider_key}, model: {model_id}")
            print(f"Configuration: temp={temperature}, max_tokens={max_tokens}, timeout={timeout_seconds}s")
            
            # Send async request with configured timeout
            timeout = aiohttp.ClientTimeout(total=timeout_seconds)
            
            async with aiohttp.ClientSession() as session:
                async with session.post(
                    f"{base_url}/chat/completions",
                    json=payload,
                    headers=headers,
                    timeout=timeout
                ) as response:
                    
                    if response.status == 200:
                        result = await response.json()
                        response_text = result.get("choices", [{}])[0].get("message", {}).get("content", "")
                        
                        # Parse JSON response
                        try:
                            # Extract JSON part (may contain markdown code blocks)
                            json_match = re.search(r'```json\s*(.*?)\s*```', response_text, re.DOTALL)
                            if json_match:
                                json_str = json_match.group(1)
                            else:
                                # Try to parse entire response
                                json_str = response_text
                            
                            facts = json.loads(json_str)
                            print(f"LLM extraction successful, obtained {len(facts)} facts")
                            return facts
                            
                        except json.JSONDecodeError as e:
                            print(f"LLM response JSON parsing failed: {e}")
                            print(f"Raw response (first 500 chars): {response_text[:500]}")
                            # Fallback to rule extraction
                            return await self._extract_facts_with_rules(content)
                            
                    else:
                        error_text = await response.text()
                        print(f"LLM API error: {response.status} - {error_text[:500]}")
                        # Fallback to rule extraction
                        return await self._extract_facts_with_rules(content)
                        
        except asyncio.TimeoutError:
            print(f"LLM API call timeout ({timeout_seconds} seconds), using rule-based extraction")
            return await self._extract_facts_with_rules(content)
        except Exception as e:
            print(f"LLM API call exception: {type(e).__name__}: {e}")
            traceback.print_exc()
            # Fallback to rule extraction
            return await self._extract_facts_with_rules(content)
    
    async def _extract_facts_with_rules(self, content: str) -> List[Dict]:
        """
        Rule-based extraction fallback method.
        Used when LLM API fails.
        
        Note: Each line is classified into at most one category using priority order:
        1. Project-related facts (highest priority)
        2. Technical facts  
        3. Learning facts (lowest priority)
        """
        MAX_FACT_LENGTH = 300
        facts = []
        lines = content.split('\n')
        
        # Keyword sets for classification (in priority order)
        project_keywords = ["project", "example", "development"]
        tech_keywords = ["api", "database", "model", "config", "code", "implementation", "python", "sql"]
        learning_keywords = ["learn", "error", "experience", "lesson", "improve", "mistake", "failure"]
        
        lines_processed = False
        
        for line in lines:
            line = line.strip()
            
            # Skip empty lines and very short lines
            if not line or len(line) < 15:
                continue
            
            # Skip pure Markdown formatting lines
            if line.startswith('#') and len(line.lstrip('#').strip()) < 10:
                continue
            if line in ('---', '***', '___', '```', '```json', '```python'):
                continue
            
            lines_processed = True
            line_lower = line.lower()
            
            # Truncate line if too long (prevent Base64-like content from bloating database)
            if len(line) > MAX_FACT_LENGTH:
                line = line[:MAX_FACT_LENGTH] + "..."
            
            # Check categories in priority order (first match wins)
            if any(keyword in line_lower for keyword in project_keywords):
                # Try to extract project name
                project_match = re.search(r'[Pp]roject[:\s]*([\w\s\-]+)', line)
                project_name = "unknown-project"
                if project_match:
                    project_name = project_match.group(1).strip().lower().replace(' ', '-')
                
                facts.append({
                    "type": f"project_{project_name}",
                    "content": line,
                    "confidence": 0.8,
                    "tags": ["project"],
                    "project_name": project_name
                })
            
            elif any(keyword in line_lower for keyword in tech_keywords):
                facts.append({
                    "type": "technical",
                    "content": line,
                    "confidence": 0.85,
                    "tags": ["technical"]
                })
            
            elif any(keyword in line_lower for keyword in learning_keywords):
                facts.append({
                    "type": "learnings",
                    "content": line,
                    "confidence": 0.9,
                    "tags": ["learning"]
                })
        
        # If no facts extracted but lines were processed, add a general fact
        if not facts and lines_processed:
            # Find the first non-filtered line to use as content
            meaningful_content = ""
            for line in lines:
                line = line.strip()
                if not line or len(line) < 15:
                    continue
                if line.startswith('#') and len(line.lstrip('#').strip()) < 10:
                    continue
                if line in ('---', '***', '___', '```', '```json', '```python'):
                    continue
                meaningful_content = line[:MAX_FACT_LENGTH] + ("..." if len(line) > MAX_FACT_LENGTH else "")
                break
            
            if meaningful_content:
                facts.append({
                    "type": "learnings",
                    "content": meaningful_content,
                    "confidence": 0.7,
                    "tags": ["general"]
                })
        
        print(f"Rule-based extraction complete, obtained {len(facts)} facts")
        return facts
    
    def _get_db_connection(self) -> sqlite3.Connection:
        """Get SQLite database connection."""
        conn = sqlite3.connect(self.db_path)
        conn.row_factory = sqlite3.Row
        return conn
    
    def store_fact_to_db(self, fact: Dict, source_file: str, date: str = None) -> bool:
        """
        Store a fact to the L1 structured table in DeepRecall database.
        
        Parameters
        ----------
        fact : Dict
            Fact object with type, content, confidence, tags
        source_file : str
            Source filename (without path)
        date : str, optional
            Date in YYYY-MM-DD format, defaults to today
            
        Returns
        -------
        bool
            True if successful, False otherwise
        """
        if date is None:
            date = datetime.now().strftime("%Y-%m-%d")
        
        fact_type = fact.get("type", "unknown")
        content = fact.get("content", "")
        confidence = float(fact.get("confidence", 0.0))
        tags_list = fact.get("tags", [])
        tags = ",".join(tags_list) if tags_list else ""
        
        # Generate content hash for deduplication (SHA-256 for collision resistance)
        content_hash = hashlib.sha256(content.encode()).hexdigest()
        
        conn = None
        try:
            conn = self._get_db_connection()
            cursor = conn.cursor()
            
            # Check if fact already exists (by content hash)
            cursor.execute(
                "SELECT id FROM l1_structured WHERE content_hash = ?",
                (content_hash,)
            )
            existing = cursor.fetchone()
            
            if existing:
                print(f"  Fact already exists (hash: {content_hash[:8]}), skipping")
                return False
            
            # Insert new fact
            cursor.execute(
                """
                INSERT INTO l1_structured 
                (date, source_file, fact_type, confidence, tags, content_hash, content)
                VALUES (?, ?, ?, ?, ?, ?, ?)
                """,
                (date, source_file, fact_type, confidence, tags, content_hash, content)
            )
            
            conn.commit()
            self.stats["facts_stored"] += 1
            return True
            
        except Exception as e:
            print(f"Error storing fact to database: {e}")
            return False
        finally:
            if conn:
                conn.close()
    
    def store_raw_content_to_db(self, source_file: str, content: str, date: str = None) -> bool:
        """
        Store raw content to the L2 archive table in DeepRecall database.
        
        Parameters
        ----------
        source_file : str
            Source filename (without path)
        content : str
            Raw content to store
        date : str, optional
            Date in YYYY-MM-DD format, defaults to today
            
        Returns
        -------
        bool
            True if successful, False otherwise
        """
        if date is None:
            date = datetime.now().strftime("%Y-%m-%d")
        
        conn = None
        try:
            conn = self._get_db_connection()
            cursor = conn.cursor()
            
            # Check if raw content already exists (by source_file)
            cursor.execute(
                "SELECT id FROM l2_archive WHERE source_file = ?",
                (source_file,)
            )
            existing = cursor.fetchone()
            
            if existing:
                print(f"  Raw content already exists for {source_file}, updating")
                cursor.execute(
                    """
                    UPDATE l2_archive 
                    SET raw_content = ?, date = ?
                    WHERE source_file = ?
                    """,
                    (content, date, source_file)
                )
            else:
                # Insert new raw content
                cursor.execute(
                    """
                    INSERT INTO l2_archive 
                    (date, source_file, raw_content)
                    VALUES (?, ?, ?)
                    """,
                    (date, source_file, content)
                )
            
            conn.commit()
            self.stats["raw_content_stored"] += 1
            return True
            
        except Exception as e:
            print(f"Error storing raw content to database: {e}")
            return False
        finally:
            if conn:
                conn.close()
    
    def mark_file_as_processed(self, file_path: Path) -> bool:
        """
        Mark a file as processed in the database.
        
        Parameters
        ----------
        file_path : Path
            Path to the original file
            
        Returns
        -------
        bool
            True if successful, False otherwise
        """
        try:
            # Use MemoryRetriever to mark file as processed in database
            success = self.retriever.mark_file_as_processed(str(file_path))
            if success:
                print(f"  File marked as processed in database: {file_path.name}")
            return success
        except Exception as e:
            print(f"Error marking file as processed {file_path}: {e}")
            return False
    
    async def process_single_file(self, file_path: Path, store_raw: bool = True) -> bool:
        """
        Process a single memory file.
        
        Parameters
        ----------
        file_path : Path
            Path to the memory file
        store_raw : bool
            Whether to store raw content to L2 archive
            
        Returns
        -------
        bool
            True if successful, False otherwise
        """
        try:
            if not file_path.exists():
                print(f"  File does not exist: {file_path}")
                return False
            
            file_name = file_path.name
            print(f"Processing file: {file_name}")
            
            # 1. Read content
            content = file_path.read_text(encoding='utf-8', errors='ignore')
            if not content.strip():
                print(f"  File is empty, skipping")
                return False
            
            # 2. Extract date from filename if possible (YYYY-MM-DD pattern)
            date_match = re.search(r'(\d{4}-\d{2}-\d{2})', file_name)
            file_date = date_match.group(1) if date_match else None
            
            # 3. Store raw content to L2 archive (if requested)
            if store_raw:
                self.store_raw_content_to_db(file_name, content, file_date)
            
            # 4. Use LLM to extract facts
            facts = await self.extract_facts_with_llm(content)
            print(f"  Extracted {len(facts)} facts")
            
            # 5. Store facts to L1 structured table
            for fact in facts:
                self.store_fact_to_db(fact, file_name, file_date)
            
            # 6. Mark file as processed
            self.mark_file_as_processed(file_path)
            
            self.stats["files_processed"] += 1
            self.stats["facts_extracted"] += len(facts)
            
            return True
            
        except Exception as e:
            print(f"Error processing file {file_path}: {e}")
            return False
    
    async def process_all_files(self, store_raw: bool = True) -> Dict:
        """
        Process all unprocessed memory files in the memory directory.
        
        Parameters
        ----------
        store_raw : bool
            Whether to store raw content to L2 archive
            
        Returns
        -------
        Dict
            Processing statistics
        """
        print(f"Starting to process all raw memory files in: {self.memory_dir}")
        
        # Use retriever's optimized batch query to get unprocessed files
        unprocessed_file_paths = self.retriever.get_unprocessed_files(self.memory_dir)
        
        print(f"Found {len(unprocessed_file_paths)} unprocessed files")
        
        # Process each file
        for file_path_str in unprocessed_file_paths:
            file_path = Path(file_path_str)
            await self.process_single_file(file_path, store_raw)
        
        # Print statistics
        print("\n" + "="*60)
        print("Processing Complete - Statistics:")
        print(f"  Files processed:      {self.stats['files_processed']}")
        print(f"  Facts extracted:      {self.stats['facts_extracted']}")
        print(f"  Facts stored to L1:   {self.stats['facts_stored']}")
        print(f"  Raw content stored:   {self.stats['raw_content_stored']}")
        print("="*60)
        
        return self.stats.copy()
    
    def test_configuration(self) -> Dict[str, Any]:
        """
        Test and display configuration status.
        
        Returns
        -------
        Dict[str, Any]
            Configuration test results including:
            - openclaw_providers: List of provider names
            - available_providers: List of available providers with baseUrl and apiKey
            - deeprecall_config: DeepRecall configuration
            - selected_provider: Selected provider based on configuration
        """
        result = {
            "openclaw_providers": [],
            "available_providers": [],
            "deeprecall_config": {},
            "selected_provider": None
        }
        
        # Get OpenClaw configuration
        openclaw_config = self._get_openclaw_config()
        available_providers = []
        
        print("\n=== DeepRecall Configuration Test ===\n")
        print("1. OpenClaw Model Providers:")
        print(f"   Found {len(openclaw_config)} provider(s)")
        
        for provider_name, provider_config in openclaw_config.items():
            has_base_url = "baseUrl" in provider_config
            has_api_key = bool(provider_config.get("apiKey"))
            models = provider_config.get("models", [])
            status = "✅ Available" if has_base_url and has_api_key else "⚠️  Incomplete"
            
            print(f"\n   {provider_name}: {status}")
            if has_base_url:
                print(f"      baseUrl: {provider_config.get('baseUrl')}")
            if models:
                print(f"      models: {len(models)} model(s)")
                for model in models[:2]:
                    model_id = model.get("id", model) if isinstance(model, dict) else model
                    print(f"        - {model_id}")
                if len(models) > 2:
                    print(f"        ... and {len(models) - 2} more")
            
            if has_base_url and has_api_key:
                available_providers.append(provider_name)
        
        # Get DeepRecall configuration
        deeprecall_config = self.config.get("summarizer", {})
        print("\n2. DeepRecall Summarizer Configuration:")
        
        if deeprecall_config:
            print("   ✅ Custom configuration loaded")
            for key, value in deeprecall_config.items():
                print(f"      {key}: {value}")
        else:
            print("   ℹ️  Using default configuration")
        
        print("\n3. Provider Selection:")
        preferred = deeprecall_config.get("preferred_provider")
        
        if available_providers:
            print(f"   Available providers: {available_providers}")
            print("   Selection priority:")
            print("     1. preferred_provider from DeepRecall config")
            print("     2. First available provider with baseUrl and apiKey")
            print("     3. Rule-based extraction (fallback)")
            
            if preferred and preferred in available_providers:
                selected = preferred
                print(f"   ✅ Will use configured provider: {preferred}")
            elif preferred:
                print(f"   ⚠️  Configured provider '{preferred}' not available")
                print(f"   ⚠️  Will auto-select from: {available_providers}")
                selected = available_providers[0] if available_providers else None
            else:
                print(f"   ℹ️  No preferred provider configured")
                print(f"   ℹ️  Will auto-select from: {available_providers}")
                selected = available_providers[0] if available_providers else None
        else:
            print("   ❌ No available providers found")
            print("   ⚠️  Summarizer will use rule-based extraction")
            selected = None
        
        if selected:
            print(f"   📍 Selected provider: {selected}")
        
        print("\n=== Configuration Test Complete ===")
        
        # Populate result dictionary
        result["openclaw_providers"] = list(openclaw_config.keys())
        result["available_providers"] = available_providers
        result["deeprecall_config"] = deeprecall_config
        result["selected_provider"] = selected
        
        return result


# Module-level singleton function for DeepRecallSummarizer
def _get_summarizer(db_path: str = None, memory_dir: str = None) -> "DeepRecallSummarizer":
    """
    Get a fresh DeepRecallSummarizer instance (not a true singleton, 
    ensures fresh stats for each call).
    
    Returns
    -------
    DeepRecallSummarizer
        New summarizer instance with fresh statistics
    """
    return DeepRecallSummarizer(db_path=db_path, memory_dir=memory_dir)

# Module-level wrapper functions for OpenClaw tool registration
async def summarize_memory_files(
    process_all: bool = False, 
    process_file: str = None, 
    no_store_raw: bool = False
) -> dict:
    """
    Module-level wrapper for OpenClaw tool registration.
    
    Parameters
    ----------
    process_all : bool
        Process all unprocessed memory files
    process_file : str, optional
        Process a specific memory file (relative to memory directory)
    no_store_raw : bool
        Do not store raw content to L2 archive
        
    Returns
    -------
    dict
        Processing statistics or result dictionary
    """
    summarizer = _get_summarizer()
    store_raw = not no_store_raw
    
    if process_file:
        memory_dir = get_workspace_memory_dir()
        file_path = Path(memory_dir) / process_file
        success = await summarizer.process_single_file(file_path, store_raw=store_raw)
        return {"success": success, "file": process_file, "stats": summarizer.stats}
    elif process_all:
        stats = await summarizer.process_all_files(store_raw=store_raw)
        return {"success": True, "stats": stats}
    else:
        return {"error": "Either process_all or process_file must be specified"}


def main():
    """Command-line interface for DeepRecall Summarizer."""
    
    parser = argparse.ArgumentParser(
        description="DeepRecall Memory Summarizer - LLM-powered fact extraction for SQLite memory system"
    )
    parser.add_argument(
        "--process-all",
        action="store_true",
        help="Process all unprocessed memory files"
    )
    parser.add_argument(
        "--process-file",
        help="Process a specific memory file (relative to memory directory)"
    )
    parser.add_argument(
        "--no-store-raw",
        action="store_true",
        help="Do not store raw content to L2 archive (only extract facts)"
    )
    parser.add_argument(
        "--db-path",
        help="Custom database path (default: auto-detected)"
    )
    parser.add_argument(
        "--memory-dir",
        help="Custom memory directory (default: auto-detected)"
    )
    parser.add_argument(
        "--test-config",
        action="store_true",
        help="Test OpenClaw configuration and API connectivity"
    )
    
    args = parser.parse_args()
    
    # Initialize summarizer
    summarizer = DeepRecallSummarizer(db_path=args.db_path, memory_dir=args.memory_dir)
    
    print("DeepRecall Memory Summarizer")
    print(f"Database: {summarizer.db_path}")
    print(f"Memory directory: {summarizer.memory_dir}")
    
    if args.test_config:
        # Test configuration using the unified method
        summarizer.test_configuration()
    
    elif args.process_all:
        # Process all files
        store_raw = not args.no_store_raw
        print(f"\nProcessing all files (store_raw: {store_raw})...")
        stats = asyncio.run(summarizer.process_all_files(store_raw=store_raw))
        
        # Save statistics to file
        stats_file = Path(summarizer.memory_dir) / "summarizer_stats.json"
        with open(stats_file, 'w', encoding='utf-8') as f:
            json.dump(stats, f, indent=2)
        print(f"Statistics saved to: {stats_file}")
    
    elif args.process_file:
        # Process single file
        store_raw = not args.no_store_raw
        file_path = Path(args.process_file)
        
        if not file_path.is_absolute():
            # Assume relative to memory directory
            file_path = Path(summarizer.memory_dir) / file_path
        
        print(f"\nProcessing single file: {file_path} (store_raw: {store_raw})...")
        success = asyncio.run(summarizer.process_single_file(file_path, store_raw))
        
        if success:
            print(f"File processed successfully")
        else:
            print(f"File processing failed")
            sys.exit(1)
    
    else:
        # Show help
        print("\nAvailable commands:")
        print("  --process-all       Process all unprocessed memory files")
        print("  --process-file      Process a specific memory file")
        print("  --no-store-raw      Do not store raw content (only facts)")
        print("  --test-config       Test OpenClaw configuration")
        print("\nExamples:")
        print("  python3 memory_summarizer.py --process-all")
        print("  python3 memory_summarizer.py --process-file 2024-01-01-daily-log.md")
        print("  python3 memory_summarizer.py --test-config")

if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)