Skill: marcosathanasoulis/openclaw-universal-memory (version 1.0.0)

--- name: openclaw-universal-memory description: Connector-agnostic Postgres + pgvector memory ingestion and retrieval with incremental cursor history. --- # OpenClaw Universal Memory This skill provides a generic memory layer for heterogeneous data: - canonical entity/chunk schema, - connector-style ingestion with cursors, - searchable memory in Postgres. ## Use Cases - Normalize records from multiple systems into one schema. - Keep incremental sync history (`cursor` per connector/account). - Build RAG-ready chunk storage in pgvector. ## Prerequisites - Postgres with `vector` extension. - Local package installed: `pip install -e .`. - Python dependency for DB I/O: - `pip install "psycopg[binary]>=3.2"` - DSN provided via environment variable (`DATABASE_DSN` by default). ## Security Boundaries - Do not pass raw passwords/tokens in command-line arguments. - Prefer OS secret store or process environment injection for DSN. - This skill only reads/writes your configured Postgres database; it does not call external APIs directly. - Use least-privilege DB credentials (`SELECT/INSERT/UPDATE/DELETE` on `um_*` tables only). - Review and trust any custom connector before running it. ## Responsible Use Caveat - Use this only for accounts/data you legitimately control or are authorized to process. - You are responsible for privacy, retention, and regulatory compliance. - This project is provided under Apache 2.0 without operational warranty. - This implementation is mostly AI-generated code with experienced engineer oversight; validate before production use. ## Commands Store DB credentials once (recommended): ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action configure-dsn ``` Initialize schema: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action init-schema \ --dsn-env DATABASE_DSN ``` Ingest JSON/NDJSON: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action ingest-json \ --dsn-env DATABASE_DSN \ --source gmail \ --account [REDACTED_SECRET] \ --entity-type email \ --input /path/to/records.ndjson ``` Ingest from built-in connectors: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action ingest-connector \ --connector google \ --account [REDACTED_SECRET] \ --dsn-env DATABASE_DSN \ --limit 300 ``` Validate connector auth/config before ingest: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action validate-connector \ --connector google \ --account [REDACTED_SECRET] \ --dsn-env DATABASE_DSN \ --limit 1 ``` Search: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action search \ --dsn-env DATABASE_DSN \ --query "Deryk" \ --limit 20 ``` Recent ingest history: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action events \ --dsn-env DATABASE_DSN \ --limit 20 ``` Doctor check: ```bash python skills/openclaw-universal-memory/scripts/run_memory.py \ --action doctor ``` Scheduling reference: - `docs/SCHEDULING.md` (cron examples, 15-minute default, connector toggles) ## Connector Contract (for custom adapters) A connector returns normalized records + next cursor: - `external_id` - `entity_type` - `title` - `body_text` - `raw_json` - `meta_json` - `next_cursor` This keeps ingestion generic and supports arbitrary source systems. Starter connector templates: - `src/openclaw_memory/connectors/templates.py` Step-by-step setup guide (Gmail/Slack/Asana/iMessage): - `docs/CONNECTOR_SETUP_WALKTHROUGH.md` ## Community We welcome connector contributions via PR. See `docs/CONNECTOR_CONTRIBUTING.md` for required contract, tests, and setup instructions.

path: scripts/run_memory.py
content: #!/usr/bin/env python3
from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path


def _find_repo_root() -> Path:
    here = Path(__file__).resolve()
    return here.parents[3]


def _parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="OpenClaw universal memory launcher")
    p.add_argument(
        "--action",
        required=True,
        choices=[
            "init-schema",
            "ingest-json",
            "ingest-connector",
            "validate-connector",
            "search",
            "events",
            "doctor",
            "configure-dsn",
        ],
    )
    p.add_argument("--dsn", default="")
    p.add_argument(
        "--dsn-env",
        default="DATABASE_DSN",
        help="Environment variable to read DSN from if --dsn is not provided",
    )
    p.add_argument("--dsn-file", default="", help="Path to env-style file with DATABASE_DSN=...")
    p.add_argument("--dsn-key", default="DATABASE_DSN")
    p.add_argument("--source")
    p.add_argument("--connector")
    p.add_argument("--connector-config", default="")
    p.add_argument("--account")
    p.add_argument("--entity-type")
    p.add_argument("--input")
    p.add_argument("--query")
    p.add_argument("--limit", type=int, default=20)
    p.add_argument("--cursor", default="")
    p.add_argument("--id-field", default="id")
    p.add_argument("--title-field", default="title")
    p.add_argument("--body-field", default="body")
    p.add_argument("--config-path", default="")
    p.add_argument("--force", action="store_true")
    return p.parse_args()


def main() -> None:
    args = _parse_args()
    repo_root = _find_repo_root()
    subprocess_env = os.environ.copy()
    if args.dsn.strip():
        subprocess_env[args.dsn_env] = args.dsn.strip()
    repo_src = str((repo_root / "src").resolve())
    existing_pythonpath = subprocess_env.get("PYTHONPATH", "").strip()
    subprocess_env["PYTHONPATH"] = f"{repo_src}:{existing_pythonpath}" if existing_pythonpath else repo_src

    base = [
        sys.executable,
        "-m",
        "openclaw_memory.cli",
        args.action,
        "--dsn-env",
        args.dsn_env,
        "--dsn-file",
        args.dsn_file,
        "--dsn-key",
        args.dsn_key,
    ]
    if args.action == "configure-dsn":
        if args.dsn.strip():
            base.extend(["--dsn", args.dsn.strip()])
        if args.config_path.strip():
            base.extend(["--config-path", args.config_path.strip()])
        if args.force:
            base.append("--force")
        result = subprocess.run(base, cwd=repo_root, capture_output=True, text=True, env=subprocess_env)
        if result.stdout:
            print(result.stdout.strip())
        if result.stderr:
            print(result.stderr.strip(), file=sys.stderr)
        raise SystemExit(result.returncode)
    if args.action == "ingest-json":
        required = {
            "source": args.source,
            "account": args.account,
            "entity_type": args.entity_type,
            "input": args.input,
        }
        missing = [k for k, v in required.items() if not v]
        if missing:
            print(json.dumps({"ok": False, "error": f"Missing required args: {', '.join(missing)}"}, indent=2))
            raise SystemExit(2)
        base.extend(
            [
                "--source",
                args.source,
                "--account",
                args.account,
                "--entity-type",
                args.entity_type,
                "--input",
                args.input,
                "--limit",
                str(args.limit),
                "--cursor",
                args.cursor,
                "--id-field",
                args.id_field,
                "--title-field",
                args.title_field,
                "--body-field",
                args.body_field,
            ]
        )
    elif args.action in {"ingest-connector", "validate-connector"}:
        required = {
            "connector": args.connector,
            "account": args.account,
        }
        missing = [k for k, v in required.items() if not v]
        if missing:
            print(json.dumps({"ok": False, "error": f"Missing required args: {', '.join(missing)}"}, indent=2))
            raise SystemExit(2)
        base.extend(
            [
                "--connector",
                args.connector,
                "--account",
                args.account,
                "--limit",
                str(args.limit),
                "--cursor",
                args.cursor,
                "--connector-config",
                args.connector_config,
            ]
        )
        if args.source:
            base.extend(["--source", args.source])
        if args.action == "ingest-connector" and args.entity_type:
            base.extend(["--entity-type", args.entity_type])
    elif args.action == "search":
        if not args.query:
            print(json.dumps({"ok": False, "error": "Missing required arg: --query"}, indent=2))
            raise SystemExit(2)
        base.extend(["--query", args.query, "--limit", str(args.limit)])
        if args.source:
            base.extend(["--source", args.source])
        if args.account:
            base.extend(["--account", args.account])
        if args.entity_type:
            base.extend(["--entity-type", args.entity_type])
    elif args.action == "events":
        base.extend(["--limit", str(args.limit)])

    result = subprocess.run(base, cwd=repo_root, capture_output=True, text=True, env=subprocess_env)
    if result.stdout:
        print(result.stdout.strip())
    if result.stderr:
        print(result.stderr.strip(), file=sys.stderr)
    raise SystemExit(result.returncode)


if __name__ == "__main__":
    main()


path: setup.json
content: {
  "setupComplete": false,
  "prerequisites": {
    "python": {
      "required": true,
      "installed": false,
      "description": "Python 3.10+"
    },
    "packageInstalled": {
      "required": true,
      "installed": false,
      "description": "openclaw-universal-memory installed locally (pip install -e .)"
    },
    "psycopg": {
      "required": true,
      "installed": false,
      "description": "Postgres + connector deps installed (pip install -e .)"
    },
    "postgresDsn": {
      "required": true,
      "configured": false,
      "description": "Use configure-dsn (recommended), DATABASE_DSN env, or --dsn/--dsn-file"
    }
  },
  "setupInstructions": [
    "1. Install package: pip install -e .",
    "2. Install DB driver: pip install \"psycopg[binary]>=3.2\".",
    "3. Configure DSN securely: run_memory.py --action configure-dsn (or set DATABASE_DSN).",
    "4. Ensure Postgres has pgvector extension enabled.",
    "5. Run doctor check: run_memory.py --action doctor.",
    "6. Initialize schema via run_memory.py --action init-schema.",
    "7. Configure connectors (google/slack/asana/imessage) per docs/CONNECTOR_SETUP_WALKTHROUGH.md.",
    "8. Validate each connector before ingestion: run_memory.py --action validate-connector ..."
  ],
  "verifySetup": "python skills/openclaw-universal-memory/scripts/run_memory.py --action doctor"
}


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: clean (LOW)