Skill: ogi98rus/lightrag-memory (version 1.0.1)

--- name: lightrag-memory description: >- LightRAG-based semantic memory system for AI agents. Provides efficient long-term knowledge storage and retrieval using vector embeddings and knowledge graphs. Use when: (1) semantic search over agent memory files is needed, (2) reducing token usage by avoiding full-file re-reading, (3) building a knowledge graph from text documents, (4) querying memory with natural language instead of keyword matching. Supports naive, local, global, and hybrid query modes. metadata: { "openclaw": { "requires": { "bins": ["python3", "pip"], "env": ["OPENAI_API_KEY"], }, "install": [ { "id": "lightrag-memory-deps", "kind": "pip", "requirements": "requirements.txt", "label": "Install LightRAG dependencies", }, ], }, } --- # LightRAG Memory Semantic memory system with vector search + knowledge graph. Replaces reading entire memory files on every request with targeted retrieval (~1-3K tokens vs 30K+). ## Quick Setup ```bash cd skills/lightrag-memory pip install -r requirements.txt ``` Set environment variables: ```bash export OPENAI_API_KEY="your-key" export OPENAI_BASE_URL="https://your-api-endpoint/v1" # optional, defaults to OpenAI ``` Or create a `.env` file in the skill directory. ## Commands ### Index memory files Index `MEMORY.md` and `memory/*.md` from workspace: ```bash python3 scripts/rag.py index ``` ### Insert content ```bash # From file python3 scripts/rag.py insert --file /path/to/file.md --source "filename" # From text python3 scripts/rag.py insert --text "Important fact" --source "manual" # From stdin echo "Some text" | python3 scripts/rag.py insert --source "stdin" ``` ### Query ```bash # Hybrid search (best results, costs more API calls) python3 scripts/rag.py query "What do I know about the user?" --mode hybrid # Local search (entities + relationships, balanced) python3 scripts/rag.py query "What projects were discussed?" --mode local # Naive search (simple vector lookup, cheapest) python3 scripts/rag.py query "Any notes about deployment?" --mode naive # Global search (broad context, expensive) python3 scripts/rag.py query "Summarize everything" --mode global ``` ## Query Modes | Mode | What it searches | Cost | Best for | |------|-----------------|------|----------| | naive | Vector embeddings only | Lowest | Quick fact lookup | | local | Entities + relationships | Low | Specific entities | | global | Community-level context | High | Broad understanding | | hybrid | Local + global | Highest | Comprehensive answers | ## Storage Data stored in `~/.openclaw/workspace/lightrag_storage/` by default. Override with `LIGHTARG_WORKING_DIR` env var. ## Integration Pattern For agent memory systems, index files on change and query on demand: ```bash # Check if reindex needed (files modified since last index) find MEMORY.md memory/ -name '*.md' -newer memory/lightrag-last-index.txt # Reindex if needed python3 scripts/rag.py index touch memory/lightrag-last-index.txt # Query when context needed python3 scripts/rag.py query "user preferences" --mode naive ``` ## Architecture - **Embeddings**: `text-embedding-3-small` (1536 dim) via OpenAI-compatible API - **LLM**: `gpt-4o-mini` for entity extraction and answer generation - **Storage**: JSON-based vector DB + GraphML knowledge graph - **Batching**: 8 items per embedding batch, 4 concurrent async requests ## Troubleshooting **OPENAI_API_KEY not set**: Ensure env vars are exported or `.env` exists. **numpy RuntimeError (X86_V2)**: On older CPUs lacking AVX2, install `pip install "numpy<2.0"`. **Slow first index**: Initial indexing processes all files. Subsequent updates are incremental. **Reset storage**: `rm -rf <storage_dir>/* && python3 scripts/rag.py index`

path: requirements.txt
content: lightrag-hku
openai
python-dotenv
numpy


path: scripts/rag.py
content: #!/usr/bin/env python3
"""
LightRAG memory management for AI agent skills.
Index, insert, and query semantic memory using vector embeddings + knowledge graph.
"""
import os
import sys
import argparse
import asyncio
import numpy as np
from pathlib import Path

from lightrag import LightRAG, QueryParam
from lightrag.llm.openai import openai_complete_if_cache
from lightrag.utils import EmbeddingFunc

# Storage directory
WORKING_DIR = os.environ.get(
    "LIGHTARG_WORKING_DIR",
    os.path.expanduser("~/.openclaw/workspace/lightrag_storage")
)
os.makedirs(WORKING_DIR, exist_ok=True)


async def _custom_embed(texts, model="text-embedding-3-small", base_url=None):
    """Embedding function with custom base_url support."""
    import openai

    [REDACTED_SECRET]"OPENAI_API_KEY")
    if base_url is None:
        base_url = os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1")

    if not base_url.endswith("/v1"):
        base_url = base_url.rstrip("/") + "/v1"

    client = openai.AsyncOpenAI([REDACTED_SECRET], base_url=base_url)
    response = await client.embeddings.create(
        model=model, input=texts, encoding_format="float"
    )
    return np.array([item.embedding for item in response.data])


def get_rag():
    """Create configured LightRAG instance."""
    [REDACTED_SECRET]"OPENAI_API_KEY")
    if not [REDACTED_SECRET]"Error: OPENAI_API_KEY not set", file=sys.stderr)
        sys.exit(1)

    base_url = os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1")

    async def llm_func(prompt, system_prompt=None, history_messages=None, **kw):
        return await openai_complete_if_cache(
            model="gpt-4o-mini",
            prompt=prompt,
            system_prompt=system_prompt,
            history_messages=history_messages,
            base_url=base_url,
            [REDACTED_SECRET],
            **kw,
        )

    async def embed_func(texts):
        return await _custom_embed(texts, base_url=base_url)

    embedding = EmbeddingFunc(
        embedding_dim=1536,
        func=embed_func,
        max_token_size=8192,
    )

    return LightRAG(
        working_dir=WORKING_DIR,
        llm_model_func=llm_func,
        embedding_func=embedding,
        embedding_func_max_async=4,
        embedding_batch_num=8,
    )


async def insert_text(text, source=None):
    rag = get_rag()
    await rag.initialize_storages()
    await rag.ainsert(text)
    print(f"Inserted text from {source or 'stdin'}")


async def insert_file(filepath):
    with open(filepath, "r", encoding="utf-8") as f:
        text = f.read()
    await insert_text(text, source=filepath)


async def do_query(question, mode="hybrid"):
    rag = get_rag()
    await rag.initialize_storages()
    result = await rag.aquery(question, param=QueryParam(mode=mode))
    print(result)


async def index_memory_files(workspace=None):
    """Index MEMORY.md and memory/*.md from workspace."""
    if workspace is None:
        workspace = os.path.expanduser("~/.openclaw/workspace")

    files = []
    memory_md = os.path.join(workspace, "MEMORY.md")
    if os.path.exists(memory_md):
        files.append(memory_md)

    memory_dir = os.path.join(workspace, "memory")
    if os.path.isdir(memory_dir):
        files.extend(str(p) for p in Path(memory_dir).glob("*.md"))

    if not files:
        print("No memory files found to index.")
        return

    print(f"Indexing {len(files)} files...")
    rag = get_rag()
    await rag.initialize_storages()

    for fp in files:
        print(f"  - {fp}")
        try:
            with open(fp, "r", encoding="utf-8") as f:
                text = f.read()
            await rag.ainsert(text)
            print("    ✓ Done")
        except Exception as e:
            print(f"    ✗ Error: {e}")

    print("Indexing complete!")


def main():
    # Load .env from script directory if present
    script_dir = Path(__file__).parent
    env_file = script_dir.parent / ".env"
    if env_file.exists():
        try:
            from dotenv import load_dotenv
            load_dotenv(env_file)
        except ImportError:
            pass

    parser = argparse.ArgumentParser(description="LightRAG memory management")
    sub = parser.add_subparsers(dest="command")

    # insert
    ins = sub.add_parser("insert", help="Insert text into knowledge base")
    ins.add_argument("--file", help="File path to insert")
    ins.add_argument("--text", help="Text to insert")
    ins.add_argument("--source", help="Source identifier")

    # query
    qry = sub.add_parser("query", help="Query knowledge base")
    qry.add_argument("question", help="Question to ask")
    qry.add_argument(
        "--mode", default="hybrid",
        choices=["naive", "local", "global", "hybrid"],
    )

    # index
    idx = sub.add_parser("index", help="Index workspace memory files")
    idx.add_argument("--workspace", help="Workspace root path")

    args = parser.parse_args()

    if args.command == "insert":
        if args.file:
            asyncio.run(insert_file(args.file))
        elif args.text:
            asyncio.run(insert_text(args.text, source=args.source))
        else:
            asyncio.run(insert_text(sys.stdin.read(), source=args.source))
    elif args.command == "query":
        asyncio.run(do_query(args.question, mode=args.mode))
    elif args.command == "index":
        asyncio.run(index_memory_files(workspace=args.workspace))
    else:
        parser.print_help()


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)