Skill: lookupmark/lookupmark-local-rag (version 1.9.1)

--- name: local-rag description: > Semantic search over local files using all-MiniLM-L6-v2 embeddings and ms-marco-MiniLM-L-6-v2 cross-encoder reranking with ChromaDB and parent-child chunking. Optimized for low-RAM ARM devices (Raspberry Pi). Use when the user asks questions about their documents, wants to find information in their files without specifying exact paths, or needs to search across PDFs, DOCX, TXT, MD, TEX files. Triggers on "cerca nei miei file", "find in my files", "search documents", "trova nel computer", "what does my thesis say about", "search notes". NOT for exact file path requests, web search, or sending files. --- # Local RAG Semantic search over indexed local files with parent-child chunking for precise retrieval with full context. ## Architecture | Component | Model | Size | |-----------|-------|------| | Embeddings | [REDACTED_SECRET] | ~80MB | | Reranker | [REDACTED_SECRET] | ~80MB | | Vector DB | ChromaDB (persistent, cosine similarity, HNSW) | varies | | Chunking | Parent-child | — | **Memory strategy**: Embedding model loaded first → freed with `gc.collect()` → reranker loaded → freed after scoring. This keeps peak RAM ~400MB on ARM. ## Chunking Strategy - **Child chunks**: 128 words, 24 overlap → embedded for semantic search - **Parent chunks**: 768 words → stored as full context, returned to user - When a child matches → its parent is returned, giving surrounding context ## Running All scripts must use the venv Python: ```bash VENV=~/.local/share/local-rag/venv/bin/python ``` ### Indexing ```bash # Incremental index (default — skips unchanged files via SHA-256 hash) $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py # Re-index from scratch $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py --reindex # Custom paths $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py --paths ~/Documenti ~/Progetti # Batch indexing (per-subfolder with git checkpoints, for low-RAM systems) bash ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index-batch.sh ``` ### Querying ```bash # Basic query $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "what are the termination clauses?" # More results $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "Falcon LLM" --top-k 30 --top-n 5 # JSON output for programmatic use $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "transformer architecture" --json # With timeout $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "deep learning" --timeout 60 ``` Options: - `--top-k N` — Child candidates from vector search (default: 20) - `--top-n N` — Final parent results after reranking (default: 3) - `--json` — JSON output - `--timeout N` — Max seconds per query (default: 120) ### Monitoring ```bash $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py # Status $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --watch # Auto-refresh $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --log # Logs $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --errors # Errors only $VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --git # Git checkpoints ``` ## Supported Formats **Documents only** (no code files): - Text: `.txt`, `.md`, `.csv`, `.json`, `.yaml`, `.yml`, `.toml`, `.tex`, `.bib` - Documents: `.pdf` (pdfminer.six), `.docx` (python-docx), `.pptx` **Excluded**: `.py`, `.js`, `.sh`, `.ipynb`, `.html`, `.css` and all code files. ## Limits (for 4GB ARM) - PDF max size: 5MB (larger PDFs cause OOM with pdfminer) - Max file size: 30MB - Embedding batch size: 1 (conservative) - Excluded dirs: `.git`, `.venv`, `node_modules`, `__pycache__`, `labs`, `exercises`, `src`, `scripts`, `ablation`, `test*`, `fixtures` ## Storage | Path | Purpose | |------|---------| | `~/.local/share/local-rag/chromadb/` | ChromaDB data (git repo for rollback) | | `~/.local/share/local-rag/venv/` | Python venv with dependencies | | `~/.local/share/local-rag/index.lock` | Prevents concurrent indexing | | `~/.local/share/local-rag/index-batch.log` | Batch indexing log | | `~/.local/share/local-rag/queries.log` | Query history log | ## Security - **ALLOWED_ROOTS**: Only `~/Documenti/github/thesis`, `~/Documenti/github/polito`, `~/Documenti`, `~/Scaricati` - **BLOCKED_PATTERNS**: `.ssh`, `.gnupg`, `.env`, `credentials`, `tokens`, `.config/openclaw` - Credentials directory is blacklisted — never indexed ## Workflow 1. Run `index.py` — builds/rebuilds the index (incremental via SHA-256 hash check) 2. Run periodically to pick up new/changed files (daily cron recommended) 3. Use `query.py` to search with natural language 4. Results include: file path, relevance score, matched snippet, full parent context 5. Check `monitor.py` for stats and `queries.log` for query history

path: _meta.json
content: ownerId: kn717xksfb8vf0ja4mdr5eqqbs8457k4
slug: lookupmark-local-rag
version: 1.1.0
publishedAt: 1775226497446

path: scripts/index-batch-ram.sh
content: #!/usr/bin/env bash
# index-batch-ram.sh — Two-phase indexing to avoid OOM on low-RAM systems.
# Phase 1: Text files only (md, txt, tex) — 25 files per round
# Phase 2: PDF files up to 25MB — 1 file per round
set -euo pipefail

VENV_PY="$HOME/.local/share/local-rag/venv/bin/python"
SCRIPT="$HOME/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py"

MAX_ROUNDS=50

echo "=========================================="
echo "FASE 1: Indicizzazione file testuali"
echo "=========================================="

for i in $(seq 1 $MAX_ROUNDS); do
    echo "=== Phase 1 — Round $i/$MAX_ROUNDS (max 25 files) ==="
    OUTPUT=$($VENV_PY "$SCRIPT" --text-only --max-files 25 2>&1) || true
    echo "$OUTPUT"
    
    if echo "$OUTPUT" | grep -q "Nuovi indicizzati:       0" && echo "$OUTPUT" | grep -q "Aggiornati:              0"; then
        echo "=== Phase 1 complete. All text files indexed. ==="
        break
    fi
    
    echo "--- Round $i complete, waiting 5s ---"
    sleep 5
done

echo ""
echo "=========================================="
echo "FASE 2: Indicizzazione PDF (1 per volta)"
echo "=========================================="

for i in $(seq 1 $MAX_ROUNDS); do
    echo "=== Phase 2 — Round $i/$MAX_ROUNDS (1 PDF) ==="
    OUTPUT=$($VENV_PY "$SCRIPT" --pdf-only --max-files 1 2>&1) || true
    echo "$OUTPUT"
    
    if echo "$OUTPUT" | grep -q "Nuovi indicizzati:       0" && echo "$OUTPUT" | grep -q "Aggiornati:              0"; then
        echo "=== Phase 2 complete. All PDFs indexed. ==="
        break
    fi
    
    echo "--- Round $i complete, waiting 5s ---"
    sleep 5
done

echo ""
echo "=========================================="
echo "INDICIZZAZIONE COMPLETATA"
echo "=========================================="


path: scripts/index-batch.sh
content: #!/bin/bash
# Robust batch indexer with git rollback fallback and watchdog.
# Usage: bash index-batch.sh [PATH1 PATH2 ...]
# If no paths given, indexes all subdirectories of ~/Documenti/github/ recursively.

set -euo pipefail

export TMPDIR="${TMPDIR:-$HOME/.local/share/local-rag/tmp}"
PYTHON="$HOME/.local/share/local-rag/venv/bin/python"
INDEX="$HOME/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py"
DB="$HOME/.local/share/local-rag/chromadb"
LOCK="$HOME/.local/share/local-rag/index.lock"
LOG="$HOME/.local/share/local-rag/index-batch.log"
OOM_PROTECT="$HOME/.local/share/local-rag/oom-protect.pid"

MAX_RETRIES=2
TIMEOUT_MINUTES=30

log() { echo "[$(date '+%H:%M:%S')] $*" | tee -a "$LOG"; }

cleanup() {
    rm -f "$LOCK" "$OOM_PROTECT" 2>/dev/null
    # Kill any lingering python index process
    pkill -f "lookupmark-local-rag/scripts/index.py" 2>/dev/null || true
}
trap cleanup EXIT

# Ensure DB git repo exists
if [ ! -d "$DB/.git" ]; then
    mkdir -p "$DB"
    cd "$DB" && git init && git commit --allow-empty -m "clean slate"
    log "Initialized git repo at $DB"
fi

git_checkpoint() {
    cd "$DB"
    git add -A
    if git diff --cached --quiet; then
        log "  No changes to commit"
    else
        git commit -m "$1" 2>&1 | tee -a "$LOG"
    fi
    cd -
}

git_rollback() {
    log "  Rolling back to previous checkpoint..."
    cd "$DB"
    # Find last good commit (skip current broken state)
    PREV=$(git log --oneline -2 --format="%H" | tail -1)
    if [ -n "$PREV" ]; then
        git checkout -- .
        git reset --hard "$PREV"
        log "  Rolled back to $(git log --oneline -1)"
    fi
    cd -
}

index_path() {
    local path="$1"
    rm -f "$LOCK"

    # Disk space check
    local free_mb=$(df -m --output=avail "$DB" | tail -1 | tr -d ' ')
    if [ "$free_mb" -lt 500 ]; then
        log "  ❌ Low disk space (${free_mb}MB free < 500MB threshold) — aborting"
        return 1
    fi

    log "Indexing: $path"

    # Run with timeout and OOM protection
    timeout "$((TIMEOUT_MINUTES * 60))" "$PYTHON" "$INDEX" --paths "$path" >> "$LOG" 2>&1
    local exit_code=$?

    if [ $exit_code -eq 0 ]; then
        log "  ✅ Success"
        git_checkpoint "$(basename "$path")"
        return 0
    elif [ $exit_code -eq 137 ]; then
        log "  ❌ OOM killed (SIGKILL) — rolling back"
        git_rollback
        return 1
    elif [ $exit_code -eq 124 ]; then
        log "  ❌ Timeout (${TIMEOUT_MINUTES}min) — rolling back"
        git_rollback
        return 1
    else
        log "  ❌ Exit code $exit_code — rolling back"
        git_rollback
        return 1
    fi
}

# Collect paths
if [ $# -gt 0 ]; then
    PATHS=("$@")
else
    # Auto-discover: all subdirs with indexable files
    mapfile -t PATHS < <(find ~/Documenti/github/thesis ~/Documenti/github/polito \
        -mindepth 1 -maxdepth 4 -type d \
        ! -path "*/labs/*" ! -path "*/exercises/*" ! -path "*/.git/*" \
        ! -path "*/node_modules/*" ! -path "*/__pycache__/*" ! -path "*/.venv/*" \
        ! -path "*src*" ! -path "*scripts*" ! -path "*tests*" \
        ! -path "*notebooks*" \
        2>/dev/null | sort -u)
fi

FAILED=()
TOTAL=${#PATHS[@]}
DONE=0

log "=== Starting batch indexing: $TOTAL paths ==="

for p in "${PATHS[@]}"; do
    DONE=$((DONE + 1))
    log "[$DONE/$TOTAL] Processing: $p"

    # Retry loop
    SUCCESS=false
    for attempt in $(seq 1 $((MAX_RETRIES + 1))); do
        if index_path "$p"; then
            SUCCESS=true
            break
        fi
        if [ $attempt -le $MAX_RETRIES ]; then
            log "  Retry $attempt/$MAX_RETRIES in 10s..."
            sleep 10
        fi
    done

    if [ "$SUCCESS" = false ]; then
        FAILED+=("$p")
        log "  ⚠️  Permanently failed after $((MAX_RETRIES + 1)) attempts: $p"
    fi
done

# Summary
log "=== DONE ==="
if [ ${#FAILED[@]} -eq 0 ]; then
    log "All $TOTAL paths indexed successfully! ✅"
else
    log "Completed: $((TOTAL - ${#FAILED[@]}))/$TOTAL"
    log "Failed paths:"
    for f in "${FAILED[@]}"; do
        log "  - $f"
    done
fi

# Print final DB stats
"$PYTHON" -c "
import chromadb
c = chromadb.PersistentClient(path='$DB')
p = c.get_or_create_collection('parents')
ch = c.get_or_create_collection('children')
print(f'DB: {p.count()} parents, {ch.count()} children')
" 2>/dev/null | tee -a "$LOG"


path: scripts/index.py
content: #!/usr/bin/env python3
"""Index local files into ChromaDB with all-MiniLM-L6-v2 embeddings and parent-child chunking.

Parent chunks (large) are stored as documents for retrieval.
Child chunks (small) are embedded for precise semantic search.
When a child matches, the parent is returned for full context.

Usage:
    index.py [--reindex] [--paths PATH1 PATH2 ...]

Default paths: ~/Documenti and ~/Scaricati.
Use --reindex to drop existing data and re-index from scratch.
"""

import argparse
import hashlib
import logging
import os
import sys
from pathlib import Path

import chromadb
from sentence_transformers import SentenceTransformer

# Silence pdfminer FontBBox spam
logging.getLogger("pdfminer").setLevel(logging.ERROR)

# --- Config ---

EMBED_MODEL = [REDACTED_SECRET]
PARENT_SIZE = 768       # words per parent chunk
CHILD_SIZE = 128        # words per child chunk
CHILD_OVERLAP = 24      # overlap between children
MAX_FILE_SIZE_MB = 25  # Max file size for PDFs (raised from 5 to support larger slides)
# Auto-detect batch size based on available RAM
import struct
try:
    with open('/proc/meminfo') as _f:
        _memline = _f.readline()
        _total_kb = int(_memline.split()[1])
    BATCH_SIZE = 1  # Forced to 1 — Pi 5 has limited RAM, larger batches cause OOM
except Exception:
    BATCH_SIZE = 1

# Security: only allow indexing under these directories
ALLOWED_ROOTS = [
    os.path.expanduser("~/Documenti/github/thesis"),
    os.path.expanduser("~/Documenti/github/polito"),
    os.path.expanduser("~/Documenti"),
    os.path.expanduser("~/Scaricati"),
]

BLOCKED_PATTERNS = {".ssh", ".gnupg", ".env", "credentials", "tokens", ".config/openclaw"}

DEFAULT_PATHS = [
    "~/Documenti",
    "~/Scaricati",
]

TEXT_EXTENSIONS = {
    ".txt", ".md", ".rst",
    ".tex", ".bib",
}

DOCUMENT_EXTENSIONS = {
    ".pdf",
    ".docx", ".odt", ".rtf", ".epub", ".xlsx", ".pptx",
}

ALL_EXTENSIONS = TEXT_EXTENSIONS | DOCUMENT_EXTENSIONS

DB_DIR = os.path.expanduser("~/.local/share/local-rag/chromadb")


# --- Helpers ---

def file_hash(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()


def split_words(text: str) -> list[str]:
    return text.split()


def make_parent_chunks(words: list[str], size: int = PARENT_SIZE) -> list[str]:
    """Split words into parent chunks (no overlap)."""
    parents = []
    for i in range(0, len(words), size):
        chunk = " ".join(words[i : i + size])
        if chunk.strip():
            parents.append(chunk)
    return parents


def make_child_chunks(parent_words: list[str], size: int = CHILD_SIZE, overlap: int = CHILD_OVERLAP) -> list[str]:
    """Split a parent's words into overlapping child chunks."""
    children = []
    for i in range(0, len(parent_words), size - overlap):
        chunk = " ".join(parent_words[i : i + size])
        if chunk.strip():
            children.append(chunk)
    return children


def extract_text(filepath: str) -> str | None:
    ext = Path(filepath).suffix.lower()

    if ext in TEXT_EXTENSIONS:
        try:
            return Path(filepath).read_text(encoding="utf-8", errors="ignore")
        except Exception:
            return None

    if ext == ".pdf":
        try:
            # Skip PDFs > MAX_FILE_SIZE_MB to avoid OOM
            if os.path.getsize(filepath) > MAX_FILE_SIZE_MB * 1024 * 1024:
                print(f"    Skipping {filepath}: PDF too large ({os.path.getsize(filepath)//1024//1024}MB > {MAX_FILE_SIZE_MB}MB limit)")
                return None
            from pdfminer.high_level import extract_text
            text = extract_text(filepath) or ""
            if len(text) > 10 * 1024 * 1024:
                print(f"    Skipping {filepath}: extracted text too large ({len(text)//1024//1024}MB > 10MB)")
                return None
            return text
        except Exception:
            return None

    if ext == ".docx":
        try:
            from docx import Document
            return "\n".join(p.text for p in Document(filepath).paragraphs)
        except ImportError:
            print(f"    Skipping {filepath}: python-docx not installed. Install with: pip install python-docx", file=sys.stderr)
            return None
        except Exception:
            return None

    return None


def crawl(root: str) -> list[str]:
    files = []
    root = os.path.expanduser(root)
    if not os.path.isdir(root):
        print(f"  Skipping {root} (not a directory)")
        return files
    # Security: validate path is under allowed roots
    root_real = os.path.realpath(root)
    allowed = any(root_real.startswith(os.path.realpath(ar)) for ar in ALLOWED_ROOTS)
    if not allowed:
        print(f"  Skipping {root} (outside allowed directories)")
        return files
    for dirpath, dirnames, filenames in os.walk(root):
        # Security: skip sensitive directories
        dirnames[:] = [d for d in dirnames
                       if not d.startswith(".")
                       and d not in {
                           "node_modules", "__pycache__", ".git", ".venv", "venv",
                           "labs", "exercises", *BLOCKED_PATTERNS,
                       }]
        for fn in filenames:
            ext = Path(fn).suffix.lower()
            if ext in ALL_EXTENSIONS:
                full = os.path.join(dirpath, fn)
                size_mb = os.path.getsize(full) / (1024 * 1024)
                if size_mb <= MAX_FILE_SIZE_MB:
                    files.append(full)
    return files


# --- Main ---

LOCK_FILE = os.path.expanduser("~/.local/share/local-rag/index.lock")


def check_memory():
    """Check available memory. Returns available MB."""
    try:
        with open("/proc/meminfo") as f:
            for line in f:
                if line.startswith("MemAvailable:"):
                    return int(line.split()[1]) // 1024
    except Exception:
        pass
    return 999  # If we can't read, assume OK


def gc_and_check(min_mb=150):
    """Force GC and check memory. Returns True if OK, False if low."""
    import gc; gc.collect()
    avail = check_memory()
    if avail < min_mb:
        print(f"  WARNING: Low memory ({avail}MB available). Flushing and continuing carefully.")
        return False
    return True


def main():
    # Check available memory before doing anything heavy
    avail = check_memory()
    if avail < 200:
        print(f"ERROR: Insufficient memory ({avail}MB available, need 200MB). Aborting.", file=sys.stderr)
        sys.exit(1)
    print(f"Memory check: {avail}MB available")

    # Prevent concurrent runs
    import fcntl
    os.makedirs(os.path.dirname(LOCK_FILE), exist_ok=True)
    lock_fd = open(LOCK_FILE, "w")
    try:
        fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError:
        print("Another indexing process is already running. Exiting.")
        sys.exit(0)

    parser = argparse.ArgumentParser(description="Index local files with parent-child chunking")
    parser.add_argument("--reindex", action="store_true", help="Drop existing data and re-index")
    parser.add_argument("--paths", nargs="+", default=DEFAULT_PATHS, help="Directories to index")
    parser.add_argument("--max-files", type=int, default=0, help="Stop after N new/updated files (0=unlimited)")
    parser.add_argument("--text-only", action="store_true", help="Index only text files (md, txt, tex, bib)")
    parser.add_argument("--pdf-only", action="store_true", help="Index only PDF files")
    args = parser.parse_args()

    print(f"Loading embedding model: {EMBED_MODEL}...")
    model = SentenceTransformer(EMBED_MODEL)

    # Force garbage collection after model load
    import gc; gc.collect()

    print(f"Opening ChromaDB at {DB_DIR}...")
    client = chromadb.PersistentClient(path=DB_DIR)

    if args.reindex:
        print("Dropping existing collections...")
        for name in ["parents", "children"]:
            try:
                client.delete_collection(name)
            except Exception:
                pass

    parents_col = client.get_or_create_collection("parents", metadata={"hnsw:space": "cosine"})
    children_col = client.get_or_create_collection("children", metadata={"hnsw:space": "cosine"})

    # Track changes
    added_files = []
    updated_files = []
    removed_files = []
    skipped_unchanged = 0
    total_parents = 0
    total_children = 0
    stopped_early = False

    for root in args.paths:
        print(f"\nScanning {root}...")
        files = crawl(root)
        print(f"  Found {len(files)} files to index")

        # Collect ALL filepaths found on disk for this root (for orphan detection)
        all_disk_paths = set(files)

        for filepath in files:

            text = extract_text(filepath)
            if not text or not text.strip():
                continue

            # Filter by mode
            ext = Path(filepath).suffix.lower()
            if args.text_only and ext not in {".txt", ".md", ".rst", ".tex", ".bib"}:
                continue
            if args.pdf_only and ext != ".pdf":
                continue

            fhash = file_hash(filepath)

            # Check if already indexed with same hash
            existing = parents_col.get(
                where={"filepath": filepath},
                include=["metadatas"],
                limit=1,
            )
            if existing["metadatas"]:
                old_hash = existing["metadatas"][0].get("file_hash")
                if old_hash == fhash:
                    skipped_unchanged += 1
                    continue  # Unchanged

                # File changed — remove old chunks
                parents_col.delete(where={"filepath": filepath})
                children_col.delete(where={"filepath": filepath})
                updated_files.append(filepath)

            else:
                added_files.append(filepath)

            words = split_words(text)
            parents = make_parent_chunks(words)

            all_child_texts = []
            all_child_metas = []
            all_child_ids = []
            parent_ids = []
            parent_docs = []
            parent_metas = []

            for p_idx, parent_text in enumerate(parents):
                parent_id = f"{fhash}_p{p_idx}"
                parent_ids.append(parent_id)
                parent_docs.append(parent_text)
                parent_metas.append({
                    "filepath": filepath,
                    "parent_idx": p_idx,
                    "file_hash": fhash,
                })

                p_words = split_words(parent_text)
                child_texts = make_child_chunks(p_words)
                for c_idx, child_text in enumerate(child_texts):
                    child_id = f"{fhash}_p{p_idx}_c{c_idx}"
                    all_child_ids.append(child_id)
                    all_child_texts.append(child_text)
                    all_child_metas.append({
                        "filepath": filepath,
                        "parent_idx": p_idx,
                        "child_idx": c_idx,
                        "file_hash": fhash,
                    })

            # Store parents (no embeddings needed — just text storage)
            parents_col.add(ids=parent_ids, documents=parent_docs, metadatas=parent_metas)

            # Embed and store children in batches
            for batch_start in range(0, len(all_child_texts), BATCH_SIZE):
                batch_texts = all_child_texts[batch_start : batch_start + BATCH_SIZE]
                batch_ids = all_child_ids[batch_start : batch_start + BATCH_SIZE]
                batch_metas = all_child_metas[batch_start : batch_start + BATCH_SIZE]

                embeddings = model.encode(batch_texts, show_progress_bar=False).tolist()
                children_col.add(ids=batch_ids, embeddings=embeddings, documents=batch_texts, metadatas=batch_metas)
                del embeddings
                gc.collect()

            total_parents += len(parent_ids)
            total_children += len(all_child_ids)

            # Free memory per-file
            del all_child_texts, all_child_ids, all_child_metas, parent_ids, parent_docs, parent_metas, words, parents, text
            gc.collect()

            # Check memory every 10 files — stop gracefully if low
            processed = len(added_files) + len(updated_files)
            if processed % 10 == 0:
                if not gc_and_check(100):
                    print(f"  Memory low after {processed} files. Stopping gracefully.")
                    print(f"  Remaining files will be indexed in next run.")
                    stopped_early = True
                    break

            # Stop after max-files if set
            if args.max_files > 0 and processed >= args.max_files:
                print(f"  Reached --max-files limit ({args.max_files}). Stopping.")
                print(f"  Remaining files will be indexed in next run.")
                stopped_early = True
                break

        # Detect orphaned files (indexed but no longer on disk under this root)
        # Only do this if we're not running in batch mode (--max-files not hit)
        if not stopped_early:
            all_indexed = parents_col.get(include=["metadatas"])
            if all_indexed["metadatas"]:
                indexed_paths = set(m["filepath"] for m in all_indexed["metadatas"] if "filepath" in m)
                orphaned = indexed_paths - all_disk_paths
                for orphan in orphaned:
                    parents_col.delete(where={"filepath": orphan})
                    children_col.delete(where={"filepath": orphan})
                    removed_files.append(orphan)

    # --- Summary Report ---
    total_files = len(added_files) + len(updated_files)
    print(f"\n{'='*60}")
    print(f"INDICIZZAZIONE COMPLETATA")
    print(f"{'='*60}")
    print(f"  Totale file scansionati: {len(added_files) + len(updated_files) + skipped_unchanged + len(removed_files)}")
    print(f"  Nuovi indicizzati:       {len(added_files)}")
    print(f"  Aggiornati:              {len(updated_files)}")
    print(f"  Rimossi (orfani):        {len(removed_files)}")
    print(f"  Immobili (unchanged):    {skipped_unchanged}")
    print(f"  Parent chunks:           {total_parents}")
    print(f"  Child chunks:            {total_children}")
    print(f"  Collection parents:      {parents_col.count()}")
    print(f"  Collection children:     {children_col.count()}")

    if added_files:
        print(f"\n  File aggiunti:")
        for f in added_files:
            print(f"    + {os.path.relpath(f, os.path.expanduser('~'))}")
    if updated_files:
        print(f"\n  File aggiornati:")
        for f in updated_files:
            print(f"    ~ {os.path.relpath(f, os.path.expanduser('~'))}")
    if removed_files:
        print(f"\n  File rimossi:")
        for f in removed_files:
            print(f"    - {os.path.relpath(f, os.path.expanduser('~'))}")
    print(f"{'='*60}")


if __name__ == "__main__":
    main()


path: scripts/monitor.py
content: #!/usr/bin/env python3
"""Monitor local RAG indexing progress, logs, and errors.

Usage:
    monitor.py              # Show current status
    monitor.py --watch      # Auto-refresh every 10s
    monitor.py --log        # Show last 50 log lines
    monitor.py --log -n 100 # Show last N log lines
    monitor.py --errors     # Show only errors from log
    monitor.py --git        # Show git history (checkpoints)
"""

import argparse
import os
import subprocess
import sys
import time

DB_DIR = os.path.expanduser("~/.local/share/local-rag/chromadb")
LOG_FILE = os.path.expanduser("~/.local/share/local-rag/index-batch.log")
LOCK_FILE = os.path.expanduser("~/.local/share/local-rag/index.lock")


def get_db_stats():
    """Get ChromaDB collection counts."""
    try:
        import chromadb
        c = chromadb.PersistentClient(path=DB_DIR)
        parents = c.get_or_create_collection("parents").count()
        children = c.get_or_create_collection("children").count()
        return parents, children
    except Exception as e:
        return None, str(e)


def get_process_info():
    """Check if indexing is running."""
    try:
        result = subprocess.run(
            ["ps", "aux"], capture_output=True, text=True, timeout=5
        )
        lines = [l for l in result.stdout.splitlines() if "index.py" in l and "grep" not in l]
        if lines:
            parts = lines[0].split()
            pid = parts[1]
            cpu = parts[2]
            mem = parts[3]
            rss_mb = int(parts[5]) / 1024
            elapsed = parts[9]
            return {"running": True, "pid": pid, "cpu": cpu, "mem": mem, "rss_mb": rss_mb, "elapsed": elapsed}
    except Exception:
        pass
    return {"running": False}


def get_system_mem():
    """Get system memory info."""
    try:
        with open("/proc/meminfo") as f:
            lines = f.readlines()
        info = {}
        for line in lines:
            parts = line.split()
            if "MemTotal" in line:
                info["total_mb"] = int(parts[1]) / 1024
            elif "MemAvailable" in line:
                info["available_mb"] = int(parts[1]) / 1024
            elif "SwapTotal" in line:
                info["swap_total_mb"] = int(parts[1]) / 1024
            elif "SwapFree" in line:
                info["swap_free_mb"] = int(parts[1]) / 1024
        return info
    except Exception:
        return {}


def get_git_log():
    """Get git log of checkpoints."""
    try:
        result = subprocess.run(
            ["git", "log", "--oneline", "-10"],
            capture_output=True, text=True, cwd=DB_DIR, timeout=5
        )
        return result.stdout.strip().splitlines()
    except Exception:
        return ["(git not available)"]


def get_disk_usage():
    """Get DB size on disk."""
    try:
        result = subprocess.run(
            ["du", "-sh", DB_DIR], capture_output=True, text=True, timeout=5
        )
        return result.stdout.strip().split()[0]
    except Exception:
        return "?"


def show_status():
    """Print current status."""
    proc = get_process_info()
    parents, children = get_db_stats()
    mem = get_system_mem()
    disk = get_disk_usage()

    print("╔══════════════════════════════════════════╗")
    print("║     📊 Local RAG Index Monitor           ║")
    print("╠══════════════════════════════════════════╣")

    # Process status
    if proc["running"]:
        print(f"║ 🔵 RUNNING  PID {proc['pid']}  CPU {proc['cpu']}%  MEM {proc['rss_mb']:.0f}MB")
    else:
        print(f"║ ⚪ IDLE")

    # DB stats
    if parents is not None:
        print(f"║ Parents: {parents:,}  Children: {children:,}")
    else:
        print(f"║ DB: {children}")

    # Disk
    print(f"║ Disk: {disk}")

    # Memory
    if mem:
        used = mem.get("total_mb", 0) - mem.get("available_mb", 0)
        pct = (used / mem["total_mb"] * 100) if mem["total_mb"] > 0 else 0
        print(f"║ RAM: {used:.0f}/{mem['total_mb']:.0f}MB ({pct:.0f}%)  Available: {mem.get('available_mb', 0):.0f}MB")
        swap_used = mem.get("swap_total_mb", 0) - mem.get("swap_free_mb", 0)
        print(f"║ Swap: {swap_used:.0f}/{mem.get('swap_total_mb', 0):.0f}MB")

    # Lock
    locked = "🔒 Locked" if os.path.exists(LOCK_FILE) else "🔓 No lock"
    print(f"║ {locked}")

    print("╚══════════════════════════════════════════╝")


def show_log(lines=50, errors_only=False):
    """Show log file contents."""
    if not os.path.exists(LOG_FILE):
        print("(no log file yet)")
        return

    with open(LOG_FILE) as f:
        all_lines = f.readlines()

    if errors_only:
        all_lines = [l for l in all_lines if "❌" in l or "ERROR" in l or "FAILED" in l or "OOM" in l or "SIGKILL" in l]

    for line in all_lines[-lines:]:
        print(line.rstrip())


def show_git():
    """Show git checkpoint history."""
    commits = get_git_log()
    print("Git checkpoints (newest first):")
    for c in commits:
        print(f"  {c}")


def main():
    parser = argparse.ArgumentParser(description="Monitor local RAG indexing")
    parser.add_argument("--watch", action="store_true", help="Auto-refresh every 10s")
    parser.add_argument("--log", action="store_true", help="Show log")
    parser.add_argument("-n", type=int, default=50, help="Number of log lines")
    parser.add_argument("--errors", action="store_true", help="Show only errors")
    parser.add_argument("--git", action="store_true", help="Show git checkpoints")
    args = parser.parse_args()

    if args.log:
        show_log(args.n, args.errors)
    elif args.git:
        show_git()
    elif args.watch:
        while True:
            os.system("clear")
            show_status()
            print()
            show_git()
            print("\nRefreshing in 10s... (Ctrl+C to stop)")
            try:
                time.sleep(10)
            except KeyboardInterrupt:
                break
    else:
        show_status()


if __name__ == "__main__":
    main()


path: scripts/query.py
content: #!/usr/bin/env python3
"""Query the local RAG store with parent-child retrieval + cross-encoder reranker.

Search strategy:
1. Embed query → find top child chunks (cosine similarity)
2. Free embedding model, load reranker → rerank children
3. Resolve each top child → its parent chunk for full context
4. Return deduplicated parents ranked by best child score

Usage:
    query.py "your question" [--top-k 20] [--top-n 3] [--json]
"""

import argparse
import gc
import json
import os
import sys
import time
from datetime import datetime

import chromadb
from sentence_transformers import SentenceTransformer
from sentence_transformers import CrossEncoder

EMBED_MODEL = [REDACTED_SECRET]
RERANKER_MODEL = [REDACTED_SECRET]
DB_DIR = os.path.expanduser("~/.local/share/local-rag/chromadb")
QUERY_LOG = os.path.expanduser("~/.local/share/local-rag/queries.log")
MAX_QUERY_LEN = 2000

# Singleton model cache — avoids reloading on repeated calls
_model_cache = {}


def get_embed_model():
    """Get or create cached embedding model."""
    if "embed" not in _model_cache:
        _model_cache["embed"] = SentenceTransformer(EMBED_MODEL)
    return _model_cache["embed"]


def log_query(question: str, n_results: int, elapsed: float, top_score: float | None):
    """Append query to log file for analytics."""
    os.makedirs(os.path.dirname(QUERY_LOG), exist_ok=True)
    entry = {
        "ts": datetime.now().isoformat(),
        "q": question[:80],  # Truncate for privacy
        "results": n_results,
        "elapsed_s": round(elapsed, 2),
        "top_score": top_score,
    }
    with open(QUERY_LOG, "a") as f:
        f.write(json.dumps(entry, ensure_ascii=False) + "\n")


def query(question: str, top_k: int = 20, top_n: int = 3, timeout: int = 120, as_json: bool = False):
    """Search children → rerank → resolve parents."""
    # Sanitize input
    question = question[:MAX_QUERY_LEN].strip()
    if not question:
        return {"error": "Empty query"}

    t0 = time.time()

    # Step 1: Embed query (cached model)
    model = get_embed_model()
    q_embedding = model.encode([question], show_progress_bar=False).tolist()

    # Do NOT free model — keep cached for next query

    # Open DB
    client = chromadb.PersistentClient(path=DB_DIR)
    try:
        children_col = client.get_collection("children")
        parents_col = client.get_collection("parents")
    except Exception:
        return {"error": "No indexed data found. Run index.py first."}

    if children_col.count() == 0:
        return {"error": "Collection is empty. Run index.py to index your files."}

    # Vector search on children
    results = children_col.query(
        query_embeddings=q_embedding,
        n_results=min(top_k, children_col.count()),
        include=["documents", "metadatas", "distances"],
    )

    if not results["documents"][0]:
        return {"error": "No results found."}

    children = [
        {"text": doc, "meta": meta}
        for doc, meta in zip(results["documents"][0], results["metadatas"][0])
    ]

    # Step 2: Rerank with cross-encoder (cached for reuse)
    try:
        if "reranker" not in _model_cache:
            _model_cache["reranker"] = CrossEncoder(RERANKER_MODEL)
        reranker = _model_cache["reranker"]
        pairs = [[question, c["text"]] for c in children]
        scores = reranker.predict(pairs)

        for c, score in zip(children, scores):
            c["score"] = float(score)

        children.sort(key=lambda x: x["score"], reverse=True)
    except Exception as e:
        print(f"Warning: Reranker failed ({e}), using vector search order", file=sys.stderr)
        for c in children:
            c["score"] = 1.0

    # Step 3: Resolve parents (deduplicate by parent_idx per filepath)
    seen_parents = set()
    ranked_results = []

    for child in children:
        filepath = child["meta"]["filepath"]
        p_idx = child["meta"]["parent_idx"]
        parent_key = (filepath, p_idx)

        if parent_key in seen_parents:
            continue
        seen_parents.add(parent_key)

        parent_results = parents_col.get(
            where={"$and": [{"filepath": filepath}, {"parent_idx": p_idx}]},
            include=["documents"],
        )
        parent_text = parent_results["documents"][0] if parent_results["documents"] else child["text"]

        ranked_results.append({
            "score": round(child["score"], 4),
            "filepath": filepath,
            "parent_idx": p_idx,
            "parent_text": parent_text[:2000],
            "matching_child": child["text"][:500],
        })

        if len(ranked_results) >= top_n:
            break

    elapsed = time.time() - t0
    if elapsed > timeout:
        return {"error": f"Query timed out after {timeout}s"}

    top_score = ranked_results[0]["score"] if ranked_results else None
    log_query(question, len(ranked_results), elapsed, top_score)

    return {
        "question": question,
        "total_candidates": len(children),
        "results": ranked_results,
    }


def main():
    parser = argparse.ArgumentParser(description="Query local RAG with parent-child retrieval")
    parser.add_argument("question", help="Your question")
    parser.add_argument("--top-k", type=int, default=20, help="Initial child candidates")
    parser.add_argument("--top-n", type=int, default=3, help="Final parent results")
    parser.add_argument("--json", action="store_true", help="Output as JSON")
    parser.add_argument("--timeout", type=int, default=120, help="Max seconds per query")
    args = parser.parse_args()

    result = query(args.question, args.top_k, args.top_n, args.timeout)

    if "error" in result:
        print(f"ERROR: {result['error']}", file=sys.stderr)
        sys.exit(1)

    if args.json:
        print(json.dumps(result, indent=2, ensure_ascii=False))
    else:
        print(f"Question: {result['question']}")
        print(f"Candidates searched: {result['total_candidates']}\n")
        for i, r in enumerate(result["results"], 1):
            print(f"{'='*60}")
            print(f"Result {i} (score: {r['score']})")
            print(f"File: {r['filepath']} (parent chunk {r['parent_idx']})")
            print(f"\nMatched snippet:\n  {r['matching_child']}")
            print(f"\nFull context:\n{r['parent_text']}\n")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)