Skill: jameseball/enhanced-memory (version 1.0.0)

--- name: enhanced-memory description: Enhanced memory search with hybrid vector+keyword scoring, temporal routing, filepath scoring, adaptive weighting, pseudo-relevance feedback, salience scoring, and knowledge graph cross-references. Replaces the default memory search with a 4-signal fusion retrieval system. Use when searching memories, indexing memory files, building cross-references, or scoring memory salience. Requires Ollama with nomic-embed-text model. --- # Enhanced Memory Drop-in enhancement for OpenClaw's memory system. Replaces flat vector search with a 4-signal hybrid retrieval pipeline that achieved **0.782 MRR** (vs ~0.45 baseline vector-only). ## Setup ```bash # Install Ollama and pull the embedding model ollama pull nomic-embed-text # Index your memory files (run from workspace root) python3 skills/enhanced-memory/scripts/embed_memories.py # Optional: build cross-reference graph python3 skills/enhanced-memory/scripts/crossref_memories.py build ``` Re-run `embed_memories.py` whenever memory files change significantly. ## Scripts ### `scripts/search_memory.py` — Primary Search Hybrid 4-signal retrieval with automatic adaptation: ```bash python3 skills/enhanced-memory/scripts/search_memory.py "query" [top_n] ``` **Signals fused:** 1. **Vector similarity** (0.4) — cosine similarity via nomic-embed-text embeddings 2. **Keyword matching** (0.25) — query term overlap with chunk text 3. **Header matching** (0.1) — query terms in section headers 4. **Filepath scoring** (0.25) — query terms matching file/directory names **Automatic behaviors:** - **Temporal routing** — date references ("yesterday", "Feb 8", "last Monday") get 3x boost on matching files - **Adaptive weighting** — when keyword overlap is low, shifts to 85% vector weight - **Pseudo-relevance feedback (PRF)** — when top score < 0.45, expands query with terms from initial results and re-scores ### `scripts/enhanced_memory_search.py` — JSON-Compatible Search Same pipeline with JSON output format compatible with OpenClaw's memory_search tool: ```bash python3 skills/enhanced-memory/scripts/enhanced_memory_search.py --json "query" ``` Returns `{results: [{path, startLine, endLine, score, snippet, header}], ...}`. ### `scripts/embed_memories.py` — Indexing Chunks all `.md` files in `memory/` plus core workspace files (MEMORY.md, AGENTS.md, etc.) by markdown headers and embeds them: ```bash python3 skills/enhanced-memory/scripts/embed_memories.py ``` Outputs `memory/vectors.json`. Batches embeddings in groups of 20, truncates chunks to 2000 chars. ### `scripts/memory_salience.py` — Salience Scoring Surfaces stale/important memory items for heartbeat self-prompting: ```bash python3 skills/enhanced-memory/scripts/memory_salience.py # Human-readable prompts python3 skills/enhanced-memory/scripts/memory_salience.py --json # Programmatic output python3 skills/enhanced-memory/scripts/memory_salience.py --top 5 # More items ``` Scores `importance × staleness` considering: file type (topic > core > daily), size, access frequency, and query gap correlation. ### `scripts/crossref_memories.py` — Knowledge Graph Builds cross-reference links between memory chunks using embedding similarity: ```bash python3 skills/enhanced-memory/scripts/crossref_memories.py build # Build index python3 skills/enhanced-memory/scripts/crossref_memories.py show <file> # Show refs for file python3 skills/enhanced-memory/scripts/crossref_memories.py graph # Graph statistics ``` Uses file-representative approach (top 5 chunks per file) to reduce O(n²) to manageable comparisons. Threshold: 0.75 cosine similarity. ## Configuration All tunable constants are at the top of each script. Key parameters: | Parameter | Default | Script | Purpose | |-----------|---------|--------|---------| | `VECTOR_WEIGHT` | 0.4 | search_memory.py | Weight for vector similarity | | `KEYWORD_WEIGHT` | 0.25 | search_memory.py | Weight for keyword overlap | | `FILEPATH_WEIGHT` | 0.25 | search_memory.py | Weight for filepath matching | | `TEMPORAL_BOOST` | 3.0 | search_memory.py | Multiplier for date-matching files | | `PRF_THRESHOLD` | 0.45 | search_memory.py | Score below which PRF activates | | `SIMILARITY_THRESHOLD` | 0.75 | crossref_memories.py | Min similarity for cross-ref links | | `MODEL` | nomic-embed-text | all | Ollama embedding model | To use a different embedding model (e.g., `mxbai-embed-large`), change `MODEL` in each script and re-run `embed_memories.py`. ## Integration To replace the default memory search, point your agent's search tool at these scripts. The scripts expect: - `memory/` directory relative to workspace root containing `.md` files - `memory/vectors.json` (created by `embed_memories.py`) - Ollama running locally on port 11434 All scripts use only Python stdlib + Ollama HTTP API. No pip dependencies.

path: scripts/crossref_memories.py
content: #!/usr/bin/env python3
"""Build cross-reference network between memory chunks.

Finds semantically related chunks across different files using embedding similarity.
Uses file-representative approach to reduce O(n²) comparisons.

Usage:
    python3 crossref_memories.py build          # Build cross-reference index
    python3 crossref_memories.py show <file>    # Show references for a file
    python3 crossref_memories.py graph          # Print graph statistics

Environment variables:
    MEMORY_DIR - Override memory directory path
"""

import json, math, os, sys
from collections import defaultdict
from pathlib import Path

MEMORY_DIR = Path(os.environ.get('MEMORY_DIR', Path(__file__).resolve().parent.parent.parent.parent / 'memory'))
VECTORS_FILE = MEMORY_DIR / 'vectors.json'
CROSSREF_FILE = MEMORY_DIR / 'crossrefs.json'

SIMILARITY_THRESHOLD = 0.75
MAX_REPS_PER_FILE = 5
MAX_REFS_PER_CHUNK = 5


def load_vectors():
    with open(VECTORS_FILE) as f:
        return [c for c in json.load(f) if 'embedding' in c]


def cosine_sim(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(x * x for x in b))
    return dot / (na * nb) if na and nb else 0.0


def build_crossrefs():
    """Build cross-reference index between chunks in different files."""
    chunks = load_vectors()
    print(f'Loaded {len(chunks)} chunks')

    by_file = defaultdict(list)
    for i, c in enumerate(chunks):
        by_file[c['file']].append(i)

    files = list(by_file.keys())
    print(f'Files: {len(files)}')

    # Pick representative chunks per file (longest text)
    reps = {}
    for f, indices in by_file.items():
        sorted_idx = sorted(indices, key=lambda i: -len(chunks[i].get('text', '')))
        reps[f] = sorted_idx[:MAX_REPS_PER_FILE]

    crossrefs = {}
    file_links = defaultdict(lambda: defaultdict(float))
    comparisons = 0

    for fi, file_a in enumerate(files):
        for idx_a in reps[file_a]:
            ca = chunks[idx_a]
            key_a = f'{ca["file"]}:{ca.get("line", 0)}'
            refs = []
            for file_b in files:
                if file_b == file_a:
                    continue
                for idx_b in reps[file_b]:
                    cb = chunks[idx_b]
                    sim = cosine_sim(ca['embedding'], cb['embedding'])
                    comparisons += 1
                    if sim >= SIMILARITY_THRESHOLD:
                        refs.append((f'{cb["file"]}:{cb.get("line", 0)}', sim, cb.get('header', ''), cb['file']))
                        file_links[file_a][file_b] = max(file_links[file_a][file_b], sim)
            refs.sort(key=lambda x: -x[1])
            if refs:
                crossrefs[key_a] = [{'target': r[0], 'sim': round(r[1], 3),
                                      'header': r[2], 'file': r[3]} for r in refs[:MAX_REFS_PER_CHUNK]]
        if (fi + 1) % 20 == 0:
            print(f'  Processed {fi + 1}/{len(files)} files...')

    file_graph = {}
    for fa, links in file_links.items():
        top = sorted(links.items(), key=lambda x: -x[1])[:10]
        file_graph[fa] = [{'file': fb, 'strength': round(s, 3)} for fb, s in top if s >= SIMILARITY_THRESHOLD]

    result = {
        'total_chunks': len(chunks),
        'total_crossrefs': sum(len(v) for v in crossrefs.values()),
        'chunks_with_refs': len(crossrefs),
        'chunk_refs': crossrefs,
        'file_graph': file_graph,
    }

    with open(CROSSREF_FILE, 'w') as f:
        json.dump(result, f, indent=2)

    print(f'\nCross-reference index built:')
    print(f'  Chunks with refs: {len(crossrefs)} / {len(chunks)}')
    print(f'  Total cross-refs: {result["total_crossrefs"]}')
    print(f'  Files in graph: {len(file_graph)}')
    print(f'  Comparisons: {comparisons:,}')


def show_file_refs(target):
    with open(CROSSREF_FILE) as f:
        data = json.load(f)

    print(f'\n=== Cross-references for: {target} ===\n')
    fg = data.get('file_graph', {})
    if target in fg:
        print('File-level connections:')
        for l in fg[target]:
            print(f'  → {l["file"]} (strength: {l["strength"]})')

    print(f'\nChunk-level references:')
    for key, refs in data.get('chunk_refs', {}).items():
        if key.startswith(target + ':'):
            print(f'\n  {key}:')
            for r in refs:
                print(f'    → {r["target"]} [{r["header"]}] (sim: {r["sim"]})')

    print(f'\nIncoming references:')
    count = 0
    for key, refs in data.get('chunk_refs', {}).items():
        for r in refs:
            if r['file'] == target:
                print(f'  ← {key} [{r["header"]}] (sim: {r["sim"]})')
                count += 1
                if count >= 20:
                    break
        if count >= 20:
            break


def show_graph_stats():
    with open(CROSSREF_FILE) as f:
        data = json.load(f)
    print(f'Total chunks: {data["total_chunks"]}')
    print(f'Chunks with refs: {data["chunks_with_refs"]}')
    print(f'Total cross-refs: {data["total_crossrefs"]}')
    fg = data.get('file_graph', {})
    print(f'\nFile graph ({len(fg)} files):')
    for f_name, links in sorted(fg.items(), key=lambda x: -len(x[1]))[:15]:
        avg = sum(l['strength'] for l in links) / len(links) if links else 0
        print(f'  {f_name[:40]:<42} {len(links):>3} connections (avg: {avg:.3f})')


def main():
    if len(sys.argv) < 2:
        print(__doc__)
        sys.exit(1)
    cmd = sys.argv[1]
    if cmd == 'build':
        build_crossrefs()
    elif cmd == 'show' and len(sys.argv) >= 3:
        show_file_refs(sys.argv[2])
    elif cmd == 'graph':
        show_graph_stats()
    else:
        print(__doc__)


if __name__ == '__main__':
    main()


path: scripts/embed_memories.py
content: #!/usr/bin/env python3
"""Embed memory files into vectors for semantic search.

Chunks markdown files by headers and embeds each chunk via Ollama.
Outputs memory/vectors.json.

Usage:
    python3 embed_memories.py [memory_dir]

Environment variables:
    MEMORY_DIR    - Override memory directory path
    OLLAMA_URL    - Override Ollama API URL (default: http://localhost:11434/api/embed)
    EMBED_MODEL   - Override embedding model (default: nomic-embed-text)
"""

import json, os, re, sys, urllib.request

MEMORY_DIR = os.environ.get('MEMORY_DIR', os.path.join(os.path.dirname(__file__), '..', '..', '..', 'memory'))
VECTORS_FILE = os.path.join(MEMORY_DIR, 'vectors.json')
OLLAMA_URL = os.environ.get('OLLAMA_URL', 'http://localhost:11434/api/embed')
MODEL = os.environ.get('EMBED_MODEL', 'nomic-embed-text')

# Core workspace files to also index (relative to workspace root)
CORE_FILES = ['MEMORY.md', 'AGENTS.md', 'USER.md', 'SOUL.md', 'research.md']


def get_md_files(memory_dir):
    """Collect all .md files from memory directory and core workspace files."""
    files = []
    for root, _, fnames in os.walk(memory_dir):
        for f in fnames:
            if f.endswith('.md'):
                files.append(os.path.join(root, f))

    workspace = os.path.abspath(os.path.join(memory_dir, '..'))
    for name in CORE_FILES:
        path = os.path.join(workspace, name)
        if os.path.exists(path):
            files.append(os.path.abspath(path))
    return files


def split_into_chunks(filepath, workspace_root):
    """Split a markdown file into chunks by headers."""
    with open(filepath) as f:
        content = f.read()

    chunks = []
    header = os.path.basename(filepath)
    lines = []
    start = 1

    for i, line in enumerate(content.split('\n'), 1):
        if re.match(r'^#{1,4}\s', line):
            if lines:
                text = '\n'.join(lines).strip()
                if text and len(text) > 20:
                    chunks.append({
                        'file': os.path.relpath(filepath, workspace_root),
                        'header': header,
                        'line': start,
                        'text': text,
                    })
            header = line.lstrip('#').strip()
            lines = [line]
            start = i
        else:
            lines.append(line)

    if lines:
        text = '\n'.join(lines).strip()
        if text and len(text) > 20:
            chunks.append({
                'file': os.path.relpath(filepath, workspace_root),
                'header': header,
                'line': start,
                'text': text,
            })
    return chunks


def embed_batch(texts):
    """Get embeddings for a batch of texts from Ollama."""
    data = json.dumps({'model': MODEL, 'input': texts}).encode()
    req = urllib.request.Request(OLLAMA_URL, data=data, headers={'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.loads(resp.read())['embeddings']


def main():
    memory_dir = sys.argv[1] if len(sys.argv) > 1 else MEMORY_DIR
    memory_dir = os.path.abspath(memory_dir)
    workspace_root = os.path.abspath(os.path.join(memory_dir, '..'))

    files = get_md_files(memory_dir)
    all_chunks = []
    for f in files:
        all_chunks.extend(split_into_chunks(f, workspace_root))

    if not all_chunks:
        print('No chunks found.')
        return

    print(f'Embedding {len(all_chunks)} chunks from {len(files)} files...')

    for i in range(0, len(all_chunks), 20):
        batch = all_chunks[i:i + 20]
        texts = [c['text'][:2000] for c in batch]
        vectors = embed_batch(texts)
        for chunk, vec in zip(batch, vectors):
            chunk['embedding'] = vec
        done = min(i + 20, len(all_chunks))
        print(f'  {done}/{len(all_chunks)} chunks embedded')

    vectors_path = os.path.join(memory_dir, 'vectors.json')
    with open(vectors_path, 'w') as f:
        json.dump(all_chunks, f, indent=None)

    print(f'Saved {len(all_chunks)} vectors to {vectors_path}')


if __name__ == '__main__':
    main()


path: scripts/memory_salience.py
content: #!/usr/bin/env python3
"""Memory salience scorer for heartbeat self-prompting.

Surfaces important-but-stale memory items. Scores: importance × staleness.
Factors: file type, size, access frequency, query gap correlation.

Usage:
    python3 memory_salience.py              # Top 3 human-readable prompts
    python3 memory_salience.py --json       # JSON output
    python3 memory_salience.py --top 5      # More items

Environment variables:
    MEMORY_DIR - Override memory directory path
"""

import json, os, sys, argparse
from datetime import datetime
from pathlib import Path

WORKSPACE = Path(os.environ.get('MEMORY_DIR', Path(__file__).resolve().parent.parent.parent.parent)).resolve()
MEMORY_DIR = WORKSPACE / 'memory'
ACCESS_LOG = MEMORY_DIR / 'access_log.json'
QUERY_LOG = MEMORY_DIR / 'query_log.json'
NOW = datetime.now()


def load_json(path, default=None):
    if default is None:
        default = {}
    if path.exists():
        try:
            with open(path) as f:
                return json.load(f)
        except Exception:
            return default
    return default


def days_since_modified(path):
    try:
        return max(0, (NOW - datetime.fromtimestamp(os.path.getmtime(path))).days)
    except Exception:
        return 0


def get_memory_items():
    """Collect all scoreable memory items."""
    items = []

    # Topic files
    topics_dir = MEMORY_DIR / 'topics'
    if topics_dir.exists():
        for p in topics_dir.glob('*.md'):
            items.append({
                'path': str(p.relative_to(WORKSPACE)),
                'type': 'topic',
                'name': p.stem.replace('-', ' ').replace('_', ' '),
                'age_days': days_since_modified(p),
                'size': p.stat().st_size,
            })

    # Daily notes
    for p in MEMORY_DIR.glob('20??-??-??.md'):
        items.append({
            'path': str(p.relative_to(WORKSPACE)),
            'type': 'daily',
            'name': p.stem,
            'age_days': days_since_modified(p),
            'size': p.stat().st_size,
        })

    # MEMORY.md
    mem = WORKSPACE / 'MEMORY.md'
    if mem.exists():
        items.append({
            'path': 'MEMORY.md',
            'type': 'core',
            'name': 'Long-term memory',
            'age_days': days_since_modified(mem),
            'size': mem.stat().st_size,
        })

    return items


def get_query_gaps(query_log):
    """Find recent queries with low scores (potential knowledge gaps)."""
    if not query_log:
        return []
    return [e for e in query_log[-50:] if (e.get('top_score') or 0) < 0.4]


def score_salience(items, access_log, query_gaps):
    """Score each item: salience = importance × staleness."""
    scored = []
    for item in items:
        base = {'topic': 0.7, 'core': 0.8, 'daily': 0.4}[item['type']]
        size_bonus = min(0.1, item['size'] / 50000)
        accesses = access_log.get(item['path'], 0)
        access_penalty = min(0.2, accesses * 0.03)

        gap_bonus = 0
        for gap in query_gaps:
            if any(word in item['name'].lower() for word in gap['query'].lower().split()):
                gap_bonus = 0.15
                break

        importance = min(1.0, base + size_bonus - access_penalty + gap_bonus)
        age = item['age_days']
        staleness = min(1.0, age / (7 if item['type'] == 'daily' else 5))
        salience = importance * staleness

        scored.append({**item, 'importance': round(importance, 3),
                       'staleness': round(staleness, 3), 'salience': round(salience, 3),
                       'accesses': accesses})

    scored.sort(key=lambda x: -x['salience'])
    return scored


def generate_prompts(scored, top_n=3):
    """Generate natural-language prompts from top salient items."""
    prompts = []
    for item in scored[:top_n]:
        if item['salience'] < 0.05:
            continue
        if item['type'] == 'topic':
            if item['age_days'] > 3:
                prompts.append(f"📌 Topic '{item['name']}' hasn't been updated in {item['age_days']} days.")
            elif item['accesses'] == 0:
                prompts.append(f"📌 Topic '{item['name']}' was never accessed by search.")
        elif item['type'] == 'daily':
            prompts.append(f"📝 Daily notes from {item['name']} haven't been reviewed for MEMORY.md.")
        elif item['type'] == 'core' and item['age_days'] > 2:
            prompts.append(f"🧠 MEMORY.md last updated {item['age_days']} days ago.")
    return prompts


def main():
    parser = argparse.ArgumentParser(description='Memory salience scorer')
    parser.add_argument('--top', type=int, default=3)
    parser.add_argument('--json', action='store_true')
    args = parser.parse_args()

    access_log = load_json(ACCESS_LOG, {})
    query_log = load_json(QUERY_LOG, [])
    items = get_memory_items()
    scored = score_salience(items, access_log, get_query_gaps(query_log))

    if args.json:
        print(json.dumps({
            'timestamp': NOW.isoformat(),
            'items': scored[:args.top],
            'prompts': generate_prompts(scored, args.top),
            'total_items': len(items),
        }, indent=2))
    else:
        prompts = generate_prompts(scored, args.top)
        for p in prompts:
            print(p)
        if not prompts:
            print("All memory items are fresh. Nothing to surface.")


if __name__ == '__main__':
    main()


path: scripts/search_memory.py
content: #!/usr/bin/env python3
"""Hybrid 4-signal memory search with temporal routing, adaptive weighting, and PRF.

Fusion signals: vector similarity + keyword matching + header matching + filepath scoring.
Automatic behaviors: temporal routing, adaptive weighting, pseudo-relevance feedback.

Research results: 0.782 MRR on 68-query test suite (vs ~0.45 vector-only baseline).

Usage:
    python3 search_memory.py "query" [top_n]
"""

import json, math, os, re, sys, urllib.request
from datetime import datetime, timedelta
from collections import Counter

# --- Configuration ---
# Paths default to workspace layout: memory/ relative to script's grandparent.
# Override MEMORY_DIR env var to change.
MEMORY_DIR = os.environ.get('MEMORY_DIR', os.path.join(os.path.dirname(__file__), '..', '..', '..', 'memory'))
VECTORS_FILE = os.path.join(MEMORY_DIR, 'vectors.json')
OLLAMA_URL = os.environ.get('OLLAMA_URL', 'http://localhost:11434/api/embed')
MODEL = os.environ.get('EMBED_MODEL', 'nomic-embed-text')

# Fusion weights (tuned via 12 research sprints)
VECTOR_WEIGHT = 0.4
KEYWORD_WEIGHT = 0.25
HEADER_WEIGHT = 0.1
FILEPATH_WEIGHT = 0.25
TEMPORAL_BOOST = 3.0

# Adaptive weighting: when keyword overlap is very low, shift to vector-heavy
ADAPTIVE_KW_THRESHOLD = 0.1
ADAPTIVE_VECTOR_WEIGHT = 0.85
ADAPTIVE_KEYWORD_WEIGHT = 0.05
ADAPTIVE_HEADER_WEIGHT = 0.05
ADAPTIVE_FILEPATH_WEIGHT = 0.05

# Pseudo-relevance feedback
PRF_THRESHOLD = 0.45

# --- Stopwords (shared across scoring functions) ---
STOPWORDS = frozenset({
    'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been',
    'do', 'does', 'did', 'has', 'have', 'had', 'will', 'would',
    'can', 'could', 'should', 'may', 'might', 'shall', 'must',
    'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by', 'from',
    'and', 'or', 'not', 'no', 'but', 'if', 'so', 'as', 'than',
    'that', 'this', 'it', 'its', 'my', 'your', 'his', 'her',
    'we', 'they', 'them', 'what', 'when', 'where', 'how', 'who',
    'which', 'why', 'about', 'any', 'all', 'some'
})

# --- Temporal routing ---
MONTH_MAP = {
    'january': 1, 'jan': 1, 'february': 2, 'feb': 2, 'march': 3, 'mar': 3,
    'april': 4, 'apr': 4, 'may': 5, 'june': 6, 'jun': 6, 'july': 7, 'jul': 7,
    'august': 8, 'aug': 8, 'september': 9, 'sep': 9, 'sept': 9,
    'october': 10, 'oct': 10, 'november': 11, 'nov': 11, 'december': 12, 'dec': 12,
}
DAY_MAP = {
    'monday': 0, 'tuesday': 1, 'wednesday': 2, 'thursday': 3,
    'friday': 4, 'saturday': 5, 'sunday': 6
}
NUM_WORDS = {'two': 2, 'three': 3, 'four': 4, 'five': 5, 'six': 6, 'seven': 7}


def extract_dates(query, ref=None):
    """Extract date references from a query string. Returns list of 'YYYY-MM-DD' strings."""
    if ref is None:
        ref = datetime.now()
    q = query.lower()
    dates = set()

    # ISO dates (2026-02-10)
    for m in re.finditer(r'(\d{4})-(\d{2})-(\d{2})', query):
        dates.add(f'{m.group(1)}-{m.group(2)}-{m.group(3)}')

    # "February 8th", "Feb 8", "February 8, 2026"
    mp = '|'.join(MONTH_MAP.keys())
    for m in re.finditer(rf'({mp})\s+(\d{{1,2}})(?:st|nd|rd|th)?(?:\s*,?\s*(\d{{4}}))?', q):
        mo = MONTH_MAP[m.group(1)]
        day = int(m.group(2))
        yr = int(m.group(3)) if m.group(3) else ref.year
        dates.add(f'{yr:04d}-{mo:02d}-{day:02d}')

    # "8th February"
    for m in re.finditer(rf'(\d{{1,2}})(?:st|nd|rd|th)?\s+({mp})(?:\s+(\d{{4}}))?', q):
        day = int(m.group(1))
        mo = MONTH_MAP[m.group(2)]
        yr = int(m.group(3)) if m.group(3) else ref.year
        dates.add(f'{yr:04d}-{mo:02d}-{day:02d}')

    # Relative dates
    if 'today' in q:
        dates.add(ref.strftime('%Y-%m-%d'))
    if 'yesterday' in q:
        dates.add((ref - timedelta(days=1)).strftime('%Y-%m-%d'))
    if 'day before yesterday' in q:
        dates.add((ref - timedelta(days=2)).strftime('%Y-%m-%d'))

    # "N days ago", "two days ago"
    for m in re.finditer(r'(\d+)\s+days?\s+ago', q):
        dates.add((ref - timedelta(days=int(m.group(1)))).strftime('%Y-%m-%d'))
    for word, n in NUM_WORDS.items():
        if f'{word} days ago' in q or f'{word} day ago' in q:
            dates.add((ref - timedelta(days=n)).strftime('%Y-%m-%d'))

    if 'last week' in q:
        dates.add((ref - timedelta(days=7)).strftime('%Y-%m-%d'))

    for dn, dnum in DAY_MAP.items():
        if f'last {dn}' in q:
            back = (ref.weekday() - dnum) % 7
            if back == 0:
                back = 7
            dates.add((ref - timedelta(days=back)).strftime('%Y-%m-%d'))

    return list(dates)


def date_file_match(file_path, date_strings):
    """Check if a file path contains any of the extracted date strings."""
    for ds in date_strings:
        if ds in file_path:
            return True
    return False


# --- Embedding ---
def embed(text):
    """Get embedding vector from Ollama."""
    data = json.dumps({'model': MODEL, 'input': [text]}).encode()
    req = urllib.request.Request(OLLAMA_URL, data=data, headers={'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read())['embeddings'][0]


def cosine_sim(a, b):
    """Cosine similarity between two vectors."""
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(x * x for x in b))
    return dot / (na * nb) if na and nb else 0.0


# --- Scoring functions ---
def _query_words(text):
    """Extract meaningful query words (lowercase, >2 chars, no stopwords)."""
    return {w for w in re.findall(r'\w+', text.lower()) if len(w) > 2 and w not in STOPWORDS}


def keyword_score(query, text):
    """Fraction of query words found in text."""
    words = _query_words(query)
    if not words:
        return 0.0
    text_lower = text.lower()
    return sum(1 for w in words if w in text_lower) / len(words)


def filepath_score(query, filepath):
    """Fraction of query words found in the file path (with separators normalized)."""
    words = _query_words(query)
    if not words:
        return 0.0
    fp = filepath.lower().replace('-', ' ').replace('_', ' ').replace('/', ' ')
    return sum(1 for w in words if w in fp) / len(words)


def _extract_expansion_terms(top_chunks, query, max_terms=5):
    """Extract key terms from top results for pseudo-relevance feedback."""
    query_words = set(re.findall(r'\w+', query.lower()))
    extra_stops = STOPWORDS | {'also', 'just', 'more', 'other', 'new', 'like', 'then', 'now', 'only', 'very'}
    counts = Counter()
    for chunk in top_chunks[:3]:
        words = re.findall(r'\w+', chunk['text'].lower())
        fp_words = re.findall(r'\w+', chunk['file'].lower().replace('-', ' ').replace('_', ' '))
        for w in words + fp_words:
            if len(w) > 3 and w not in extra_stops and w not in query_words and not w.isdigit():
                counts[w] += 1
    return [t for t, c in counts.most_common(max_terms) if c >= 2]


def _score_chunks(chunks, query, q_vec, date_refs, has_temporal):
    """Score all chunks. Returns sorted list of (final_score, vec_sim, kw_combined, chunk)."""
    raw = []
    for c in chunks:
        if 'embedding' not in c:
            continue
        raw.append((
            cosine_sim(q_vec, c['embedding']),
            keyword_score(query, c['text']),
            keyword_score(query, c.get('header', '')),
            filepath_score(query, c['file']),
            c,
        ))

    best_kw = max((kw for _, kw, _, _, _ in raw), default=0) if raw else 0
    if best_kw < ADAPTIVE_KW_THRESHOLD:
        vw, kw_w, hw, fw = ADAPTIVE_VECTOR_WEIGHT, ADAPTIVE_KEYWORD_WEIGHT, ADAPTIVE_HEADER_WEIGHT, ADAPTIVE_FILEPATH_WEIGHT
        adaptive = True
    else:
        vw, kw_w, hw, fw = VECTOR_WEIGHT, KEYWORD_WEIGHT, HEADER_WEIGHT, FILEPATH_WEIGHT
        adaptive = False

    scored = []
    for vec_sim, kw_sim, header_kw, fp_sim, c in raw:
        final = vw * vec_sim + kw_w * kw_sim + hw * header_kw + fw * fp_sim
        if has_temporal and date_file_match(c['file'], date_refs):
            final *= TEMPORAL_BOOST
        scored.append((final, vec_sim, kw_sim + 0.3 * header_kw, c))

    scored.sort(key=lambda x: -x[0])
    return scored, adaptive


# --- Main ---
def search(query, top_n=5):
    """Run enhanced search. Returns list of (score, vec_sim, kw_score, chunk) tuples."""
    if not os.path.exists(VECTORS_FILE):
        print('No vectors.json found. Run embed_memories.py first.', file=sys.stderr)
        return []

    with open(VECTORS_FILE) as f:
        chunks = json.load(f)

    q_vec = embed(query)
    date_refs = extract_dates(query)
    has_temporal = len(date_refs) > 0

    scored, adaptive = _score_chunks(chunks, query, q_vec, date_refs, has_temporal)

    # Pseudo-relevance feedback
    prf_used = False
    if scored and scored[0][0] < PRF_THRESHOLD and not has_temporal:
        expansion = _extract_expansion_terms([s[3] for s in scored[:3]], query)
        if expansion:
            expanded_vec = embed(query + ' ' + ' '.join(expansion))
            scored2, _ = _score_chunks(chunks, query + ' ' + ' '.join(expansion), expanded_vec, date_refs, has_temporal)
            chunk_best = {}
            for s in scored + scored2:
                key = (s[3]['file'], s[3].get('line', 0))
                if key not in chunk_best or s[0] > chunk_best[key][0]:
                    chunk_best[key] = s
            scored = sorted(chunk_best.values(), key=lambda x: -x[0])
            prf_used = True

    return scored[:top_n], {
        'temporal': date_refs if has_temporal else None,
        'adaptive': adaptive,
        'prf': prf_used,
    }


def main():
    if len(sys.argv) < 2:
        print('Usage: search_memory.py "query" [top_n]')
        sys.exit(1)

    query = sys.argv[1]
    top_n = int(sys.argv[2]) if len(sys.argv) > 2 else 5
    results, meta = search(query, top_n)

    if meta.get('temporal'):
        print(f'[temporal routing: dates={meta["temporal"]}]')
    if meta.get('adaptive'):
        print(f'[adaptive weighting: low keyword overlap, using vector-heavy weights]')
    if meta.get('prf'):
        print(f'[PRF expansion applied]')

    for final, vec, kw, c in results:
        print(f'\n--- {c["file"]}:{c.get("line", "?")} [{c.get("header", "")}] (score: {final:.3f} v:{vec:.3f} k:{kw:.3f})')
        print(c['text'][:200])


if __name__ == '__main__':
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)