Skill: misolith/agent-memory-journal (version 0.2.3)

# Skill: Agent Memory Journal Manage and retrieve long-term memory using a 3-tier structure (Hot, Warm, Cold). Primary integration surface: the Python `Journal` API. The CLI is a convenience layer for shell and cron workflows. ## Tools - `agent-memory-journal note <text>`: Add episodic note. - `agent-memory-journal remember <text> --category <cat>`: Add core memory. - `agent-memory-journal search --query <q>`: Search memory. - `agent-memory-journal forget <id>`: Supersede a memory. - `agent-memory-journal ingest`: Run promotion and rebuild cycle. ## Core Categories - `decision`: Strategic choices. - `constraint`: Hard rules or limitations. - `gotcha`: Lessons learned or bugs. - `preference`: User or agent preferences. - `capability`: New skills or tools. ## Guidelines 1. **Episodic first**: Use `note` for observations. 2. **Explicit curation**: Use `remember` for facts that must persist. 3. **Atomic pinning**: Use `--pinned` for items that MUST be in the configured hot file. 4. **Verification**: Run `doctor` periodically to ensure integrity. ## Agent installation See `docs/agent_install.md` for: - workspace installation - CLI symlink setup - startup contract for agents - hot promotion target override via `.memory/config.json`

path: agent_memory/bm25.py
content: from __future__ import annotations

import math
from collections import Counter

import json
import hashlib
from pathlib import Path

from .normalize import normalize_claim

NORMALIZER_VERSION = 2


class BM25Index:
    def __init__(self, docs: list[str], k1: float = 1.5, b: float = 0.75):
        self.docs = docs
        self.k1 = k1
        self.b = b
        self.doc_tokens = [normalize_claim(doc)[1] for doc in docs]
        self.doc_freqs = [Counter(tokens) for tokens in self.doc_tokens]
        self.doc_lens = [len(tokens) for tokens in self.doc_tokens]
        self.avgdl = sum(self.doc_lens) / len(self.doc_lens) if self.doc_lens else 0.0
        self.idf = self._build_idf()

    @classmethod
    def from_cache(cls, cache_path: Path, docs: list[str], k1: float = 1.5, b: float = 0.75):
        # Hash docs to see if cache is valid
        docs_hash = hashlib.sha256("\n".join(docs).encode("utf-8")).hexdigest()
        if cache_path.exists():
            try:
                data = json.loads(cache_path.read_text(encoding="utf-8"))
                if data.get("hash") == docs_hash and data.get("normalizer_version") == NORMALIZER_VERSION:
                    instance = cls.__new__(cls)
                    instance.docs = docs
                    instance.k1 = k1
                    instance.b = b
                    instance.doc_tokens = data["doc_tokens"]
                    instance.doc_freqs = [Counter(d) for d in data["doc_freqs"]]
                    instance.doc_lens = data["doc_lens"]
                    instance.avgdl = data["avgdl"]
                    instance.idf = data["idf"]
                    return instance
            except Exception:
                pass
        
        instance = cls(docs, k1, b)
        # Save to cache
        try:
            cache_path.parent.mkdir(parents=True, exist_ok=True)
            cache_path.write_text(json.dumps({
                "hash": docs_hash,
                "normalizer_version": NORMALIZER_VERSION,
                "doc_tokens": instance.doc_tokens,
                "doc_freqs": [dict(f) for f in instance.doc_freqs],
                "doc_lens": instance.doc_lens,
                "avgdl": instance.avgdl,
                "idf": instance.idf
            }), encoding="utf-8")
        except Exception:
            pass
        return instance

    def _build_idf(self) -> dict[str, float]:
        df = Counter()
        for tokens in self.doc_tokens:
            for token in set(tokens):
                df[token] += 1
        total = len(self.doc_tokens)
        return {
            [REDACTED_SECRET] + (total - freq + 0.5) / (freq + 0.5))
            for token, freq in df.items()
        }

    def score(self, query: str) -> list[float]:
        q_tokens = normalize_claim(query)[1]
        scores: list[float] = []
        for freqs, doc_len in zip(self.doc_freqs, self.doc_lens):
            total = 0.0
            for token in q_tokens:
                if token not in freqs:
                    continue
                tf = freqs[token]
                idf = self.idf.get(token, 0.0)
                denom = tf + self.k1 * (1 - self.b + self.b * (doc_len / self.avgdl if self.avgdl else 0.0))
                total += idf * ((tf * (self.k1 + 1)) / denom)
            scores.append(total)
        return scores


path: agent_memory/ingest.py
content: from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

from .decay import archive_unpinned_core
from .hot import rebuild_agent_md
from .promote import promote_repeated_candidates
from .review import review_state
from .storage import init_memory_root


@dataclass
class IngestReport:
    promoted_count: int
    archived_count: int
    hot_written: int
    hot_skipped: int
    review_status: str
    review_hot_chars: int


def ingest_cycle(root: str | Path) -> IngestReport:
    paths = init_memory_root(root)
    promoted = promote_repeated_candidates(paths.root)
    decay = archive_unpinned_core(paths.root)
    hot = rebuild_agent_md(paths.root)
    review = review_state(paths.root)
    return IngestReport(
        promoted_count=len(promoted),
        archived_count=decay.archived,
        hot_written=int(hot['written']),
        hot_skipped=int(hot['skipped']),
        review_status=review.status,
        review_hot_chars=review.hot_chars,
    )


path: agent_memory/legacy.py
content: from __future__ import annotations

from pathlib import Path
from typing import Iterable

from .models import RecallResult
from .normalize import token_counter


class LegacyJournal:
    def __init__(self, root: str | Path = '.'):
        self.root = Path(root).expanduser().resolve()
        self.long_file = self.root / 'MEMORY.md'
        self.daily_dir = self.root / 'memory'

    def recall(self, query: str, k: int = 5, tier: str = 'all') -> list[RecallResult]:
        q = query.lower().strip()
        results: list[RecallResult] = []
        if tier in {'all', 'warm'} and self.long_file.exists():
            results.extend(self._scan_file(self.long_file, q, tier='warm'))
        if tier in {'all', 'cold'} and self.daily_dir.exists():
            for path in sorted(self.daily_dir.glob('*.md'), reverse=True):
                results.extend(self._scan_file(path, q, tier='cold'))
        results.sort(key=lambda item: (-item.score, item.path, item.line_no))
        return results[: max(1, k)]

    def note(self, text: str, category: str | None = None, importance: str = 'normal', source: str = 'agent') -> Path:
        self.daily_dir.mkdir(parents=True, exist_ok=True)
        from datetime import datetime
        path = self.daily_dir / f"{datetime.now().date()}.md"
        ts = datetime.now().strftime('%H:%M')
        suffix = ''
        if category:
            suffix += f' [category:{category}]'
        if importance != 'normal':
            suffix += f' [importance:{importance}]'
        with path.open('a', encoding='utf-8') as handle:
            handle.write(f"- {ts} {text.strip()}{suffix}\n")
        return path

    def _scan_file(self, path: Path, query: str, tier: str) -> Iterable[RecallResult]:
        for line_no, line in enumerate(path.read_text(encoding='utf-8', errors='ignore').splitlines(), start=1):
            lowered = line.lower()
            if not query:
                continue
            query_tokens = token_counter(query)
            line_tokens = token_counter(line)
            overlap = sum(min(line_tokens[t], query_tokens[t]) for t in query_tokens)
            substring_bonus = 1 if query in lowered else 0
            if overlap or substring_bonus:
                score = float(overlap + substring_bonus)
                yield RecallResult(text=line.strip(), path=str(path), line_no=line_no, score=score, tier=tier)


path: agent_memory/models.py
content: from __future__ import annotations

from dataclasses import dataclass, field
from typing import Any


@dataclass
class RecallResult:
    text: str
    path: str
    line_no: int
    score: float = 0.0
    tier: str = "cold"
    category: str | None = None
    refs: list[str] = field(default_factory=list)
    provenance: str | None = None
    status: str = "active"
    extra: dict[str, Any] = field(default_factory=dict)


path: agent_memory/decay.py
content: from __future__ import annotations

import re
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path

from .hot import _has_metadata_flag
from .storage import init_memory_root

CREATED_RE = re.compile(r'created:([^\s\]]+)')
LAST_SEEN_RE = re.compile(r'last_seen:([^\s\]]+)')


@dataclass
class DecayReport:
    archived: int
    retained: int
    archive_path: str


def _parse_iso_field(match: re.Match[str] | None) -> datetime | None:
    if not match:
        return None
    try:
        parsed = datetime.fromisoformat(match.group(1).replace('Z', '+00:00'))
    except ValueError:
        return None
    if parsed.tzinfo is None:
        parsed = parsed.replace(tzinfo=timezone.utc)
    return parsed.astimezone(timezone.utc)


def _last_activity_at(line: str) -> datetime | None:
    last_seen = _parse_iso_field(LAST_SEEN_RE.search(line))
    created = _parse_iso_field(CREATED_RE.search(line))
    if last_seen and created:
        return max(last_seen, created)
    return last_seen or created


def archive_unpinned_core(root: str | Path, max_age_days: int = 30, max_active_per_file: int = 50) -> DecayReport:
    paths = init_memory_root(root)
    archived = 0
    retained = 0
    cutoff = datetime.now(timezone.utc) - timedelta(days=max_age_days)
    for core_file in sorted(paths.core_dir.glob('*.md')):
        lines = core_file.read_text(encoding='utf-8', errors='ignore').splitlines()
        header = []
        bullets = []
        for line in lines:
            if line.strip().startswith('- '):
                bullets.append(line)
            else:
                header.append(line)
        pinned = [line for line in bullets if _has_metadata_flag(line, 'pinned:true')]
        unpinned = [line for line in bullets if not _has_metadata_flag(line, 'pinned:true')]
        stale = []
        fresh = []
        for line in unpinned:
            last_activity = _last_activity_at(line)
            if last_activity and last_activity < cutoff:
                stale.append(line)
            else:
                fresh.append(line)
        keep_count = max(0, max_active_per_file - len(pinned))
        keep_fresh = fresh[-keep_count:] if keep_count else []
        move_limit = fresh[:-keep_count] if keep_count else fresh[:]
        move_unpinned = stale + move_limit
        if move_unpinned:
            archive_target = paths.archive_core_dir / core_file.name
            if not archive_target.exists():
                archive_target.write_text(f'# Archived {core_file.stem.title()}\n\n', encoding='utf-8')
            with archive_target.open('a', encoding='utf-8') as handle:
                for line in move_unpinned:
                    handle.write(line + '\n')
            archived += len(move_unpinned)
        new_lines = header[:]
        if new_lines and new_lines[-1] != '':
            new_lines.append('')
        new_lines.extend(pinned + keep_fresh)
        core_file.write_text('\n'.join(new_lines).rstrip() + '\n', encoding='utf-8')
        retained += len(pinned) + len(keep_fresh)
    return DecayReport(archived=archived, retained=retained, archive_path=str(paths.archive_core_dir))


path: agent_memory/models_v2.py
content: from __future__ import annotations

from dataclasses import dataclass


@dataclass
class MemoryItem:
    id: str
    text: str
    category: str
    tier: str
    state: str
    source: str
    created_at: str = ''
    last_seen: str = ''
    pinned: bool = False
    supersedes: str | None = None


@dataclass
class CandidateItem:
    id: str
    text: str
    normalized_claim: str
    category: str | None
    source: str
    occurrences: int
    distinct_days: int
    score: float


path: agent_memory/normalize.py
content: from __future__ import annotations

import re
from collections import Counter

TOKEN_RE = re.compile(r"[A-Za-zÅÄÖåäö0-9-]{3,}")
STOP_WORDS = {
    "the", "and", "for", "with", "that", "this", "from", "into", "over", "under",
    "have", "has", "had", "was", "were", "are", "but", "not", "you", "your",
    "että", "tämä", "nämä", "sitä", "niitä", "kun", "joka", "jotka", "sekä", "vain",
    "vielä", "myös", "mutta", "kuin", "ovat", "oli", "olla", "voidaan", "joten",
}


def simple_stem(token: str) -> str:
    [REDACTED_SECRET])
    for suffix in (
        "ing", "ed", "es", "s", "ssa", "ssä", "sta", "stä", "lla", "llä", "lta", "ltä",
        "ksi", "tta", "ttä", "iin", "den", "jen", "nen", "n",
    ):
        if len(token) > len(suffix) + 2 and token.endswith(suffix):
            return token[: -len(suffix)]
    return token


def tokenize(text: str) -> list[str]:
    return [match.group(0).lower() for match in TOKEN_RE.finditer(text)]


def normalize_claim(text: str) -> tuple[str, list[str]]:
    # The first return value is a canonical, deduped string used for grouping/dedupe.
    # The second return value is the full ordered token multiset, which BM25 and
    # token-overlap scoring need so that repeated terms count more than once.
    clean_text = re.sub(r"\s+\[.*\]\s*$", "", text)
    tokens: list[str] = []
    for token in tokenize(clean_text):
        stemmed = simple_stem(token)
        if stemmed in STOP_WORDS:
            continue
        tokens.append(stemmed)
    canonical = " ".join(sorted(set(tokens)))
    return canonical, tokens


def jaccard_similarity(a: list[str], b: list[str]) -> float:
    sa, sb = set(a), set(b)
    if not sa and not sb:
        return 1.0
    if not sa or not sb:
        return 0.0
    return len(sa & sb) / len(sa | sb)


def claims_match(text_a: str, text_b: str, threshold: float = 0.72) -> bool:
    _na, tokens_a = normalize_claim(text_a)
    _nb, tokens_b = normalize_claim(text_b)
    return jaccard_similarity(tokens_a, tokens_b) >= threshold


def token_counter(text: str) -> Counter[str]:
    _norm, tokens = normalize_claim(text)
    return Counter(tokens)


path: agent_memory/core_recall.py
content: from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

import re
from .bm25 import BM25Index
import os
import shutil
import tempfile
from .storage import MemoryPaths, _refresh_manifest_entry, split_claim_and_metadata, utc_now_iso


@dataclass
class CoreRecallHit:
    text: str
    category: str
    path: str
    score: float
    line_no: int


def _stamp_line(line: str, timestamp: str) -> str:
    if 'last_seen:' in line:
        return re.sub(r'last_seen:[^\s\]]+', f'last_seen:{timestamp}', line)
    if line.endswith(']'):
        return line[:-1] + f' last_seen:{timestamp}]'
    return line + f' [last_seen:{timestamp}]'


def _bulk_update_last_seen(root: Path, hits: list['CoreRecallHit']) -> None:
    if not hits:
        return
    timestamp = utc_now_iso()
    by_path: dict[str, list[int]] = {}
    for hit in hits:
        by_path.setdefault(hit.path, []).append(hit.line_no)
    paths = MemoryPaths(root)
    for path_str, line_nos in by_path.items():
        target = Path(path_str)
        if not target.exists():
            continue
        lines = target.read_text(encoding='utf-8', errors='ignore').splitlines()
        touched = False
        for line_no in line_nos:
            if line_no < 1 or line_no > len(lines):
                continue
            new_line = _stamp_line(lines[line_no - 1], timestamp)
            if new_line != lines[line_no - 1]:
                lines[line_no - 1] = new_line
                touched = True
        if not touched:
            continue
        fd, temp_path = tempfile.mkstemp(dir=str(target.parent), prefix=target.name + '.tmp')
        try:
            with os.fdopen(fd, 'w', encoding='utf-8') as handle:
                handle.write('\n'.join(lines).rstrip() + '\n')
            shutil.move(temp_path, str(target))
        except Exception:
            if os.path.exists(temp_path):
                os.remove(temp_path)
            continue
        try:
            _refresh_manifest_entry(paths, target)
        except Exception:
            pass


def recall_core(root: str | Path, query: str, k: int = 5, update_last_seen: bool = True) -> list[CoreRecallHit]:
    root_path = Path(root).expanduser().resolve()
    core_dir = root_path / 'core'
    if not query.strip():
        return []
    if k < 0:
        raise ValueError('k must be >= 0')
    if k == 0 or not core_dir.exists():
        return []
    category_map = {
        'decisions': 'decision',
        'constraints': 'constraint',
        'gotchas': 'gotcha',
        'preferences': 'preference',
        'capabilities': 'capability',
    }
    docs = []
    meta = []
    for core_file in sorted(core_dir.glob('*.md')):
        category = category_map.get(core_file.stem, core_file.stem)
        for line_no, line in enumerate(core_file.read_text(encoding='utf-8', errors='ignore').splitlines(), start=1):
            stripped = line.strip()
            if not stripped.startswith('- '):
                continue
            claim, _metadata = split_claim_and_metadata(stripped)
            docs.append(claim)
            meta.append((category, str(core_file), line_no))
    if not docs:
        return []

    cache_path = root_path / 'index' / 'core_bm25.json'
    index = BM25Index.from_cache(cache_path, docs)
    scores = index.score(query)
    hits = [
        CoreRecallHit(text=doc, category=category, path=path, score=score, line_no=line_no)
        for doc, score, (category, path, line_no) in zip(docs, scores, meta)
        if score > 0
    ]
    hits.sort(key=lambda item: (-item.score, item.category, item.line_no))
    results = hits[:k]
    if update_last_seen:
        _bulk_update_last_seen(root_path, results)
    return results


path: agent_memory/security.py
content: from __future__ import annotations

import re
from pathlib import Path

ZERO_WIDTH_RE = re.compile(r"[\u200b\u200c\u200d\ufeff]")
CONTROL_RE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
BLOCKLIST = [
    re.compile(r"ignore\s+previous\s+instructions", re.IGNORECASE),
    re.compile(r"^system:\s*", re.IGNORECASE),
    re.compile(r"^developer:\s*", re.IGNORECASE),
    re.compile(r"^assistant:\s*", re.IGNORECASE),
    re.compile(r"you\s+are\s+now\s+", re.IGNORECASE),
    re.compile(r"reset\s+all\s+previous\s+instructions", re.IGNORECASE),
    re.compile(r"<\|system\|>", re.IGNORECASE),
    re.compile(r"```(?:bash|sh|shell)?", re.IGNORECASE),
]


def sanitize_text(text: str) -> str:
    cleaned = ZERO_WIDTH_RE.sub('', text)
    cleaned = CONTROL_RE.sub('', cleaned)
    return cleaned.strip()


def is_safe_memory_text(text: str) -> bool:
    cleaned = sanitize_text(text)
    if not cleaned:
        return False
    return not any(pattern.search(cleaned) for pattern in BLOCKLIST)


def require_safe_memory_text(text: str) -> str:
    cleaned = sanitize_text(text)
    if not cleaned:
        raise ValueError('empty memory text')
    for pattern in BLOCKLIST:
        if pattern.search(cleaned):
            raise ValueError(f"unsafe memory text: blocked by pattern {pattern.pattern}")
    return cleaned


def validate_hot_path(memory_root: str | Path, hot_path: str) -> Path:
    memory_root_path = Path(memory_root).expanduser().resolve()
    candidate = Path(hot_path)
    resolved = candidate.resolve() if candidate.is_absolute() else (memory_root_path / candidate).resolve()

    allowed_roots = {
        memory_root_path,
        memory_root_path.parent,
    }
    if not any(root == resolved or root in resolved.parents for root in allowed_roots):
        raise ValueError(f"invalid hot_path outside allowed roots: {hot_path}")
    return resolved


def sanitize_evidence_text(text: str) -> str:
    # Episodic/session log, preserves prompt-injection text as evidence.
    # Only invisible/control characters are stripped here.
    cleaned = sanitize_text(text)
    if not cleaned:
        raise ValueError('empty memory text')
    return cleaned


path: agent_memory/session.py
content: from __future__ import annotations

import re
from pathlib import Path

from .normalize import claims_match, token_counter
from .promote import Candidate, DEFAULT_MATCH_THRESHOLD, DEFAULT_OVERLAP_THRESHOLD
from .storage import VALID_CATEGORIES, init_memory_root

SESSION_LINE_RE = re.compile(r"^-\s+(\d{2}:\d{2})\s+(.*?)(?:\s+\[(.*)\])?$")
FIELD_RE = re.compile(r"(category|importance|source):([^\s\]]+)")


def _parse_metadata(blob: str | None) -> dict[str, str]:
    if not blob:
        return {}
    return {match.group(1): match.group(2) for match in FIELD_RE.finditer(blob)}


def _overlap_ratio(a: str, b: str) -> float:
    ca = token_counter(a)
    cb = token_counter(b)
    if not ca or not cb:
        return 0.0
    overlap = sum(min(ca[token], cb[token]) for token in ca if token in cb)
    denom = min(sum(ca.values()), sum(cb.values())) or 1
    return overlap / denom


def collect_session_candidates(root: str | Path, match_threshold: float = DEFAULT_MATCH_THRESHOLD, overlap_threshold: float = DEFAULT_OVERLAP_THRESHOLD) -> list[Candidate]:
    paths = init_memory_root(root)
    groups: list[dict] = []
    for path in sorted(paths.sessions_dir.glob('*.md')):
        session_id = path.stem
        for line_no, line in enumerate(path.read_text(encoding='utf-8', errors='ignore').splitlines(), start=1):
            match = SESSION_LINE_RE.match(line.strip())
            if not match:
                continue
            _hhmm, text, blob = match.groups()
            meta = _parse_metadata(blob)
            matched = None
            for group in groups:
                if claims_match(group['representative'], text, threshold=match_threshold) or _overlap_ratio(group['representative'], text) >= overlap_threshold:
                    matched = group
                    break
            if not matched:
                matched = {
                    'representative': text,
                    'category': meta.get('category'),
                    'importance': meta.get('importance', 'normal'),
                    'source': meta.get('source', 'agent'),
                    'refs': [],
                    'sessions': set(),
                    'occurrences': 0,
                    'tokens': [],
                    'normalized_claim': text.lower(),
                }
                groups.append(matched)
            matched['refs'].append(f'{path}:{line_no}')
            matched['sessions'].add(session_id)
            matched['occurrences'] += 1
            if not matched['category'] and meta.get('category') in VALID_CATEGORIES:
                matched['category'] = meta.get('category')
    out: list[Candidate] = []
    for group in groups:
        out.append(Candidate(
            text=group['representative'],
            normalized_claim=group['normalized_claim'],
            tokens=group['tokens'],
            category=group['category'],
            importance=group['importance'],
            source=group['source'],
            refs=group['refs'],
            distinct_days=len(group['sessions']),
            occurrences=group['occurrences'],
            score=float(group['occurrences'] + len(group['sessions'])),
        ))
    out.sort(key=lambda item: (-item.score, item.text.lower()))
    return out


path: agent_memory/__init__.py
content: from .api import Journal, LegacyJournal

__all__ = ["Journal", "LegacyJournal"]


path: agent_memory/promote.py
content: from __future__ import annotations

import re
from dataclasses import dataclass
from pathlib import Path

from .normalize import claims_match, normalize_claim, token_counter
from .storage import VALID_CATEGORIES, append_core_memory, init_memory_root

EPISODIC_LINE_RE = re.compile(r"^-\s+(\d{2}:\d{2})\s+(.*?)(?:\s+\[(.*)\])?$")
FIELD_RE = re.compile(r"(category|importance|source):([^\s\]]+)")
DEFAULT_MATCH_THRESHOLD = 0.60
DEFAULT_OVERLAP_THRESHOLD = 0.5
DEFAULT_TRIGGERS = [
    r'\bremember\b',
    r'\bdecision\b',
    r'\bdecided\b',
    r'\bfrom now on\b',
    r'\balways\b',
    r'\bpriority\b',
]


@dataclass
class Candidate:
    text: str
    normalized_claim: str
    tokens: list[str]
    category: str | None
    importance: str
    source: str
    refs: list[str]
    distinct_days: int
    occurrences: int
    score: float


def _parse_metadata(blob: str | None) -> dict[str, str]:
    if not blob:
        return {}
    return {match.group(1): match.group(2) for match in FIELD_RE.finditer(blob)}


def iter_episodic_entries(root: str | Path):
    paths = init_memory_root(root)
    for path in sorted(paths.episodic_dir.glob('*.md')):
        day = path.stem.split('/')[-1]
        lines = path.read_text(encoding='utf-8', errors='ignore').splitlines()
        for line_no, line in enumerate(lines, start=1):
            match = EPISODIC_LINE_RE.match(line.strip())
            if not match:
                continue
            hhmm, text, meta_blob = match.groups()
            meta = _parse_metadata(meta_blob)
            yield {
                'date': day,
                'time': hhmm,
                'text': text.strip(),
                'category': meta.get('category'),
                'importance': meta.get('importance', 'normal'),
                'source': meta.get('source', 'agent'),
                'ref': f'{path}:{line_no}',
            }


def _counter_overlap_ratio(a: str, b: str) -> float:
    ca = token_counter(a)
    cb = token_counter(b)
    if not ca or not cb:
        return 0.0
    overlap = sum(min(ca[token], cb[token]) for token in ca if token in cb)
    denom = min(sum(ca.values()), sum(cb.values())) or 1
    return overlap / denom


def _guess_category(text: str) -> str | None:
    from .migrate import CATEGORY_HINTS
    lowered = text.lower()
    for needle, category in CATEGORY_HINTS.items():
        if needle in lowered:
            return category
    return None


def collect_candidates(root: str | Path, match_threshold: float = DEFAULT_MATCH_THRESHOLD, overlap_threshold: float = DEFAULT_OVERLAP_THRESHOLD) -> list[Candidate]:
    groups: list[dict] = []
    for entry in iter_episodic_entries(root):
        normalized, tokens = normalize_claim(entry['text'])
        if not normalized:
            continue
        matched_group = None
        for group in groups:
            # Semantic deduplication logic: combined Jaccard and Token Overlap
            if claims_match(group['representative'], entry['text'], threshold=match_threshold) or _counter_overlap_ratio(group['representative'], entry['text']) >= overlap_threshold:
                matched_group = group
                break
        if not matched_group:
            # New candidate discovery
            guessed_cat = _guess_category(entry['text'])
            matched_group = {
                'representative': entry['text'],
                'normalized_claim': normalized,
                'tokens': tokens,
                'category': entry['category'] or guessed_cat,
                'importance': entry['importance'],
                'source': entry['source'],
                'refs': [],
                'days': set(),
                'occurrences': 0,
            }
            groups.append(matched_group)
        matched_group['refs'].append(entry['ref'])
        matched_group['days'].add(entry['date'])
        matched_group['occurrences'] += 1
        if not matched_group['category'] and entry['category'] in VALID_CATEGORIES:
            matched_group['category'] = entry['category']
        if entry['importance'] == 'high':
            matched_group['importance'] = 'high'
    candidates: list[Candidate] = []
    for group in groups:
        score = float(group['occurrences'] + len(group['days']))
        if group['importance'] == 'high':
            score += 1.0
        candidates.append(
            Candidate(
                text=group['representative'],
                normalized_claim=group['normalized_claim'],
                tokens=group['tokens'],
                category=group['category'],
                importance=group['importance'],
                source=group['source'],
                refs=list(group['refs']),
                distinct_days=len(group['days']),
                occurrences=group['occurrences'],
                score=score,
            )
        )
    candidates.sort(key=lambda item: (-item.score, -(item.distinct_days), item.text.lower()))
    return candidates


def promote_repeated_candidates(root: str | Path, min_distinct_days: int = 2, match_threshold: float = DEFAULT_MATCH_THRESHOLD, overlap_threshold: float = DEFAULT_OVERLAP_THRESHOLD) -> list[Candidate]:
    promoted: list[Candidate] = []
    for candidate in collect_candidates(root, match_threshold=match_threshold, overlap_threshold=overlap_threshold):
        if candidate.category not in VALID_CATEGORIES:
            continue
        if candidate.distinct_days < min_distinct_days:
            continue
        try:
            append_core_memory(root, category=candidate.category, text=candidate.text, source='auto')
        except ValueError:
            continue
        promoted.append(candidate)
    return promoted


path: agent_memory/review_memory.py
content: from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

from .session import collect_session_candidates
from .storage import append_session_note, init_memory_root, sanitize_session_id


@dataclass
class ReviewMemoryReport:
    notes_written: int
    session_candidates: int
    session_id: str


def log_review_findings(root: str | Path, session_id: str, findings: list[str], category: str = 'gotcha') -> ReviewMemoryReport:
    root_path = Path(root).expanduser().resolve()
    v2_root = init_memory_root(root_path if root_path.name == '.memory' else root_path / '.memory').root
    effective_session_id = sanitize_session_id(session_id)
    written = 0
    for finding in findings:
        text = finding.strip()
        if not text:
            continue
        append_session_note(v2_root, session_id=effective_session_id, text=text, category=category, importance='high', source='subagent')
        written += 1
    candidates = collect_session_candidates(v2_root)
    return ReviewMemoryReport(
        notes_written=written,
        session_candidates=len(candidates),
        session_id=effective_session_id,
    )


path: agent_memory/review.py
content: from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

from .hot import _has_metadata_flag, effective_hot_budget, rebuild_agent_md
from .storage import extract_state
from .promote import DEFAULT_MATCH_THRESHOLD, DEFAULT_OVERLAP_THRESHOLD, collect_candidates
from .session import collect_session_candidates
from .storage import init_memory_root


@dataclass
class ReviewReport:
    status: str
    episodic_candidates: int
    repeated_episodic_candidates: int
    session_candidates: int
    pinned_core_items: int
    hot_chars: int
    hot_path: str


def _count_pinned_core_items(root: str | Path) -> int:
    paths = init_memory_root(root)
    count = 0
    for core_file in paths.core_dir.glob('*.md'):
        for line in core_file.read_text(encoding='utf-8', errors='ignore').splitlines():
            stripped = line.strip()
            if not stripped.startswith('- '):
                continue
            if not _has_metadata_flag(stripped, 'pinned:true'):
                continue
            if extract_state(stripped) == 'superseded':
                continue
            count += 1
    return count


def review_state(root: str | Path) -> ReviewReport:
    paths = init_memory_root(root)
    rebuild = rebuild_agent_md(paths.root)
    hot_text = paths.hot_file.read_text(encoding='utf-8', errors='ignore')
    episodic = collect_candidates(paths.root, match_threshold=DEFAULT_MATCH_THRESHOLD, overlap_threshold=DEFAULT_OVERLAP_THRESHOLD)
    session = collect_session_candidates(paths.root, match_threshold=DEFAULT_MATCH_THRESHOLD, overlap_threshold=DEFAULT_OVERLAP_THRESHOLD)
    repeated = sum(1 for item in episodic if item.distinct_days >= 2)
    budget = effective_hot_budget(paths.config)
    status = 'OK' if len(hot_text) <= budget else 'ISSUES_FOUND'
    return ReviewReport(
        status=status,
        episodic_candidates=len(episodic),
        repeated_episodic_candidates=repeated,
        session_candidates=len(session),
        pinned_core_items=_count_pinned_core_items(paths.root),
        hot_chars=len(hot_text),
        hot_path=rebuild['path'],
    )


path: agent_memory/api.py
content: from __future__ import annotations

from pathlib import Path

from .core_recall import recall_core
from .episodic_recall import recall_episodic
from .hot import rebuild_agent_md
from .ingest import ingest_cycle
from .legacy import LegacyJournal
from .models import RecallResult
from .review_memory import log_review_findings
from .storage import append_core_memory, append_episodic_note, append_session_note, init_memory_root, supersede_memory

WARM_SCORE_WEIGHT = 2.0
PER_TIER_FETCH_MULTIPLIER = 2


class Journal:
    def __init__(self, root: str | Path = '.'):
        self.root = Path(root).expanduser().resolve()
        self.v2_root = self.root if self.root.name == '.memory' else self.root / '.memory'

    def init(self) -> Path:
        return init_memory_root(self.v2_root).root

    def recall(self, query: str, k: int = 5, tier: str = 'all'):
        if tier == 'warm':
            hits = self.recall_core(query, k=k)
            return [
                RecallResult(
                    text=h.text,
                    path=h.path,
                    line_no=h.line_no,
                    score=h.score,
                    tier='warm',
                    category=h.category
                ) for h in hits
            ]
        if tier == 'cold':
            hits = recall_episodic(self.v2_root, query=query, k=k)
            return [
                RecallResult(
                    text=h.text,
                    path=h.path,
                    line_no=h.line_no,
                    score=h.score,
                    tier='cold'
                ) for h in hits
            ]
        if tier == 'all':
            per_tier = max(1, k * PER_TIER_FETCH_MULTIPLIER)
            warm_hits = self.recall_core(query, k=per_tier)
            warm_results = [
                RecallResult(
                    text=h.text,
                    path=h.path,
                    line_no=h.line_no,
                    score=h.score * WARM_SCORE_WEIGHT,
                    tier='warm',
                    category=h.category
                ) for h in warm_hits
            ]

            cold_hits = recall_episodic(self.v2_root, query=query, k=per_tier)
            cold_results = [
                RecallResult(
                    text=h.text,
                    path=h.path,
                    line_no=h.line_no,
                    score=h.score,
                    tier='cold'
                ) for h in cold_hits
            ]

            merged = warm_results + cold_results
            merged.sort(key=lambda item: (-item.score, item.path, item.line_no))
            return merged[: max(1, k)]
        return []

    def note(self, text: str, category: str | None = None, importance: str = 'normal', source: str = 'agent') -> Path:
        return append_episodic_note(self.v2_root, text=text, category=category, importance=importance, source=source)

    def remember(self, text: str, category: str, source: str = 'agent', pinned: bool = False, supersedes: str | None = None) -> Path:
        if supersedes and not self.forget(supersedes):
            raise ValueError(f"cannot supersede missing memory id: {supersedes}")
        path = append_core_memory(self.v2_root, category=category, text=text, source=source, pinned=pinned, supersedes=supersedes)
        if pinned:
            rebuild_agent_md(self.v2_root)
        return path

    def forget(self, memory_id: str) -> bool:
        success = supersede_memory(self.v2_root, memory_id=memory_id)
        if success:
            # Cheap pass over core/; ensures any pinned line we just superseded
            # disappears from AGENT.md without waiting for the next ingest.
            rebuild_agent_md(self.v2_root)
        return success

    def session_note(self, session_id: str, text: str, category: str | None = None, importance: str = 'normal', source: str = 'agent') -> Path:
        return append_session_note(self.v2_root, session_id=session_id, text=text, category=category, importance=importance, source=source)

    def recall_core(self, query: str, k: int = 5, update_last_seen: bool = True):
        return recall_core(self.v2_root, query=query, k=k, update_last_seen=update_last_seen)

    def ingest(self):
        return ingest_cycle(self.v2_root)

    def log_review_findings(self, session_id: str, findings: list[str], category: str = 'gotcha'):
        return log_review_findings(self.v2_root, session_id=session_id, findings=findings, category=category)


path: agent_memory/cli.py
content: import argparse
import json
import os
import sys
from pathlib import Path
from datetime import datetime
from importlib.metadata import PackageNotFoundError, version as _pkg_version
from .api import Journal, LegacyJournal
from .review import review_state
from .doctor_v2 import doctor_verify
from .migrate import import_legacy_workspace
from .analytics import memory_stats, memory_topics, memory_cadence, memory_digest, extract_candidates
from .promote import collect_candidates, DEFAULT_TRIGGERS
from .episodic_recall import recall_episodic, recall_recent


def _resolve_version() -> str:
    try:
        return _pkg_version('agent-memory-journal')
    except PackageNotFoundError:
        return '0.0.0+dev'


def parse_iso_date(value: str):
    try:
        return datetime.strptime(value, "%Y-%m-%d").date()
    except ValueError as exc:
        raise argparse.ArgumentTypeError(f"Invalid date '{value}', expected YYYY-MM-DD") from exc


def main():
    ap = argparse.ArgumentParser(description="Agent Memory Journal V2")
    ap.add_argument("--root", default=os.environ.get("AGENT_MEMORY_ROOT", "."), help="Memory root directory")
    ap.add_argument("--memory-dir", default="memory", help="Legacy memory directory")
    ap.add_argument("--long-file", default="MEMORY.md", help="Legacy long file")
    ap.add_argument("--version", action="version", version=_resolve_version())
    sub = ap.add_subparsers(dest="cmd", required=True)

    # Note command
    n = sub.add_parser("note", help="Add an episodic note")
    n.add_argument("text")
    n.add_argument("--category", help="Optional category")
    n.add_argument("--importance", default="normal", choices=["normal", "high"])

    # Legacy Add command
    a = sub.add_parser("add", help="Add a note (legacy compatibility)")
    a.add_argument("--note", required=True)
    a.add_argument("--long", action="store_true")

    # Remember command
    r = sub.add_parser("remember", help="Add a core memory")
    r.add_argument("text")
    r.add_argument("--category", required=True)
    r.add_argument("--pinned", action="store_true")

    # Forget command
    f = sub.add_parser("forget", help="Supersede a core memory by ID")
    f.add_argument("id")

    # Ingest command
    sub.add_parser("ingest", help="Run ingestion cycle (promote & rebuild)")

    # Review command
    rv = sub.add_parser("review", help="Show memory layer status report")
    rv.add_argument("--json", action="store_true")

    # Doctor command
    dc = sub.add_parser("doctor", help="Verify integrity of core memory")
    dc.add_argument("--json", action="store_true")
    dc.add_argument("--strict", action="store_true")
    dc.add_argument("--fix", action="store_true")
    dc.add_argument("--after", type=parse_iso_date)
    dc.add_argument("--before", type=parse_iso_date)

    # Migrate command
    sub.add_parser("migrate", help="Import legacy memory files into V2 layout")

    # Stats command
    st = sub.add_parser("stats", help="Show episodic memory statistics")
    st.add_argument("--days", type=int, default=7)
    st.add_argument("--json", action="store_true")

    # Topics command
    tp = sub.add_parser("topics", help="Surface recurring topics")
    tp.add_argument("--days", type=int, default=14)
    tp.add_argument("--json", action="store_true")

    # Cadence command
    cd = sub.add_parser("cadence", help="Show daily note volume")
    cd.add_argument("--days", type=int, default=14)
    cd.add_argument("--json", action="store_true")

    # Digest command
    dg = sub.add_parser("digest", help="Print operational digest")
    dg.add_argument("--days", type=int, default=7)
    dg.add_argument("--json", action="store_true")

    # Candidates command
    can = sub.add_parser("candidates", help="Surface promotion candidates")
    can.add_argument("--days", type=int, default=7)
    can.add_argument("--json", action="store_true")

    # Extract command
    ex = sub.add_parser("extract", help="Extract memory lines from text")
    ex.add_argument("--file", type=Path)

    # Recent command
    rc = sub.add_parser("recent", help="Show recent episodic notes")
    rc.add_argument("--days", type=int, default=2)
    rc.add_argument("--json", action="store_true")
    rc.add_argument("--grep")

    # Search command
    sh = sub.add_parser("search", help="Search memory")
    sh.add_argument("--query", required=True)
    sh.add_argument("--tier", choices=["all", "warm", "cold"], default="all")
    sh.add_argument("--json", action="store_true")

    # Init command
    it = sub.add_parser("init", help="Initialize memory root")
    it.add_argument("--with-config", action="store_true")

    args = ap.parse_args()
    journal = Journal(root=args.root)

    if args.cmd == "note":
        path = journal.note(args.text, category=args.category, importance=args.importance)
        print(f"Note added to {path}")
    elif args.cmd == "add":
        if args.long:
            path = journal.remember(args.note, category="decision")
            print(f"Memory remembered in {path}")
        else:
            path = journal.note(args.note)
            print(f"Note added to {path}")
    elif args.cmd == "remember":
        path = journal.remember(args.text, category=args.category, pinned=args.pinned)
        print(f"Memory remembered in {path}")
    elif args.cmd == "forget":
        success = journal.forget(args.id)
        if success:
            print(f"Memory {args.id} superseded.")
        else:
            print(f"Memory {args.id} not found.")
    elif args.cmd == "ingest":
        report = journal.ingest()
        print(json.dumps(report.__dict__ if hasattr(report, "__dict__") else report, indent=2))
    elif args.cmd == "review":
        report = review_state(journal.v2_root)
        if args.json:
            print(json.dumps(report.__dict__ if hasattr(report, "__dict__") else report, indent=2))
        else:
            print(f"Status: {report.status}")
            print(f"Episodic candidates: {report.episodic_candidates}")
            print(f"Pinned items: {report.pinned_core_items}")
    elif args.cmd == "doctor":
        report = doctor_verify(journal.v2_root, fix=args.fix)
        if args.json:
            print(json.dumps(report.__dict__ if hasattr(report, "__dict__") else report, indent=2))
        else:
            print(f"Status: {report.status}")
        if args.strict and report.status != "OK":
            sys.exit(1)
    elif args.cmd == "migrate":
        results = import_legacy_workspace(journal.root if journal.root.name != ".memory" else journal.root.parent)
        print(json.dumps(results, indent=2))
    elif args.cmd == "stats":
        stats = memory_stats(journal.v2_root, days=args.days)
        print(json.dumps(stats, indent=2))
    elif args.cmd == "topics":
        topics = memory_topics(journal.v2_root, days=args.days)
        print(json.dumps(topics, indent=2))
    elif args.cmd == "cadence":
        cadence = memory_cadence(journal.v2_root, days=args.days)
        print(json.dumps(cadence, indent=2))
    elif args.cmd == "digest":
        digest = memory_digest(journal.v2_root, days=args.days)
        print(json.dumps(digest, indent=2))
    elif args.cmd == "candidates":
        candidates = collect_candidates(journal.v2_root)
        print(json.dumps([c.__dict__ for c in candidates], indent=2))
    elif args.cmd == "extract":
        text = args.file.read_text(encoding="utf-8") if args.file else sys.stdin.read()
        lines = extract_candidates(text, triggers=DEFAULT_TRIGGERS)
        print(json.dumps(lines, indent=2))
    elif args.cmd == "recent":
        if args.grep:
            hits = recall_episodic(journal.v2_root, query=args.grep, k=100)
        else:
            hits = recall_recent(journal.v2_root, days=args.days, k=100)
        if args.json:
            print(json.dumps([h.__dict__ for h in hits], indent=2))
        else:
            for h in hits:
                print(f"- {h.text}")
    elif args.cmd == "search":
        hits = journal.recall(args.query, tier=args.tier)
        if args.json:
            print(json.dumps([{"text": h.text, "path": h.path, "score": h.score, "tier": getattr(h, "tier", "cold")} for h in hits], indent=2))
        else:
            for h in hits:
                tier = getattr(h, "tier", "cold")
                print(f"[{tier}] {h.text}")
    elif args.cmd == "init":
        journal.init()
        print(json.dumps({"created": [], "root": str(journal.v2_root)}))


if __name__ == "__main__":
    main()


path: agent_memory/doctor_v2.py
content: from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path

from .hot import effective_hot_budget
from .storage import init_memory_root


@dataclass
class DoctorReport:
    status: str
    hot_chars: int
    hot_over_limit: bool
    manifest_mismatches: list[str]
    checked_files: int


def _hash_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def refresh_manifest(root: str | Path) -> dict[str, object]:
    paths = init_memory_root(root)
    tracked = {}
    for file_path in sorted(paths.core_dir.glob('*.md')):
        tracked[str(file_path.relative_to(paths.root))] = _hash_file(file_path)
    manifest = {
        'schema': 1,
        'layout': 'agent-memory-v2',
        'core_sha256': tracked,
    }
    target = paths.index_dir / 'manifest.json'
    target.write_text(json.dumps(manifest, indent=2) + '\n', encoding='utf-8')
    return manifest


def doctor_verify(root: str | Path, fix: bool = False) -> DoctorReport:
    paths = init_memory_root(root)
    manifest_path = paths.index_dir / 'manifest.json'
    if not manifest_path.exists() or fix:
        refresh_manifest(paths.root)

    mismatches: list[str] = []
    try:
        manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
    except Exception:
        manifest = {}
        mismatches.append('invalid:manifest-json')

    schema = manifest.get('schema')
    layout = manifest.get('layout')
    tracked = manifest.get('core_sha256')

    if schema != 1:
        mismatches.append('invalid:schema')
    if layout != 'agent-memory-v2':
        mismatches.append('invalid:layout')
    if not isinstance(tracked, dict) or not tracked:
        manifest = refresh_manifest(paths.root)
        tracked = manifest.get('core_sha256', {})

    checked = 0
    if not isinstance(tracked, dict):
        mismatches.append('invalid:core_sha256')
        tracked = {}

    for rel_path, expected in tracked.items():
        if not isinstance(rel_path, str) or not isinstance(expected, str):
            mismatches.append('invalid:core_sha256-entry')
            continue
        file_path = paths.root / rel_path
        checked += 1
        if not file_path.exists():
            mismatches.append(f'missing:{rel_path}')
            continue
        actual = _hash_file(file_path)
        if actual != expected:
            mismatches.append(f'mismatch:{rel_path}')
    hot_chars = len(paths.hot_file.read_text(encoding='utf-8', errors='ignore')) if paths.hot_file.exists() else 0
    hot_budget = effective_hot_budget(paths.config)
    hot_over_limit = hot_chars > hot_budget
    status = 'OK' if not mismatches and not hot_over_limit and checked > 0 else 'ISSUES_FOUND'
    return DoctorReport(
        status=status,
        hot_chars=hot_chars,
        hot_over_limit=hot_over_limit,
        manifest_mismatches=mismatches,
        checked_files=checked,
    )


path: agent_memory/examples.py
content: from __future__ import annotations

EXAMPLE_SNIPPETS = {
    'init': "from agent_memory import Journal\nj = Journal(root='.')\nj.init()",
    'write': "j.note('Decision: keep AGENT hot set tiny', category='decision', importance='high')\nj.remember('AGENT.md must stay small and pinned-only', category='constraint', pinned=True)",
    'session': "j.session_note('review-42', 'Pinned detection is too loose', category='gotcha', importance='high')",
    'recall': "from agent_memory.core_recall import recall_core\nhits = recall_core('.memory', 'pinned hot set', k=5)",
    'forget': "j.forget('dec-1234567890')",
    'ingest': "from agent_memory.ingest import ingest_cycle\nreport = ingest_cycle('.memory')",
}


def render_examples() -> str:
    sections = ['# Examples', '']
    for name, snippet in EXAMPLE_SNIPPETS.items():
        sections.append(f'## {name}')
        sections.append('```python')
        sections.append(snippet)
        sections.append('```')
        sections.append('')
    return '\n'.join(sections).rstrip() + '\n'


path: agent_memory/storage.py
content: from __future__ import annotations

import hashlib
import json
import re
import os
import shutil
import tempfile
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path

from .models_v2 import MemoryItem
from .security import require_safe_memory_text, sanitize_evidence_text, validate_hot_path

VALID_CATEGORIES = {"decision", "constraint", "gotcha", "preference", "capability"}
META_RE = re.compile(r"\s+\[(.*)\]\s*$")
ID_RE = re.compile(r"(?:^|\s)id:([^\s\]]+)")
STATE_RE = re.compile(r"(?:^|\s)state:([^\s\]]+)")


def sanitize_session_id(session_id: str) -> str:
    safe_session_id = ''.join(ch for ch in session_id if ch.isalnum() or ch in {'-', '_'})
    if not safe_session_id:
        raise ValueError('session_id must contain at least one safe character')
    return safe_session_id


def utc_now_iso() -> str:
    return datetime.now(timezone.utc).replace(microsecond=0).isoformat()


def make_memory_id(category: str, text: str) -> str:
    digest = hashlib.sha1(f'{category}:{text}'.encode('utf-8')).hexdigest()[:12]
    return f'{category[:3]}-{digest}'


def split_claim_and_metadata(line: str) -> tuple[str, str]:
    stripped = line.strip()
    if stripped.startswith('- '):
        stripped = stripped[2:]
    match = META_RE.search(stripped)
    if not match:
        return stripped.strip(), ''
    meta = match.group(1).strip()
    claim = stripped[:match.start()].strip()
    return claim, meta


def extract_id(line: str) -> str | None:
    _claim, meta = split_claim_and_metadata(line)
    match = ID_RE.search(meta)
    return match.group(1) if match else None


def extract_state(line: str) -> str | None:
    _claim, meta = split_claim_and_metadata(line)
    match = STATE_RE.search(meta)
    return match.group(1) if match else None


def has_active_memory(target: Path, memory_id: str) -> bool:
    if not target.exists():
        return False
    for line in target.read_text(encoding='utf-8', errors='ignore').splitlines():
        if not line.strip().startswith('- '):
            continue
        if extract_id(line) == memory_id and extract_state(line) != 'superseded':
            return True
    return False


def render_memory_item(item: MemoryItem) -> str:
    flags = [
        f'id:{item.id}',
        f'state:{item.state}',
        f'source:{item.source}',
        f'created:{item.created_at}',
    ]
    if item.last_seen:
        flags.append(f'last_seen:{item.last_seen}')
    if item.pinned:
        flags.append('pinned:true')
    if item.supersedes:
        flags.append(f'supersedes:{item.supersedes}')
    return f"- {item.text} [{' '.join(flags)}]"


def _load_memory_config(root: Path) -> dict:
    config_path = root / 'config.json'
    if not config_path.exists():
        return {}
    try:
        data = json.loads(config_path.read_text(encoding='utf-8'))
        return data if isinstance(data, dict) else {}
    except Exception:
        return {}


@dataclass
class MemoryPaths:
    root: Path

    @property
    def config_file(self) -> Path:
        return self.root / 'config.json'

    @property
    def config(self) -> dict:
        return _load_memory_config(self.root)

    @property
    def hot_file(self) -> Path:
        hot_path = self.config.get('hot_path', 'AGENT.md')
        return validate_hot_path(self.root, hot_path)

    @property
    def episodic_dir(self) -> Path:
        return self.root / 'episodic'

    @property
    def core_dir(self) -> Path:
        return self.root / 'core'

    @property
    def sessions_dir(self) -> Path:
        return self.root / 'sessions'

    @property
    def archive_core_dir(self) -> Path:
        return self.root / 'archive' / 'core'

    @property
    def index_dir(self) -> Path:
        return self.root / 'index'

    def core_file(self, category: str) -> Path:
        normalized = category.strip().lower()
        if normalized not in VALID_CATEGORIES:
            raise ValueError(f"Invalid category '{category}'")
        mapping = {
            'decision': 'decisions.md',
            'constraint': 'constraints.md',
            'gotcha': 'gotchas.md',
            'preference': 'preferences.md',
            'capability': 'capabilities.md',
        }
        return self.core_dir / mapping[normalized]


def init_memory_root(root: str | Path) -> MemoryPaths:
    paths = MemoryPaths(Path(root).expanduser().resolve())
    paths.root.mkdir(parents=True, exist_ok=True)
    paths.episodic_dir.mkdir(parents=True, exist_ok=True)
    paths.core_dir.mkdir(parents=True, exist_ok=True)
    paths.sessions_dir.mkdir(parents=True, exist_ok=True)
    paths.archive_core_dir.mkdir(parents=True, exist_ok=True)
    paths.index_dir.mkdir(parents=True, exist_ok=True)
    if not paths.config_file.exists():
        paths.config_file.write_text(json.dumps({
            'hot_path': 'AGENT.md',
            'hot_header': '# AGENT.md',
            'hot_max_chars': 2048
        }, indent=2) + '\n', encoding='utf-8')
    if not paths.hot_file.exists():
        paths.hot_file.parent.mkdir(parents=True, exist_ok=True)
        header = paths.config.get('hot_header', '# AGENT.md')
        paths.hot_file.write_text(f'{header}\n\n', encoding='utf-8')
    for category in sorted(VALID_CATEGORIES):
        target = paths.core_file(category)
        if not target.exists():
            title = target.stem.replace('_', ' ').replace('-', ' ').title()
            target.write_text(f'# {title}\n\n', encoding='utf-8')
    manifest = paths.index_dir / 'manifest.json'
    if not manifest.exists():
        manifest.write_text('{"schema":1,"layout":"agent-memory-v2"}\n', encoding='utf-8')
    return paths


def _refresh_manifest_entry(paths: 'MemoryPaths', file_path: Path) -> None:
    manifest_path = paths.index_dir / 'manifest.json'
    try:
        manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
        if not isinstance(manifest, dict):
            manifest = {}
    except Exception:
        manifest = {}
    manifest.setdefault('schema', 1)
    manifest.setdefault('layout', 'agent-memory-v2')
    tracked = manifest.get('core_sha256')
    if not isinstance(tracked, dict):
        tracked = {}
        manifest['core_sha256'] = tracked
    if not file_path.exists():
        return
    digest = hashlib.sha256(file_path.read_bytes()).hexdigest()
    rel = str(file_path.relative_to(paths.root))
    tracked[rel] = digest
    manifest_path.parent.mkdir(parents=True, exist_ok=True)
    manifest_path.write_text(json.dumps(manifest, indent=2) + '\n', encoding='utf-8')


def _atomic_append(path: Path, text: str):
    path.parent.mkdir(parents=True, exist_ok=True)
    if not path.exists():
        path.write_text(text, encoding='utf-8')
        return

    # Use a temporary file to prevent corruption during concurrent writes
    fd, temp_path = tempfile.mkstemp(dir=str(path.parent), prefix=path.name + ".tmp")
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            # Copy existing content
            f.write(path.read_text(encoding='utf-8', errors='ignore'))
            # Append new text
            f.write(text)
        # Atomically replace the original file
        shutil.move(temp_path, str(path))
    except Exception:
        if os.path.exists(temp_path):
            os.remove(temp_path)
        raise


def append_episodic_note(root: str | Path, text: str, category: str | None = None, importance: str = 'normal', source: str = 'agent') -> Path:
    paths = init_memory_root(root)
    now = datetime.now(timezone.utc)
    path = paths.episodic_dir / f"{now.date()}.md"
    cleaned = sanitize_evidence_text(text)
    suffix = []
    if category:
        suffix.append(f'category:{category}')
    if importance != 'normal':
        suffix.append(f'importance:{importance}')
    if source:
        suffix.append(f'source:{source}')
    trailer = f" [{' '.join(suffix)}]" if suffix else ''
    _atomic_append(path, f"- {now.strftime('%H:%M')} {cleaned}{trailer}\n")
    return path


def append_core_memory(root: str | Path, category: str, text: str, source: str = 'agent', pinned: bool = False, supersedes: str | None = None, state: str = 'active') -> Path:
    paths = init_memory_root(root)
    target = paths.core_file(category)
    cleaned = require_safe_memory_text(text)

    if supersedes:
        # Validate that the supersedes ID actually exists
        found_target = False
        for core_file in paths.core_dir.glob('*.md'):
            if f'id:{supersedes}' in core_file.read_text(encoding='utf-8', errors='ignore'):
                found_target = True
                break
        if not found_target:
            raise ValueError(f"supersedes ID '{supersedes}' not found in core memory")

    item = MemoryItem(
        id=make_memory_id(category, cleaned),
        text=cleaned,
        category=category,
        tier='warm',
        state=state,
        source=source,
        created_at=utc_now_iso(),
        last_seen=utc_now_iso(),
        pinned=pinned,
        supersedes=supersedes,
    )
    if has_active_memory(target, item.id):
        return target
    _atomic_append(target, render_memory_item(item) + '\n')
    _refresh_manifest_entry(paths, target)
    return target


def append_session_note(root: str | Path, session_id: str, text: str, category: str | None = None, importance: str = 'normal', source: str = 'agent') -> Path:
    paths = init_memory_root(root)
    safe_session_id = sanitize_session_id(session_id)
    target = paths.sessions_dir / f'{safe_session_id}.md'
    now = datetime.now(timezone.utc)
    cleaned = sanitize_evidence_text(text)
    suffix = []
    if category:
        suffix.append(f'category:{category}')
    if importance != 'normal':
        suffix.append(f'importance:{importance}')
    if source:
        suffix.append(f'source:{source}')
    trailer = f" [{' '.join(suffix)}]" if suffix else ''
    _atomic_append(target, f"- {now.strftime('%H:%M')} {cleaned}{trailer}\n")
    return target


def supersede_memory(root: str | Path, memory_id: str, new_state: str = 'superseded') -> bool:
    paths = init_memory_root(root)
    found = False
    for core_file in paths.core_dir.glob('*.md'):
        content = core_file.read_text(encoding='utf-8', errors='ignore')
        if f'id:{memory_id}' not in content:
            continue

        new_lines = []
        lines = content.splitlines()
        for line in lines:
            if not line.strip().startswith('- '):
                new_lines.append(line)
                continue

            current_id = extract_id(line)
            if current_id == memory_id:
                # Update state:active -> state:superseded
                new_line = re.sub(r'state:[^\s\]]+', f'state:{new_state}', line)
                new_lines.append(new_line)
                found = True
            else:
                new_lines.append(line)

        if found:
            # Atomic rewrite
            fd, temp_path = tempfile.mkstemp(dir=str(core_file.parent), prefix=core_file.name + ".tmp")
            try:
                with os.fdopen(fd, 'w', encoding='utf-8') as f:
                    f.write('\n'.join(new_lines).rstrip() + '\n')
                shutil.move(temp_path, str(core_file))
            except Exception:
                if os.path.exists(temp_path):
                    os.remove(temp_path)
                raise
            _refresh_manifest_entry(paths, core_file)
            break
    return found


path: agent_memory/analytics.py
content: from __future__ import annotations

import re
from collections import Counter
from datetime import datetime, timedelta
from pathlib import Path

from .normalize import tokenize
from .storage import init_memory_root


def _iter_files(paths, days: int, after_date=None, before_date=None):
    discovered = []
    for p in paths.episodic_dir.glob("*.md"):
        try:
            d = datetime.strptime(p.stem, "%Y-%m-%d").date()
            discovered.append((d, p))
        except ValueError:
            continue
    if not discovered:
        return []
    latest_file_date = max(d for d, _p in discovered)
    anchor_date = min(datetime.now().date(), latest_file_date) if not before_date else before_date
    rolling_cutoff = anchor_date - timedelta(days=max(0, days))
    min_date = max(rolling_cutoff, after_date) if after_date else rolling_cutoff
    max_date = before_date
    candidates = []
    for d, p in discovered:
        if d < min_date:
            continue
        if max_date and d > max_date:
            continue
        candidates.append(p)
    return sorted(candidates, reverse=True)


def _note_words(paths, days: int, after_date=None, before_date=None):
    files = _iter_files(paths, days, after_date=after_date, before_date=before_date)
    words = []
    notes = []
    stop = {
        "the", "and", "for", "with", "that", "this", "from", "have", "was", "were", "are", "but", "not", "into", "out", "new", "added", "gained", "now", "also", "recent", "notes",
        "että", "joka", "tämä", "tuo", "ovat", "oli", "kun", "tai", "jos", "nyt", "sekä", "vielä", "kanssa", "uusi", "lisätty"
    }
    line_re = re.compile(r"^-\s+(\d{2}:\d{2})\s+(.*)$")
    for p in files:
        try:
            lines = p.read_text(encoding="utf-8", errors="ignore").splitlines()
        except Exception:
            continue
        for ln in lines:
            m = line_re.match(ln.strip())
            if not m:
                continue
            note = m.group(2).split(" [", 1)[0].strip()
            notes.append(note)
            words.extend(w.lower() for w in re.findall(r"[A-Za-zÅÄÖåäö0-9-]{3,}", note) if w.lower() not in stop)
    return notes, words


def memory_stats(root: str | Path, days: int = 7, top: int = 10):
    paths = init_memory_root(root)
    files = _iter_files(paths, days)
    note_count = 0
    hourly = Counter()
    daily_counts = Counter()
    notes, words = _note_words(paths, days)
    line_re = re.compile(r"^-\s+(\d{2}:\d{2})\s+(.*)$")
    for p in files:
        try:
            lines = p.read_text(encoding="utf-8", errors="ignore").splitlines()
        except Exception:
            continue
        for ln in lines:
            m = line_re.match(ln.strip())
            if not m:
                continue
            hhmm = m.group(1)
            hour = hhmm.split(":")[0]
            note_count += 1
            hourly[hour] += 1
            daily_counts[p.stem] += 1
    return {
        "days_scanned": days,
        "files_found": len(files),
        "days_with_notes": len(daily_counts),
        "note_count": note_count,
        "busiest_hours": [{"hour": h, "count": c} for h, c in hourly.most_common(3)],
        "top_words": [{"word": w, "count": c} for w, c in Counter(words).most_common(top)],
    }


def memory_topics(root: str | Path, days: int = 14, top: int = 8, samples: int = 2, min_count: int = 2, after_date=None, before_date=None):
    paths = init_memory_root(root)
    notes, words = _note_words(paths, days, after_date=after_date, before_date=before_date)
    counts = Counter(words)
    word_samples = {}
    for note in notes:
        tokens = set(tokenize(note))
        for word, count in counts.items():
            if count < min_count:
                continue
            if word in tokens:
                word_samples.setdefault(word, [])
                if len(word_samples[word]) < samples and note not in word_samples[word]:
                    word_samples[word].append(note)
    topics = []
    for word, count in counts.most_common():
        if count < min_count:
            continue
        topics.append({"word": word, "count": count, "samples": word_samples.get(word, [])})
        if len(topics) >= max(1, top):
            break
    return {"days_scanned": days, "note_count": len(notes), "topics": topics}


def memory_cadence(root: str | Path, days: int = 14, top_hours: int = 3, after_date=None, before_date=None):
    paths = init_memory_root(root)
    files = _iter_files(paths, days, after_date=after_date, before_date=before_date)
    per_day = []
    hourly = Counter()
    line_re = re.compile(r"^-\s+(\d{2}:\d{2})\s+(.*)$")
    for p in files:
        count = 0
        try:
            lines = p.read_text(encoding="utf-8", errors="ignore").splitlines()
        except Exception:
            continue
        for ln in lines:
            m = line_re.match(ln.strip())
            if not m:
                continue
            count += 1
            hourly[m.group(1).split(":")[0]] += 1
        per_day.append({"date": p.stem, "count": count})
    return {"days_scanned": days, "per_day": per_day, "busiest_hours": [{"hour": h, "count": c} for h, c in hourly.most_common(top_hours)]}


def memory_digest(root: str | Path, days: int = 7, recent_limit: int = 5, top: int = 5, after_date=None, before_date=None):
    paths = init_memory_root(root)
    stats = memory_stats(paths.root, days=max(1, days), top=max(1, top))
    cadence = memory_cadence(paths.root, days=max(1, days), top_hours=max(1, min(top, 3)), after_date=after_date, before_date=before_date)
    topics = memory_topics(paths.root, days=max(1, days), top=max(1, top), samples=1, min_count=2, after_date=after_date, before_date=before_date)
    from .episodic_recall import recall_recent
    recent = recall_recent(paths.root, days=max(1, days), k=recent_limit)
    return {"days_scanned": days, "stats": stats, "cadence": cadence, "topics": topics, "recent": [r.__dict__ for r in recent]}


def extract_candidates(text: str, triggers: list[str] | None = None) -> list[str]:
    from .promote import DEFAULT_TRIGGERS
    trig = triggers or DEFAULT_TRIGGERS
    lines = [ln.strip() for ln in text.splitlines() if ln.strip()]
    out = []
    for ln in lines:
        if any(re.search(t, ln.lower()) for t in trig):
            out.append(ln)
    return out


path: agent_memory/hot.py
content: from __future__ import annotations

from pathlib import Path

from .storage import init_memory_root, extract_state, split_claim_and_metadata

HOT_LIMIT_CHARS = 2048
HOT_MIN_CHARS = 256
HOT_MAX_CHARS_CEILING = 10000


def _has_metadata_flag(line: str, flag: str) -> bool:
    if '[' not in line or ']' not in line:
        return False
    meta = line.rsplit('[', 1)[-1].rstrip(']')
    return flag in meta.split()


def effective_hot_budget(config: dict, override: int | None = None) -> int:
    if override is not None:
        return max(0, int(override))
    raw = config.get('hot_max_chars', HOT_LIMIT_CHARS)
    return min(max(HOT_MIN_CHARS, int(raw)), HOT_MAX_CHARS_CEILING)


def rebuild_agent_md(root: str | Path, max_chars: int | None = None) -> dict[str, object]:
    paths = init_memory_root(root)
    selected: list[str] = []
    budget = effective_hot_budget(paths.config, override=max_chars)
    header = paths.config.get('hot_header', '# AGENT.md')

    for core_file in sorted(paths.core_dir.glob('*.md')):
        for line in core_file.read_text(encoding='utf-8', errors='ignore').splitlines():
            stripped = line.strip()
            if not stripped.startswith('- '):
                continue
            if not _has_metadata_flag(stripped, 'pinned:true'):
                continue
            if extract_state(stripped) == 'superseded':
                continue
            claim, _meta = split_claim_and_metadata(stripped)
            selected.append(f'- {claim}')

    content_lines = [header, '']
    used = len('\n'.join(content_lines))
    kept: list[str] = []
    skipped: list[str] = []
    for item in selected:
        addition = len(item) + 1
        if used + addition > budget:
            skipped.append(item)
            continue
        kept.append(item)
        used += addition

    if kept:
        content_lines.extend(kept)
    else:
        content_lines.append('')

    paths.hot_file.write_text('\n'.join(content_lines).rstrip() + '\n', encoding='utf-8')
    return {
        'written': len(kept),
        'skipped': len(skipped),
        'max_chars': budget,
        'path': str(paths.hot_file),
    }


path: agent_memory/migrate.py
content: from __future__ import annotations

import re
from pathlib import Path

from .normalize import claims_match
from .storage import VALID_CATEGORIES, append_core_memory, init_memory_root

CATEGORY_HINTS = {
    'decision': 'decision',
    'decided': 'decision',
    'must': 'constraint',
    'always': 'constraint',
    'avoid': 'gotcha',
    'bug': 'gotcha',
    'prefer': 'preference',
    'likes': 'preference',
    'can now': 'capability',
    'supports': 'capability',
    'restored': 'capability',
}
TIMESTAMP_PREFIX_RE = re.compile(r'^\d{1,2}:\d{2}\s+')


def _guess_category(text: str) -> str:
    lowered = text.lower()
    for needle, category in CATEGORY_HINTS.items():
        if needle in lowered:
            return category
    return 'decision'


def _strip_legacy_prefix(text: str) -> str:
    cleaned = text.strip()
    if cleaned.startswith('- '):
        cleaned = cleaned[2:].lstrip()
    cleaned = TIMESTAMP_PREFIX_RE.sub('', cleaned)
    return cleaned.strip()


def _iter_legacy_daily(root: Path):
    legacy = root / 'memory'
    if not legacy.exists():
        return
    for path in sorted(legacy.glob('*.md')):
        if path.name.startswith('.'):
            continue
        for line in path.read_text(encoding='utf-8', errors='ignore').splitlines():
            stripped = line.strip()
            if stripped.startswith('- '):
                yield path.stem, stripped


def _iter_legacy_long(root: Path):
    long_file = root / 'MEMORY.md'
    if not long_file.exists():
        return
    for line in long_file.read_text(encoding='utf-8', errors='ignore').splitlines():
        stripped = line.strip()
        if stripped.startswith('- '):
            yield stripped[2:].strip()


def _existing_core_claims(v2_root: Path) -> list[str]:
    paths = init_memory_root(v2_root)
    claims: list[str] = []
    for core_file in paths.core_dir.glob('*.md'):
        for line in core_file.read_text(encoding='utf-8', errors='ignore').splitlines():
            stripped = line.strip()
            if stripped.startswith('- '):
                claims.append(stripped[2:].split(' [', 1)[0].strip())
    return claims


def _append_imported_episodic(v2_root: Path, date_str: str, text: str, category: str) -> tuple[Path, bool]:
    paths = init_memory_root(v2_root)
    target = paths.episodic_dir / f'{date_str}.md'
    line = f'- 00:00 {text} [category:{category} source:import]\n'
    existing_lines = target.read_text(encoding='utf-8', errors='ignore').splitlines() if target.exists() else []
    if line.rstrip('\n') not in existing_lines:
        with target.open('a', encoding='utf-8') as handle:
            handle.write(line)
        return target, True
    return target, False


def import_legacy_workspace(root: str | Path) -> dict[str, int]:
    workspace = Path(root).expanduser().resolve()
    v2 = init_memory_root(workspace / '.memory')

    imported_episodic = 0
    imported_core = 0
    seen_core: list[str] = _existing_core_claims(v2.root)

    for date_str, line in _iter_legacy_daily(workspace) or []:
        text = _strip_legacy_prefix(line)
        category = _guess_category(text)
        _target, added = _append_imported_episodic(v2.root, date_str=date_str, text=text, category=category)
        if added:
            imported_episodic += 1

    for text in _iter_legacy_long(workspace) or []:
        if any(claims_match(existing, text, threshold=0.85) for existing in seen_core):
            continue
        category = _guess_category(text)
        if category not in VALID_CATEGORIES:
            category = 'decision'
        append_core_memory(v2.root, category=category, text=text, source='import', pinned=False)
        seen_core.append(text)
        imported_core += 1

    return {
        'episodic_imported': imported_episodic,
        'core_imported': imported_core,
    }


path: agent_memory/episodic_recall.py
content: from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path

from .bm25 import BM25Index


@dataclass
class EpisodicRecallHit:
    text: str
    path: str
    score: float
    line_no: int


def recall_recent(root: str | Path, days: int = 2, k: int = 100) -> list[EpisodicRecallHit]:
    root_path = Path(root).expanduser().resolve()
    episodic_dir = root_path / 'episodic'
    if not episodic_dir.exists() or k <= 0:
        return []
    cutoff = datetime.now(timezone.utc).date() - timedelta(days=max(0, days))
    hits: list[EpisodicRecallHit] = []
    for daily_file in sorted(episodic_dir.glob('*.md'), reverse=True):
        try:
            day = datetime.strptime(daily_file.stem, '%Y-%m-%d').date()
        except ValueError:
            continue
        if day < cutoff:
            break
        for line_no, line in enumerate(daily_file.read_text(encoding='utf-8', errors='ignore').splitlines(), start=1):
            stripped = line.strip()
            if not stripped.startswith('- '):
                continue
            hits.append(EpisodicRecallHit(text=stripped[2:], path=str(daily_file), score=0.0, line_no=line_no))
            if len(hits) >= k:
                return hits
    return hits


def recall_episodic(root: str | Path, query: str, k: int = 5) -> list[EpisodicRecallHit]:
    root_path = Path(root).expanduser().resolve()
    episodic_dir = root_path / 'episodic'
    if not query.strip():
        return []
    if k < 0:
        raise ValueError('k must be >= 0')
    if k == 0 or not episodic_dir.exists():
        return []
    docs = []
    meta = []
    for daily_file in sorted(episodic_dir.glob('*.md')):
        for line_no, line in enumerate(daily_file.read_text(encoding='utf-8', errors='ignore').splitlines(), start=1):
            stripped = line.strip()
            if not stripped.startswith('- '):
                continue
            docs.append(stripped[2:])
            meta.append((str(daily_file), line_no))
    if not docs:
        return []
    
    root_path = Path(root).expanduser().resolve()
    cache_path = root_path / 'index' / 'episodic_bm25.json'
    index = BM25Index.from_cache(cache_path, docs)
    scores = index.score(query)
    hits = [
        EpisodicRecallHit(text=doc, path=path, score=score, line_no=line_no)
        for doc, score, (path, line_no) in zip(docs, scores, meta)
        if score > 0
    ]
    hits.sort(key=lambda item: (-item.score, item.path, item.line_no))
    return hits[:k]


path: CHANGELOG.md
content: # Changelog

## 0.2.3 (2026-04-25)
- fix `digest` CLI command crashing with `AttributeError` on `MemoryPaths.v2_root`
- make `rebuild_agent_md(max_chars=...)` honour its caller argument and apply the 256-char floor only to config-derived values; `doctor` and `review` use the same effective budget instead of a hard-coded 2048
- wire `archive_unpinned_core` into `ingest_cycle` so the documented 30-day decay actually happens; `IngestReport` now includes `archived_count`
- BM25 normaliser now keeps token multiplicity, so repeated terms increase TF as the index promised; cache version bumped
- `AGENT.md` is rendered with claim text only (metadata stays in `core/`), recovering ~70% of the hot budget for actual content
- `Journal.remember(..., pinned=True)` and `Journal.forget(...)` now rebuild `AGENT.md` synchronously instead of waiting for the next ingest
- `recall_core` updates `last_seen` in a single batched rewrite per file and refreshes the manifest, so `doctor` stays `OK` after recall
- core writes (`append_core_memory`, `supersede_memory`) refresh the integrity manifest, so `doctor` no longer reports `ISSUES_FOUND` after every legitimate write
- new `recall_recent(root, days, k)` function; `agent-memory-journal recent` now works without `--grep`
- `Journal.recall(tier="all")` weights warm hits 2× and fetches a wider per-tier window before merging
- `_count_pinned_core_items` filters superseded entries so the review report matches what `AGENT.md` actually contains
- CLI `--version` is sourced from `importlib.metadata` instead of a hard-coded literal
- new tests: end-to-end CLI coverage, ingest+decay, recall ranking, hot metadata stripping
- new `tests.yml` GitHub Actions workflow runs pytest on Python 3.10/3.11/3.12

## 0.2.2 (2026-04-25)
- make decay use `last_seen` as the primary activity signal, with `created` as fallback
- add `hot_path` validation so hot-memory promotion stays confined to the workspace or memory root
- make supersedes fail fast when the target id does not exist
- move `LegacyJournal` out of `api.py` into a dedicated `legacy.py` module
- harden hot-set config handling with bounded character budgets and cleaner rebuild logic
- version BM25 cache invalidation against normalizer changes

## 0.2.1 (2026-04-25)
- add agent installation guide for workspace embedding and CLI bootstrap
- add `.memory/config.json` support for overriding the hot promotion target, header, and character budget
- document how to promote pinned hot memory into files like root-level `AGENTS.md`
- clarify that hot memory limits are currently character-based, not token-based

## 0.2.0 (2026-04-25)
- **3-Tier Memory Architecture**: Implemented `.memory/` layout with Hot (`AGENT.md`), Warm (`core/`), and Cold (`episodic/`) tiers.
- **Probabilistic Retrieval**: Replaced substring search with **BM25 ranking** for all recall operations.
- **Weighted Recall**: Warm (Core) memories now outrank Episodic logs in unified search (1.5x score multiplier).
- **Atomic Persistence**: Implemented "write-to-temp-then-rename" for all file operations to prevent corruption.
- **"Forget" Capability**: Added `forget` command and API method to supersede memories by ID.
- **Autonomous Promotion**: Automated detection and promotion of repeated facts from episodic to core memory.
- **Integrity Doctor**: Manifest-based verification of core memory files with SHA256 checksums.
- **Safety Boundary**: Strict sanitation for Hot/Core tiers while allowing raw evidence in Episodic logs.
- **Session Support**: Isolated session memory tracking with cross-session klustering.
- **Decay & Archive**: 30-day unpinned memory archival to prevent "junk drawer" core files.
- **Migration Path**: Built-in `migrate` command to port legacy `MEMORY.md` and `memory/` folders.

## 0.1.0
- initialize standalone `agent-memory-journal` repository
- rename CLI engine to `agent_memory_journal.py`
- decouple memory root from workspace-specific hard-coded paths
- add JSON config for trigger patterns
- add agent-facing `SKILL.md` and installation guidance
- expand README with layout, config, testing, and roadmap
- add CLI, JSON, config, and help/version tests


path: pyproject.toml
content: [build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "agent-memory-journal"
version = "0.2.3"
description = "Lightweight CLI for durable agent memory journaling, recall, and review"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [
  { name = "Miso Lith" }
]
keywords = ["memory", "agent", "cli", "journal", "recall"]
classifiers = [
  "Development Status :: 3 - Alpha",
  "Intended Audience :: Developers",
  "License :: OSI Approved :: MIT License",
  "Programming Language :: Python :: 3",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: 3.12",
  "Operating System :: POSIX :: Linux",
  "Environment :: Console"
]

[project.scripts]
agent-memory-journal = "agent_memory_journal:main"

[tool.setuptools]
py-modules = ["agent_memory_journal"]
packages = ["agent_memory"]


path: docs/agent_install.md
content: # Agent Installation Guide

This project is designed to be embedded into an agent workspace.

## Primary integration surface

The primary integration surface is the Python `Journal` API.

```python
from agent_memory import Journal

j = Journal(root=".")
j.init()
j.note("Observed a durable preference")
hits = j.recall("preference")
```

Use the CLI only as a convenience layer for shell workflows, cron jobs, and manual operations.

## Default installation

1. Clone the repository into the workspace:

```bash
git clone [REDACTED_SECRET]:misolith/agent-memory-journal.git skills/agent-memory
```

2. Ensure the Python package is importable by the agent runtime.

Examples:
- install it into the runtime environment with `pip install .`
- or add the repository root to `PYTHONPATH`

3. Initialize memory in the workspace root.

Python-first:

```python
from agent_memory import Journal
j = Journal(root=".")
j.init()
```

Optional CLI path:

```bash
ln -sf "$PWD/skills/agent-memory/agent_memory_journal.py" "$PWD/bin/agent-memory-journal"
chmod +x "$PWD/skills/agent-memory/agent_memory_journal.py"
agent-memory-journal --root "$PWD" init
```

This creates:

```text
.memory/
├── AGENT.md
├── config.json
├── core/
├── episodic/
├── sessions/
├── index/
└── archive/
```

## Agent bootstrap instructions

For agents, the recommended startup contract is:

1. Read `.memory/AGENT.md`
2. Use `agent-memory-journal search --query "..."` for deep recall
3. Use `agent-memory-journal note "..."` for episodic observations
4. Use `agent-memory-journal remember "..." --category ...` for durable facts
5. Run `agent-memory-journal ingest` during maintenance / heartbeat cycles

## Hot memory target override

By default, pinned hot memory is promoted into:

- `.memory/AGENT.md`

You can override that by editing `.memory/config.json`.

### Default config

```json
{
  "hot_path": "AGENT.md",
  "hot_header": "# AGENT.md",
  "hot_max_chars": 2048
}
```

`hot_max_chars` is enforced as a **character limit**.
It is not token-aware yet. That is a deliberate tradeoff so the project stays dependency-light for now.

### Example: promote hot memory to workspace root `AGENTS.md`

```json
{
  "hot_path": "../AGENTS.md",
  "hot_header": "# AGENTS.md - Auto-generated hot memory",
  "hot_max_chars": 4000
}
```

This makes `ingest` and `rebuild_agent_md()` write the pinned hot set into the workspace root `AGENTS.md` instead of `.memory/AGENT.md`.

## Future enhancement

A later version may support a tokenizer-backed limit like `hot_max_tokens`, but that is not implemented yet.
Until then, document and treat the hot budget as chars, not tokens.

## Recommended pattern

- Keep `.memory/AGENT.md` as the default hot set for most agents.
- Override `hot_path` only when the host framework already has a canonical root memory file, for example:
  - `AGENTS.md`
  - `CLAUDE.md`
  - `SYSTEM.md`

## Safety note

If you override `hot_path` to a root-level file, that file becomes generated output. Do not manually edit it unless your workflow explicitly supports round-tripping generated memory files.


path: README.md
content: # Agent Memory Journal V2 (Zeus)

A robust, 3-tier memory system designed for autonomous agents. It moves beyond simple notes to a structured cognitive architecture featuring Hot, Warm, and Cold storage layers.

## Cognitive Architecture

The system organizes memory into three tiers to balance immediate context availability with long-term retrieval:

1.  **Hot Set (`AGENT.md`)**:
    *   **Always in context**: Intended to be loaded with every agent prompt.
    *   **Size-constrained**: Default 2048 character budget (override via `.memory/config.json`).
    *   **Curated**: Contains only "pinned" core memories that directly influence agent behavior.
    *   **Claim-only rendering**: Metadata (id/state/timestamps) stays in `core/`; the hot file ships just the claim text.
2.  **Warm Memory (`core/`)**:
    *   **Structured facts**: Categorized into `decisions`, `constraints`, `gotchas`, `preferences`, and `capabilities`.
    *   **BM25 Searchable**: Retrieved on-demand using probabilistic ranking.
    *   **Metadata-rich**: Each item has a stable ID, state (active/superseded), creation timestamp, and `last_seen` tracking.
3.  **Cold Archive (`episodic/`)**:
    *   **Raw logs**: Daily chronological record of events and observations.
    *   **Source evidence**: Used for autonomous promotion and auditing.

## Installation

```bash
pip install .
```

For agent embedding and workspace bootstrap, see:

- `docs/agent_install.md`

## CLI Interface

### Recording Observations
```bash
# Add a daily episodic note
agent-memory-journal note "User prefers dark mode for all UI elements" --category preference

# Add a core constraint (pinned to AGENT.md)
agent-memory-journal remember "Never use public wifi for sensitive ops" --category constraint --pinned
```

### Retrieval and Maintenance
```bash
# Search across all tiers (warm hits are weighted 2x to prefer curated memory)
# This also updates 'last_seen' metadata for retrieved core hits.
agent-memory-journal search --query "network safety"

# Supersede an outdated memory
agent-memory-journal forget con-12345

# Perform ingestion (Promote repeated facts & rebuild AGENT.md)
agent-memory-journal ingest

# Verify memory integrity
agent-memory-journal doctor --strict
```

## Python API

The `Journal` class is the primary entry point for agent integration.
The CLI is a convenience wrapper, not the primary interface.

```python
from agent_memory import Journal

# Initialize
j = Journal(root="~/.memory")
j.init()

# Store episodic memory
j.note("Detected high latency on primary gateway")

# Store core memory with supersedes flow
j.remember("Gateway 10.0.0.1 is unstable", category="gotcha", supersedes="got-prev-id")

# Search (returns RecallResult objects)
hits = j.recall("gateway status", tier="all")
for hit in hits:
    print(f"[{hit.tier}] {hit.text} (Score: {hit.score})")
```

## Advanced Features

### Autonomous Promotion
The `ingest` cycle automatically detects facts repeated across different days in the episodic logs. If a fact appears on at least 2 distinct days, it is promoted to the **Warm** (Core) tier as a `decision` or based on keyword triggers.

### Memory Decay & Archiving
To prevent the Warm tier from becoming a "junk drawer," `ingest` runs `archive_unpinned_core` as part of its cycle: unpinned core memories whose `last_seen` (or `created`, as fallback) is older than 30 days are moved into `archive/core/`. Pinned memories are never decayed automatically — they leave the active set only via `forget`.

### Session Management
Short-term session memory is stored in `.memory/sessions/<session_id>.md`.
*   **Lifecycle**: Currently, sessions are persistent files. Future versions will support TTL-based cleanup.
*   **Context**: Agents can use `j.session_note(session_id, text)` to record ephemeral context that doesn't yet belong in the long-term episodic log.

### Integrity & Safety
*   **Doctor**: Uses SHA256 manifests to ensure core memory files haven't been tampered with or corrupted. Manifest entries are refreshed automatically on `remember`/`forget` so the integrity check only flags real out-of-band edits.
*   **Sanitization**: Core memories are screened for prompt-injection patterns and control characters before promotion.

## Directory Layout

```text
.memory/
├── AGENT.md           # Hot Set (Auto-generated by default)
├── config.json        # Hot target + size override config
├── core/              # Warm Memory (decisions, constraints, etc.)
├── episodic/          # Cold Memory (YYYY-MM-DD.md)
├── sessions/          # Session-specific context
├── index/             # Search indexes and manifests
└── archive/           # Decayed core memories
```

## Hot promotion target override

Pinned memories are promoted into `.memory/AGENT.md` by default.

You can override that in `.memory/config.json`:

```json
{
  "hot_path": "../AGENTS.md",
  "hot_header": "# AGENTS.md - Auto-generated hot memory",
  "hot_max_chars": 4000
}
```

`hot_max_chars` is currently a **character budget**, not a token budget.
That choice is intentional for now to avoid adding a tokenizer dependency like `tiktoken`.
A token-aware limit may be added later, but today the implementation enforces chars only.

This is useful when an agent framework expects a root-level memory file like `AGENTS.md`.


path: agent_memory_journal.py
content: #!/usr/bin/env python3
from agent_memory.cli import main

if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)