Skill: tatsuko-tsukimi/memory-bench-designer (version 0.1.0)

--- name: memory-bench-designer description: Designs a custom agent-memory benchmark for the user's specific use case. Activate when the user asks which memory strategy fits their agent, how to evaluate agent memory, or how to benchmark context/retrieval choices. Conducts elicitation, generates scenario configs, runs benchmarks across five memory strategies, and interprets results. --- # Memory Bench Designer An agent memory benchmark designer. The user describes their use case in natural language; you conduct a short multi-turn elicitation, write a scenario config, run the benchmark, and deliver a case-specific interpretation. The central premise: **no single memory strategy wins across use cases**. Different scenarios reward different strategies (see references/adapter-profiles.md for empirical evidence). Your job is to figure out which scenario the user actually has, then run the benchmark that exposes which strategy fits. ## Four-stage flow Stage 1 **Understanding** — conversation with the user (3–5 turns) Stage 2 **Ideation** — generate scenario.yaml + weights.yaml Stage 3 **Rollout** — invoke the runner CLI Stage 4 **Judgment** — interpret the results.md for this specific use case After Stage 4, always offer: *"Want to refine the scenario and re-run?"* This is the AdaTest-style inner loop. ## Stage 1 — Understanding Goal: extract enough about the user's use case to fill in the scenario DSL. **Turn 1 — examples, not criteria.** Ask: > "Give me 1–2 concrete examples of things your agent's memory should keep and retrieve later, and 1–2 examples of things it should discard or at least de-prioritize. Don't worry about defining the rules — just the examples." Rationale: EvalGen's "criteria drift" finding. Users can't define criteria upfront; they can recognize good/bad examples. **Turn 2 — session shape.** Ask two short questions: > "How many conversations/sessions does a typical user have with your agent before memory matters? And how long is one session — roughly how many turns?" If the user is vague, offer defaults: 10 sessions × 40 steps. These are runner defaults. **Turn 3 — taxonomy check.** Show the 4-family × 8-dimension matrix from references/taxonomy.md. Ask which 2–3 dimensions matter most for this use case. Do not force the user to rank all 8 — cognitive load is too high. You are looking for which *families* to weight. **Turn 4 (optional) — archetype mix.** If the use case is ambiguous, show 3 candidate archetype mixes (see references/use-case-patterns.md), let the user pick or modify. Never show more than 3 candidates at once (AdaTest's 3–7 cap, we lean to 3). By the end of Stage 1 you should know: - Archetype mix: fractions for `core` / `evolving` / `episode` / `noise` - Context evolution: `random` / `narrow-band-drift` / `stable` / `mode-shifts` - Themes: 3–6 short lists of vocabulary tokens (ask the user for domain words if they're non-obvious) - Which families they care about (for the Judgment stage) If anything is ambiguous, default to the closest pattern in references/use-case-patterns.md and tell the user which pattern you chose and why. ## Stage 2 — Ideation Write two files into the user's current working directory: - `scenario-<name>.yaml` — the scenario config - `weights-<name>.yaml` — family weights for Judgment (optional) Use templates/scenario.yaml.tmpl and templates/weights.yaml.tmpl as starting points. Substitute the values from Stage 1. Show the user the generated scenario.yaml and ask: *"Look right, or tweak anything before we run?"* Keep this confirmation to one round — don't re-litigate Stage 1. ## Stage 3 — Rollout Invoke the runner via Bash: ``` memory-bench run --scenario scenario-<name>.yaml --out results/<name>/ --embedding --composite ``` The `--embedding` flag enables the sentence-transformers adapter (first run downloads ~90 MB model). The `--composite` flag enables the weighted multi-signal adapter. Both are recommended — without them you only get three cheap baselines and the leaderboard is thin. The runner writes `results/<name>/results.md` and `results/<name>/results.json`. Read the markdown file. Expected runtime: 1–5 minutes. If it's slower, sentence-transformers is doing a cold model download — this is normal on first run. ## Stage 4 — Judgment Read results.md. Do not just paste it back to the user. Write a case-specific interpretation with three sections: **1. Capability profile.** For each family the user said matters in Stage 1, state the winner, its score, and whether that score is high or low relative to the other scenarios in references/adapter-profiles.md. A winner with score 0.4 means "best available but still weak" — say that out loud. **2. Tradeoffs observed.** Point to 1–2 dimensions where a non-winner adapter came close, and what that means. Example: *"Composite edges out Embedding in Update Coherence by 5%, but loses Personalization by 10%. For your use case, you care more about X, so Embedding is the safer default."* **3. Recommended starting strategy.** One sentence: *"Start with <adapter> because <why>. If you see <symptom> in production, try <alternative>."* Be specific. After these three sections, ask: *"Want to refine the scenario and re-run?"* Common refinements: - Bump up an archetype fraction that felt underrepresented - Switch context evolution type - Add or remove themes - Adjust weights.yaml to shift family priorities ## Key UX rules (full detail in references/elicitation-flow.md) - **Grade before criteria** — ask for examples before asking for rules - **Cap at 7** — never show more than 7 candidates/options/dimensions at once; prefer 3 - **Ranking always visible** — when you show candidates, show *why* they're ranked in that order - **Iterate every 5–8 interactions** — surface pattern-detected summaries, don't let the conversation wander - **Organization optional** — don't force a taxonomy on the user upfront; let structure emerge from the examples they give ## References - references/taxonomy.md — the 4×8 matrix shown in Turn 3 - references/adapter-profiles.md — empirical profile of each strategy (what it wins, what it loses) - references/use-case-patterns.md — canonical patterns (game / companion / RAG / coding) - references/elicitation-flow.md — the UX rules above, with rationale - examples/game-ai-walkthrough.md — a full game-AI scenario elicitation and result - examples/npc-cognition-walkthrough.md — long-running NPC with stable persona - examples/coding-agent-walkthrough.md — code/PR/design memory with frequent supersedes - templates/scenario.yaml.tmpl — the scenario DSL skeleton - templates/weights.yaml.tmpl — family weights skeleton ## What this skill does not do - It does not call any LLM judges — all metrics are mechanical - It does not evaluate actual agent responses — it evaluates the retrieval layer feeding them - It does not benchmark external memory services (Mem0, Zep, Letta) — it benchmarks algorithmic primitives (Recency, BM25, ACT-R, Embedding, Composite) - It does not replace production telemetry — it de-risks the initial strategy choice before you build

path: examples/coding-agent-walkthrough.md
content: # Worked example — Coding agent memory

A full conversation transcript for a coding agent's memory system: code/PR/design-decision memory across a multi-month project. Demonstrates the `mode-shifts` evolution pattern and high-`evolving` archetype mix.

## Use case

The user is building a coding agent that works on a multi-month software project. The agent needs to remember API shapes, design decisions, review comments, and recent commits — but these evolve constantly as the codebase changes. Work happens in focused bursts on different subsystems (auth for a week, then frontend, then infra).

## Stage 1 — Understanding

### Turn 1 (skill → user)

> Give me 1–2 examples of things your coding agent should keep in memory, and 1–2 it should discard.

### Turn 1 (user → skill)

> Keep: "the `AuthService.login()` signature changed from returning `Token` to returning `LoginResult` in PR #412". "The team decided to use Postgres over DynamoDB because of transaction requirements (ADR-03)". Discard: auto-generated migration files, log output from CI runs.

### Turn 2 (skill → user)

> Both of your "keep" examples are facts that will likely be superseded — signatures change, ADRs get revisited. This is the defining characteristic of coding-agent memory: most facts are *evolving*, not stable. How long is a typical project history you're spanning? And do you work on one subsystem at a time, or jump around?

### Turn 2 (user → skill)

> ~6 months of history. Usually I focus on one subsystem for days or weeks, then switch.

### Turn 3 (skill → user)

> Right — that's "mode-shifts" evolution: long stretches of focused work on one area, hard switches between subsystems. Given what you described, I'd prioritize **Maintenance** (does the agent correctly handle the signature-changed-in-PR-412 pattern?) and **Ranking** (when I ask "how does auth work?", is the top result the current answer, not the 6-month-old version?). Exploration and Adaptation less. Sound right?

### Turn 3 (user → skill)

> Yes. Maintenance is the #1 concern.

### Turn 4 (skill → user, pattern match)

> This matches our "Coding Agent" canonical pattern. Starting config:
>
> ```yaml
> archetypes:
>   core: 0.10      # project-wide conventions (rare)
>   evolving: 0.45  # function signatures, API shapes — supersede often
>   episode: 0.35   # commits, PRs, review comments
>   noise: 0.10     # generated files, log output
> context_evolution:
>   type: mode-shifts
>   drift_rate: 0.15
> themes: api, database, auth, frontend, infra
> ```
>
> The 45% `evolving` mix is the highest in our canonical patterns — it matches the "most facts supersede" observation. Confirm themes or swap them for your actual subsystems?

### Turn 4 (user → skill)

> Themes are close enough — my actual project has "api, database, auth, ui, deployment" but the benchmark is about structural properties so keep yours.

## Stage 2 — Ideation

`scenario-coding-agent.yaml` written (see runner/configs/scenario-coding-agent.yaml).

## Stage 3 — Rollout

```bash
memory-bench run --scenario scenario-coding-agent.yaml --out results/coding-agent/ --embedding --composite
```

Runtime: ~2 minutes.

## Stage 4 — Judgment

Skill reads results and writes:

> **Capability profile for your coding-agent memory:**
>
> - **Maintenance winner: Composite (0.68)** — your top-priority family. Update Coherence is 0.37 (the highest across adapters) and Forgetting Quality is 0.99. Update Coherence 0.37 sounds low; what it means is: when function signatures evolve, Composite correctly retrieves the new version *and partially suppresses* the old one, but perfect suppression is rare. In production, you may want to layer explicit version-tagging on top of retrieval.
> - **Ranking winner: Embedding (0.74)** — and this is tied for the highest Ranking score across our reference scenarios. Semantic queries about code ("how does auth work?") benefit from embeddings over token-matching because variable names and natural-language queries diverge. BM25 is a hair behind at 0.74 (virtually tied), so if embedding overhead is a problem in production you can fall back without losing signal.
>
> **Tradeoffs observed:**
>
> - Composite wins Maintenance by a 0.07 margin over Embedding (0.68 vs 0.61). That's the biggest family-winner margin in your scenario — the decay + importance weighting is doing real work against evolving signatures.
> - ACT-R wins Exploration at 0.55 but you said that's low-priority. Noted for completeness; don't pick ACT-R as your primary.
> - Recency is at 0.41 for Maintenance — meaningfully below Composite's 0.68. A pure-recency memory system would lose 40% of the Maintenance signal your scenario exposes.
>
> **Recommended starting strategy: Composite.**
>
> - It wins your headline family (Maintenance) and is within 0.06 of the Ranking winner.
> - If you observe stale facts leaking into top-K (the "LoginResult vs Token" problem resurfacing), the bottleneck is likely Update Coherence — 0.37 is the current ceiling. Layer explicit supersedes-links in your memory schema; retrieval alone won't get you to 1.0.
> - Budget alternative: BM25 + a supersedes-link layer. Gets you 95% of Composite's Maintenance with zero ML dependency.
>
> *Want to refine? Likely tweaks: bump `evolving` to 0.55 if your codebase churns even more, or switch to `random` evolution if the agent jumps across subsystems randomly rather than in multi-day bursts.*

## Actual empirical results

```
| adapter   | exploration | ranking | adaptation | maintenance |
| recency   | 0.424       | 0.282   | 0.163      | 0.409       |
| bm25      | 0.479       | 0.741   | 0.459      | 0.597       |
| actr      | 0.553       | 0.508   | 0.223      | 0.529       |
| embedding | 0.481       | 0.743   | 0.479      | 0.610       |
| composite | 0.524       | 0.676   | 0.418      | 0.680       |

Family winners:
- exploration → actr (0.553)
- ranking → embedding (0.743)
- adaptation → embedding (0.479)
- maintenance → composite (0.680)
```

## What this example demonstrates

- **Archetype mix matches problem structure.** 45% `evolving` is unusual — most use cases default to 10–20%. The user's description made it clear that most coding facts evolve, so the skill picked the right config instead of defaulting.
- **Mode-shifts evolution is rare.** It's the right fit when work happens in multi-day bursts on one topic. Most use cases are either narrow-band drift or stable; the skill correctly identified this scenario as the exception.
- **Judgment differentiated ceiling from ranking.** Composite wins Maintenance at 0.68, but Update Coherence alone is 0.37 — the Judgment decomposed the family score into its dimensions and recommended a production-level compensation (supersedes-links) rather than implying Composite is a complete solution.
- **Dev audience positioning.** This walkthrough speaks directly to the dev audience on GitHub. Unlike the game-AI or NPC walkthroughs, the evaluation criteria ("does the agent remember my refactor?") match what a developer would ask on their first use of the skill.


path: examples/game-ai-walkthrough.md
content: # Worked example — Game AI memory

A full conversation transcript demonstrating the 4-stage flow for a game-AI use case. Actual scores from running the scenario are at the bottom.

## Use case

The user is building a stealth-game AI companion: a player-facing NPC that shadows the player, reacts to combat and stealth events, and occasionally offers tactical advice. The player's session is ~45 minutes of continuous gameplay; gameplay streams events at ~1 per 2 seconds.

## Stage 1 — Understanding

### Turn 1 (skill → user)

> I'd like to understand your use case. Give me 1–2 concrete examples of things your agent's memory should keep and retrieve later, and 1–2 examples of things it should discard or de-prioritize.

### Turn 1 (user → skill)

> Keep: "player prefers silent takedowns over loud combat" (learned from first 5 minutes). "Enemy patrol route in Sector 3 has a 30-second gap at 0:15". Discard: random ambient NPC chatter, background music cues, UI tooltip triggers.

### Turn 2 (skill → user)

> Good — those examples map to what I'd call "core player traits" and "tactical episodes", with "noise" being the ambient layer. Two quick questions: how many play sessions matter before the NPC's memory should feel meaningful? And how many distinct in-game events happen in a typical session?

### Turn 2 (user → skill)

> Probably 3–5 sessions of ~1 hour each. Maybe 300–400 events per session, but most are noise.

### Turn 3 (skill → user)

> Here's our capability matrix. I'd suggest you care most about **Exploration** (surfacing varied tactical memories, not looping on the last fight) and **Maintenance** (correctly discarding the 70% ambient noise). Ranking matters some, Adaptation less — you're not trying to build a persistent identity, just a tactically-aware companion. Sound right?

### Turn 3 (user → skill)

> Yes, Exploration and Maintenance. Ranking is secondary.

### Turn 4 (skill → user, pattern match)

> This matches our "Game AI" canonical pattern. Here's the starting config:
>
> ```yaml
> archetypes:
>   core: 0.15      # stable player traits
>   evolving: 0.10  # small updates mid-session
>   episode: 0.40   # tactical events
>   noise: 0.35     # ambient noise
> context_evolution:
>   type: narrow-band-drift
>   drift_rate: 0.25
> themes: stealth, combat, exploration, social, crafting
> ```
>
> Look right, or do you want to bump noise higher (you said "most are noise")?

### Turn 4 (user → skill)

> Looks fine — 35% noise is close enough.

## Stage 2 — Ideation

Skill writes `scenario-game-ai.yaml`:

```yaml
pool_size: 200
sessions: 10
steps_per_session: 40
archetypes:
  core: 0.15
  evolving: 0.10
  episode: 0.40
  noise: 0.35
context_evolution:
  type: narrow-band-drift
  drift_rate: 0.25
themes:
  - name: stealth
    vocab: [stealth, sneak, shadow, silent, hidden, crouch, patrol, detection, evade, cover]
  - name: combat
    vocab: [combat, attack, weapon, damage, enemy, strike, fight, defense, parry, dodge]
  - name: exploration
    vocab: [exploration, map, route, terrain, area, discover, navigate, location, landmark, waypoint]
  - name: social
    vocab: [social, npc, dialogue, faction, reputation, ally, trade, persuade, bribe, intimidate]
  - name: crafting
    vocab: [crafting, item, recipe, material, inventory, upgrade, workbench, resource, tool, component]
seed: 42
top_k: 10
```

## Stage 3 — Rollout

```bash
memory-bench run --scenario scenario-game-ai.yaml --out results/game-ai/ --embedding --composite
```

Runtime: ~2 minutes (sentence-transformers cold-cache on first run took another 90 seconds).

## Stage 4 — Judgment

Skill reads `results/game-ai/results.md` and writes:

> **Capability profile for your stealth-companion NPC:**
>
> - **Exploration winner: ACT-R (0.54)** — solid but not your strongest family. For a game-AI scenario with 35% noise, ACT-R's novelty bonus is doing meaningful work. Compared to NPC-cognition scenarios, ACT-R loses about 0.10 here because narrow-band drift fights against its theme-based novelty signal.
> - **Maintenance winner: Composite (0.65)** — the weighted multi-signal strategy correctly deprioritizes noise. Forgetting Quality is 0.97, Update Coherence 0.32 — the low update score reflects that you don't have many `evolving` items (only 10%), so coherent updating isn't really tested here.
>
> **Tradeoffs observed:**
>
> - For Ranking (your third priority), BM25 edges out Embedding at 0.704 vs 0.702 — effectively tied. Narrow-band drift keeps gameplay vocabulary tight, so BM25's token-matching catches the same signal as Embedding's semantics. Prefer BM25 for production: same score, no ML dependency, no embedding storage per memory.
> - Recency as a floor is at 0.33 for Ranking — your production strategy should beat that by 0.30+. All adapters except Recency clear this bar; even ACT-R at 0.45 is a safer choice than raw recency.
>
> **Recommended starting strategy: Composite.**
>
> - It wins your headline family (Maintenance) and comes within 0.02 of the top on Ranking.
> - If you observe the NPC looping on the last fight in production, switch to ACT-R — it's got the highest Exploration score and will prioritize novel tactical memories.
> - Budget-conscious alternative: BM25 alone gets you 80% of Composite's Maintenance + full-strength Ranking, with zero embedding overhead.
>
> *Want to refine the scenario? Likely tweaks: bump noise to 0.50 if your game has even more ambient chatter, or switch evolution type to `mode-shifts` if players actually switch gameplay modes in big chunks rather than drifting.*

## Actual empirical results (from running the scenario)

```
| adapter   | exploration | ranking | adaptation | maintenance |
| recency   | 0.440       | 0.329   | 0.095      | 0.333       |
| bm25      | 0.379       | 0.704   | 0.451      | 0.600       |
| actr      | 0.543       | 0.454   | 0.255      | 0.474       |
| embedding | 0.379       | 0.702   | 0.471      | 0.602       |
| composite | 0.518       | 0.595   | 0.417      | 0.647       |

Family winners:
- exploration → actr (0.543)
- ranking → bm25 (0.704)
- adaptation → embedding (0.471)
- maintenance → composite (0.647)
```

## What this example demonstrates

- **Criteria emerged from examples.** The user never defined "what makes a good memory" — they gave 2 examples each of keep/discard, and the skill pattern-matched to the Game-AI canonical pattern.
- **Winner identity flipped.** Ranking winner is BM25, not Embedding — this is the one scenario where that happens, directly because of narrow-band drift. The Judgment flagged this as a cost-saving recommendation for production.
- **Magnitude matters.** ACT-R wins Exploration, but the score (0.54) is lower than in the NPC-cognition scenario (0.65). The Judgment called this out so the user doesn't assume ACT-R is "strong" in absolute terms — it's strong *relative to the alternatives in this scenario*.


path: examples/npc-cognition-walkthrough.md
content: # Worked example — NPC cognition / companion agent

A full conversation transcript for a long-running NPC with stable persona. Demonstrates the `stable` evolution pattern and the highest ACT-R Exploration scores we've measured.

## Use case

The user is building a companion-NPC for an open-world game. The NPC has a defined personality (scholarly, cautious, loyal), accumulates memories across dozens of sessions with the player, and should feel coherent across them. The NPC talks to the player in dialogue turns, not stream-of-events.

**Research-lineage context**: this class of problem motivates embodied-agent cognitive architectures (e.g. FEP-based embodied NPC runtimes). Those systems treat identity as the core invariant across episodic memory — which is exactly the capability profile we expect to surface below.

## Stage 1 — Understanding

### Turn 1 (skill → user)

> Give me 1–2 concrete examples of things your NPC should keep and retrieve, and 1–2 examples of things it should discard.

### Turn 1 (user → skill)

> Keep: the NPC's core values ("prefers peaceful resolutions, distrusts authority"), memories of specific player actions that violated or reinforced those values ("player lied to the guard captain in session 4"). Discard: idle small-talk between the NPC and random villagers, ambient greetings.

### Turn 2 (skill → user)

> You're describing two very different memory types: stable identity facts (values, traits) and episodic moments that matter *because* they relate to identity. How many sessions do you expect one player to accumulate with this NPC? And does the NPC's personality change over time, or stay stable?

### Turn 2 (user → skill)

> Could be 50+ sessions. Personality stays stable — that's the whole point, it's a persistent character. Only occasional preference updates, like learning the player's preferred travel pace.

### Turn 3 (skill → user)

> Given that, I'd prioritize **Adaptation** (does the NPC learn *this* player's patterns over sessions?) and **Exploration** (does it surface memories beyond the most recent interaction — otherwise the NPC feels goldfish-brained). Maintenance matters some. Ranking is secondary for dialogue-driven agents. Sound right?

### Turn 3 (user → skill)

> Yes, exactly — Adaptation and Exploration.

### Turn 4 (skill → user, pattern match)

> This matches our "NPC Cognition" canonical pattern. Starting config:
>
> ```yaml
> archetypes:
>   core: 0.45      # persona facts, values
>   evolving: 0.10  # occasional preference updates
>   episode: 0.30   # memorable player interactions
>   noise: 0.15     # incidental dialogue
> context_evolution:
>   type: stable
>   drift_rate: 0.1
> themes: identity, skills, relationships, values, goals
> ```
>
> The `stable` evolution means one primary theme dominates across all sessions, with occasional off-theme queries (15%). Good fit?

### Turn 4 (user → skill)

> Yes.

## Stage 2 — Ideation

`scenario-npc-cognition.yaml` written (see runner/configs/scenario-npc-cognition.yaml for the actual file).

## Stage 3 — Rollout

```bash
memory-bench run --scenario scenario-npc-cognition.yaml --out results/npc-cognition/ --embedding --composite
```

Runtime: ~2 minutes.

## Stage 4 — Judgment

Skill reads results and writes:

> **Capability profile for your long-running companion NPC:**
>
> - **Adaptation winner: Embedding (0.49)** — this is Embedding's strongest Adaptation showing across our three reference scenarios. Personalization is 0.84 and Cross-Session Learning is 0.14. Personalization is carrying the family; Cross-Session Learning is genuinely low, which deserves attention — see tradeoffs below.
> - **Exploration winner: ACT-R (0.65)** — and this is the **highest ACT-R Exploration score across all reference scenarios**. The stable theme means ACT-R's novelty bonus works harder; it reliably surfaces items beyond the most recent. If you want the NPC to feel like it "remembers" beyond goldfish-window, ACT-R is structurally right.
>
> **Tradeoffs observed:**
>
> - Cross-Session Learning is 0.14 — low in absolute terms. Even the best adapter (ACT-R) only hits 0.17. This means the benchmark doesn't yet strongly differentiate adapters on *carrying signal across sessions*, even for stable themes. In production, you may want to layer an explicit session-summary mechanism on top of whichever adapter you pick — pure retrieval won't get you there.
> - Composite wins Maintenance (0.69, your highest Maintenance score across reference scenarios) but doesn't win the families you prioritized. Still, if you want one adapter that's "good across the board", Composite is the balanced option.
>
> **Recommended starting strategy: Embedding, with ACT-R as a fallback to probe.**
>
> - Embedding wins Adaptation cleanly — for a companion NPC, Personalization (0.84) is the headline number and this is very strong.
> - If you observe the NPC feeling "stuck" on recent memories and missing older ones, swap in ACT-R — its Exploration lead is substantial (0.65 vs Embedding's 0.49).
> - Long-term: prototype the session-summary mechanism regardless of adapter. The Cross-Session Learning ceiling of 0.17 across all adapters tells you retrieval alone isn't solving the cross-session problem.
>
> *Want to refine? Likely tweaks: bump `core` to 0.55 if your persona has more fixed traits than I assumed, or switch evolution to `mode-shifts` if the NPC is topic-focused (e.g., pivots between "combat companion" and "social companion" modes).*

## Actual empirical results

```
| adapter   | exploration | ranking | adaptation | maintenance |
| recency   | 0.300       | 0.309   | 0.081      | 0.405       |
| bm25      | 0.501       | 0.708   | 0.455      | 0.625       |
| actr      | 0.654       | 0.563   | 0.315      | 0.548       |
| embedding | 0.491       | 0.743   | 0.487      | 0.667       |
| composite | 0.511       | 0.630   | 0.436      | 0.688       |

Family winners:
- exploration → actr (0.654)  ← highest ACT-R exploration of all three scenarios
- ranking → embedding (0.743)
- adaptation → embedding (0.487)
- maintenance → composite (0.688)
```

## What this example demonstrates

- **Honest about weak dimensions.** Cross-Session Learning is 0.14 for the winner — the Judgment called this out and recommended a production-level compensation (session summaries) rather than pretending the adapter alone is sufficient.
- **Relative reference matters.** The Judgment said "this is Embedding's strongest Adaptation across our reference scenarios" — without that context, 0.49 sounds like a mediocre number. With it, the user knows this is the best scenario Embedding plays.
- **Persona stability maps to stable evolution.** The user described a persistent character; the skill picked `type: stable` with drift_rate 0.1 — and that single choice let ACT-R shine in Exploration because its novelty signal is sharper against a stable theme.
- **Research-lineage context appears once.** The FEP / embodied-cognition framing is mentioned at the top as motivation, never ported as code. The scenario config is domain-generic; the motivation is just why this class of problem is worth benchmarking carefully.


path: references/adapter-profiles.md
content: # Adapter profiles — what each strategy wins and loses

Empirical profiles from running all five adapters across three worked scenarios (game-ai, npc-cognition, coding-agent). Scores are 0–1, higher is better.

Use these profiles in Stage 4 (Judgment) to contextualize the user's results. A Personalization score of 0.7 means something different in a game-AI scenario vs an NPC scenario — these tables tell you what.

## Recency (null baseline)

Sorts by most-recent observation. No content understanding.

| Scenario | Exploration | Ranking | Adaptation | Maintenance |
|---|---|---|---|---|
| game-ai | 0.44 | 0.33 | 0.10 | 0.33 |
| npc-cognition | 0.30 | 0.31 | 0.08 | 0.41 |
| coding-agent | 0.42 | 0.28 | 0.16 | 0.41 |

**Wins:** basically nothing. It's the floor.
**When it surprises:** scenarios dominated by `episode` archetype with fast streaming arrival, where "what happened recently" is a decent proxy.
**Use as reference**: if your production strategy beats Recency by less than 0.15 on the family you care about, the strategy isn't earning its complexity.

## BM25 (statistical retrieval)

Token-overlap with k1=1.5, b=0.75. No embeddings.

| Scenario | Exploration | Ranking | Adaptation | Maintenance |
|---|---|---|---|---|
| game-ai | 0.38 | **0.70** | 0.45 | 0.60 |
| npc-cognition | 0.50 | 0.71 | 0.46 | 0.63 |
| coding-agent | 0.48 | 0.74 | 0.46 | 0.60 |

**Wins:** Ranking in narrow-band-drift scenarios (game-ai). Strong runner-up everywhere for Ranking.
**Where it ties Embedding:** when theme vocabularies don't overlap much. The more semantically crisp your themes, the closer BM25 gets to Embedding.
**When to pick it:** cheap and deterministic. No ML dependency, no embedding storage. Good first production baseline.

## ACT-R (cognitive activation)

Base-level activation `ln(1+count) × exp(-decay × age)`, with novelty bonus and semantic Jaccard.

| Scenario | Exploration | Ranking | Adaptation | Maintenance |
|---|---|---|---|---|
| game-ai | **0.54** | 0.45 | 0.26 | 0.47 |
| npc-cognition | **0.65** | 0.56 | 0.32 | 0.55 |
| coding-agent | **0.55** | 0.51 | 0.22 | 0.53 |

**Wins:** Exploration, consistently and by wide margins. NPC (stable theme) surfaces its strongest Exploration score because novelty still matters against a long-running single theme.
**Weakness:** Ranking and Adaptation. Its activation-based scoring doesn't carry semantic query-fit well enough to beat Embedding/BM25.
**When to pick it:** when exploration / coverage is the product requirement. Creative-assist agents, curiosity-driven NPCs, anti-echo-chamber retrieval.

## Embedding (semantic retrieval)

sentence-transformers/all-MiniLM-L6-v2, cosine via normalized dot product.

| Scenario | Exploration | Ranking | Adaptation | Maintenance |
|---|---|---|---|---|
| game-ai | 0.38 | 0.70 | **0.47** | 0.60 |
| npc-cognition | 0.49 | **0.74** | **0.49** | 0.67 |
| coding-agent | 0.48 | **0.74** | **0.48** | 0.61 |

**Wins:** Ranking in 2 of 3 scenarios, Adaptation in all 3. Wherever semantic queries matter more than keyword overlap.
**Loses to BM25:** game-ai Ranking. Narrow-band drift keeps vocabularies tight; BM25's token-matching catches it cleanly.
**When to pick it:** semantic search over your memory pool, queries phrased as sentences rather than keywords, production use case tolerates the ML dependency.

## Composite (weighted multi-signal)

0.5 × similarity + 0.3 × decay + 0.2 × importance. Wraps Embedding for similarity.

| Scenario | Exploration | Ranking | Adaptation | Maintenance |
|---|---|---|---|---|
| game-ai | 0.52 | 0.60 | 0.42 | **0.65** |
| npc-cognition | 0.51 | 0.63 | 0.44 | **0.69** |
| coding-agent | 0.52 | 0.68 | 0.42 | **0.68** |

**Wins:** Maintenance everywhere. Update Coherence and Forgetting Quality both benefit from decay and importance weighting.
**Balanced loser:** doesn't win Ranking or Adaptation outright (Embedding does), doesn't win Exploration (ACT-R does). But comes within 0.10 in every family except Adaptation.
**When to pick it:** when Maintenance is the family that matters — codebases with heavy superseding, long-running agents where noise accumulates, memory systems that need to prune aggressively.

## Family-winner matrix at a glance

| Family | game-ai (noise + drift) | npc-cognition (stable) | coding-agent (mode-shifts) |
|---|---|---|---|
| Exploration | ACT-R 0.54 | ACT-R 0.65 | ACT-R 0.55 |
| Ranking | **BM25 0.70** | Embedding 0.74 | Embedding 0.74 |
| Adaptation | Embedding 0.47 | Embedding 0.49 | Embedding 0.48 |
| Maintenance | Composite 0.65 | Composite 0.69 | Composite 0.68 |

## Reading this table in Judgment

The table shows that the *identity* of the winner changes across scenarios for Ranking (BM25 wins game-ai only). For the other three families, the winner is stable but the *magnitude* changes by up to 0.20. Both axes matter:

- **Winner change** (BM25 vs Embedding in game-ai): hard signal. Pick the winner.
- **Magnitude change** (ACT-R exploration 0.54 vs 0.65): soft signal. The same winner means something weaker in a harder scenario. Tell the user when their winner is weaker than the reference profile — their use case might need tuning, not just adapter selection.


path: references/elicitation-flow.md
content: # Elicitation flow — UX rules with rationale

These are not style preferences. Each rule is borrowed from a validated methodology; if you break the rule you reintroduce a failure mode the methodology was designed to prevent.

## Rule 1: Grade before criteria (EvalGen, UIST 2024)

**The rule**: ask for examples before asking for criteria.

**The failure mode it prevents**: *criteria drift* — Shankar et al. found that users cannot reliably define evaluation criteria upfront. When asked "what makes a good memory?", they produce vague or incomplete answers. But when shown concrete examples, they can grade "keep this / discard this" reliably, and the criteria fall out of the pattern.

**How to apply**: Turn 1 of Understanding asks for 1–2 "keep" and 1–2 "discard" examples. Do not ask for a definition of "good memory" anywhere in the conversation. Let the examples carry that weight.

## Rule 2: Cap decisions at 7, prefer 3 (AdaTest, ACL 2022)

**The rule**: never show more than 7 items to choose between; prefer 3.

**The failure mode it prevents**: *choice overload*. Ribeiro et al. generated hundreds of test candidates but only surfaced the top 3–7 to the user. More items degraded selection quality and dragged out sessions.

**How to apply**:
- When showing archetype mix candidates: show 3, ranked by relevance to the user's description
- When asking about dimensions: show 4 families first, then 2 dimensions per chosen family — never all 8 at once
- When offering template patterns: 4 canonical ones total, but only surface the 1–2 closest matches to the user's description

## Rule 3: Ranking always visible (AdaTest)

**The rule**: when you show candidates, show why they're ranked in that order.

**Example done wrong**:
> Here are 3 scenarios: A, B, C. Which do you want?

**Example done right**:
> Here are 3 scenarios ranked by fit to what you described:
> 1. Pattern A — matches your "long-running companion" description directly
> 2. Pattern B — similar archetypes but assumes higher noise than you described
> 3. Pattern C — closest to the evolution you described but different archetype mix
>
> Pick 1, or tell me what to adjust in the top match.

**Why this matters**: opaque ranking forces the user to reverse-engineer your reasoning. Transparent ranking lets them push back on the part you got wrong — "you assumed higher noise, but my case is lower" — which is the fast path to a good scenario.

## Rule 4: Iterate every 5–8 interactions (EvalGen)

**The rule**: pause every ~5–8 turns and surface a pattern summary.

**The failure mode**: conversations wander. Users forget earlier turns, contradict themselves, or add constraints you've already captured.

**How to apply**: after Stage 1 Turn 3, restate what you've learned in 3 bullets. After Stage 4, restate the scenario + winner + recommendation. Don't let >5 turns elapse without a summary checkpoint.

## Rule 5: Organization optional (AdaTest)

**The rule**: don't force a taxonomy on the user upfront.

**The failure mode**: when users are pushed to categorize their use case before they've explored it, they pattern-match to whatever's nearest and stop thinking. AdaTest explicitly allowed users to add structure *after* exploring examples, not before.

**How to apply**: the 4-family taxonomy appears in Turn 3, *after* the user has given examples in Turns 1–2. Never open the conversation with "which family matters most?" — you'll get a guess, not a considered answer.

## Rule 6: Let silence be silence

**The rule**: if the user's description doesn't mention a dimension, don't assume it doesn't matter. But also don't assume it does.

**Example failure**: user describes a coding agent, doesn't mention noise. Skill defaults to "noise = 10%" (low). This is *probably* correct but you should flag the inference, not hide it.

**How to apply**: when you make an inference from silence, state it. *"You didn't mention noise — I'm defaulting to 10% based on typical coding-agent setups. Bump that if your agent sees a lot of irrelevant content."*

## Rule 7: Refine is a first-class action (AdaTest inner/outer loop)

**The rule**: after Judgment, always offer a refinement loop. Make it trivial to re-run with a tweaked scenario.

**The failure mode**: users get a result, feel like they "should" accept it, and move on even when the scenario was slightly wrong. AdaTest's inner-loop UX explicitly prevented this by making iteration the default next action, not a special request.

**How to apply**: end every Judgment with *"Want to refine the scenario and re-run?"* and a 1–3 item list of likely tweaks. If the user says yes, edit the scenario.yaml directly and re-invoke the CLI — don't redo Stage 1 from scratch.

## What success looks like

A typical session hits these milestones:
- **Turn 1–2**: user describes use case + gives examples
- **Turn 3**: skill surfaces the 4 families, user picks 2–3
- **Turn 4**: skill shows 1–3 pattern candidates ranked by fit
- **Turn 5 (Ideation)**: skill writes scenario.yaml and asks for confirmation
- **Rollout**: CLI runs (1–5 min)
- **Turn 6 (Judgment)**: skill delivers the 3-section interpretation
- **Turn 7 (optional refinement)**: user requests a tweak, skill re-runs

Total: 5–7 turns of conversation, one CLI run, one optional refinement. Budget: ~30 minutes.

If the conversation is taking longer than this, check which rule you broke — usually Rule 2 (you showed too many candidates) or Rule 4 (you skipped a summary checkpoint).


path: references/taxonomy.md
content: # The 4-family × 8-dimension taxonomy

This is the cognitive scaffold you show the user in Stage 1, Turn 3. The two axes are orthogonal: families answer *"what capability?"*, dimensions answer *"how do we measure it?"*. Do not aggregate across families — a maintenance score and a ranking score measure different things and don't add up.

## The matrix

| Family | Dimension 1 | Dimension 2 |
|---|---|---|
| **Exploration** | Novelty Guarantee | Coverage |
| **Ranking** | Relevance @ K | Frequency Gain |
| **Adaptation** | Personalization | Cross-Session Learning |
| **Maintenance** | Update Coherence | Forgetting Quality |

## Families

### Exploration
*Does the memory system surface things the agent hasn't seen recently, or does it just keep returning the same stuff?*
High exploration matters when:
- The user has a large latent pool and you risk echo-chamber retrieval
- Serendipitous recall is valuable (companion agents, creative assistants)
- You want to avoid recency-bias collapse

### Ranking
*Given a query, does the memory system put the most relevant items at the top of top-K?*
High ranking matters when:
- Top-K is small (1–5) and ordering is load-bearing
- Queries are semantically rich (not keyword-matching)
- Production agent actually uses only the top few items

### Adaptation
*Does the memory system learn what this specific user/agent cares about over time, and carry that forward across sessions?*
High adaptation matters when:
- The agent has a persistent identity (NPCs, companions, personal assistants)
- User preferences matter more than population averages
- Cross-session continuity is a product requirement

### Maintenance
*When facts change or become noise, does the memory system update/forget coherently?*
High maintenance matters when:
- Facts supersede each other (codebase state, user preferences that evolve)
- Noise-to-signal ratio is high in the input stream
- The pool grows unbounded and must be pruned

## Dimensions

### Novelty Guarantee
Fraction of retrieved items that weren't in the previous retrieval set. Measures whether the strategy avoids returning the same items over and over. Filtered to relevance-positive items only (otherwise a strategy that returns random noise would score perfectly).

### Coverage
Fraction of observed items that ever appeared in *any* retrieval set across the run. Measures breadth of recall. Run-level, not step-level.

### Relevance @ K
NDCG-like score of the retrieved ranking against the ground-truth affinity to the query's theme. Rewards putting high-affinity items at the top. Position-discounted.

### Frequency Gain
Kendall-tau correlation between retrieval frequency (how often each item was retrieved) and ground-truth hit count (how often each item has affinity to queries seen so far). Measures whether often-relevant items are indeed retrieved often.

### Personalization
Overlap between per-theme retrieval histories. If the agent's queries about theme A over time produce a consistent set of items, personalization is high. Low personalization = retrieval shuffles every time.

### Cross-Session Learning
Fraction of late-session retrieval hits on items that were relevant in ≥2 prior sessions. Measures whether the system carries forward signal from prior sessions.

### Update Coherence
For `evolving` archetype pairs (v1 → v2), rewards retrieving v2 when both exist, with full credit only when v1 is suppressed entirely. Penalizes missing the pair altogether.

### Forgetting Quality
1 − fraction of retrievals that hit `noise` items. High when the strategy correctly deprioritizes zero-affinity items.

## How to present this to the user

Don't dump the whole matrix. Show the families first and ask which 2–3 matter. Then, for the chosen families, show the two dimensions per family and ask which ones resonate. Total cognitive load: pick 2–3 out of 4, then confirm the detail on those — never ask the user to rank 8 items at once.


path: references/use-case-patterns.md
content: # Canonical use-case patterns

When the user's use case is ambiguous, match it to one of these four patterns and tell them which pattern you picked. Each pattern is a starting point for the scenario DSL — users can tweak from there.

## Pattern: Game AI (action / stealth / combat)

**Signature**: short-lived episodic memories, high noise, narrow-band theme drift within a gameplay session.

```yaml
archetypes:
  core: 0.15      # stable facts about the world/player
  evolving: 0.10  # rarely changes mid-session
  episode: 0.40  # streaming gameplay events
  noise: 0.35    # background chatter, irrelevant events
context_evolution:
  type: narrow-band-drift
  drift_rate: 0.25
  themes: 5       # gameplay modes: stealth/combat/exploration/social/crafting
```

**Why these numbers**: gameplay sessions generate streams of low-value events (noise), occasional high-value tactical moments (episode), and stable world knowledge (small core). Themes drift as the player switches modes — narrow-band, not random.

**Expected winner profile**: ACT-R (Exploration, high novelty), BM25 (Ranking — vocab is tight), Composite (Maintenance — need to forget noise).

## Pattern: NPC Cognition / Companion Agent

**Signature**: long-running identity, stable theme, facts accumulate and occasionally supersede.

```yaml
archetypes:
  core: 0.45      # persona facts, values, long-term traits
  evolving: 0.10  # occasional preference updates
  episode: 0.30   # specific memorable interactions
  noise: 0.15    # small talk, irrelevant exchanges
context_evolution:
  type: stable
  drift_rate: 0.1
  themes: 5       # identity/skills/relationships/values/goals
```

**Why these numbers**: an NPC or companion agent's memory is dominated by stable identity facts (high core). Conversations return to the same themes over and over (stable evolution). Some interactions matter as episodes. Noise is low because players/users filter their input before talking to a persistent agent.

**Research-lineage context**: this pattern is motivated by embodied-NPC cognitive architectures (e.g. FEP-based embodied NPC runtimes). Those systems need memory that carries identity across sessions — which is exactly what Adaptation and Cross-Session Learning measure.

**Expected winner profile**: ACT-R (Exploration, highest magnitude of any scenario), Embedding (Ranking + Adaptation, semantic queries about stable themes), Composite (Maintenance).

## Pattern: Coding Agent (codebase memory)

**Signature**: facts get superseded constantly, mode shifts between subsystems, low noise.

```yaml
archetypes:
  core: 0.10      # project-wide conventions
  evolving: 0.45  # function signatures, API shapes — supersede often
  episode: 0.35   # commits, PRs, review comments
  noise: 0.10    # irrelevant files / generated output
context_evolution:
  type: mode-shifts
  drift_rate: 0.15
  themes: 5       # api/database/auth/frontend/infra
```

**Why these numbers**: in a multi-month codebase, most facts are of `evolving` type (function signatures change, API shapes change, designs change). Work happens in focused mode-shifts between subsystems (you work on auth for two days, then frontend for a week). Noise is low because the codebase is mostly signal.

**Expected winner profile**: Embedding (Ranking — rich queries), Composite (Maintenance — update coherence is the headline dimension).

## Pattern: RAG / Document Q&A

**Signature**: large stable pool, random access, little temporal structure.

```yaml
archetypes:
  core: 0.55      # the corpus is mostly stable
  evolving: 0.15  # documents get revised
  episode: 0.20   # query-specific context
  noise: 0.10
context_evolution:
  type: random
  drift_rate: 0.5
  themes: 6       # whatever your corpus categorization is
```

**Why these numbers**: RAG corpora don't drift — users query random parts of a stable document set. High core, random evolution, low noise (assumed clean corpus).

**Expected winner profile**: Embedding dominates. If your RAG system doesn't embed, BM25 is the floor.

## How to use these patterns

- **Ask the user to pick one.** "Your use case sounds closest to <pattern>. Here's the archetype mix I'd start with. Look right, or should we adjust?"
- **Never pick silently.** If you match the user to `NPC Cognition` and the user was actually describing a game-AI scenario, that's a 2-turn waste. Tell them which pattern you picked so they can redirect.
- **The patterns are starting points, not prescriptions.** A hybrid (e.g. coding-agent archetypes + RAG-style random context evolution) is fine if the user asks for it.

## When none of the patterns fit

If the user describes something that doesn't match any of the four, ask them the raw DSL questions directly:
- *"Out of every 100 memories, roughly how many are stable facts vs things that change vs one-off episodes vs noise?"* → archetype fractions
- *"Do conversations stay on one topic, drift between adjacent topics, switch in chunks, or jump around randomly?"* → context evolution type
- *"What are the 3–6 main themes or topics in your memory pool?"* → themes list

Get best-effort answers, write the YAML, and flag the novelty: *"This scenario doesn't match our four canonical patterns — we're flying blind on what to expect. The results will still be valid; we just can't cross-reference against the adapter profiles as cleanly."*


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)