docs/release/technical-blog.md

PRISM-Memory: Turn Conversations Into Durable, Searchable Memory

The Problem

Most long-chat systems do not actually have memory. They have transcript search. That works until someone asks a later question that depends on a hard constraint, a changed plan, a dated fact, or a contradiction that happened months ago.

PRISM-Memory focuses on the part of the stack that usually stays hidden: the step that decides what should become memory at all.

The release model is a 7B adapter that writes short proposition-level memory records from dialogue. Those records are then indexed by a hybrid retrieval stack and used later for recall.

What This Release Shows

The useful result is narrow and practical:

• a 7B open model can replace the GPT-4.1 extraction step in this memory pipeline
• it scores 0.4768 on LongMemEval versus 0.4650 for the GPT-4.1-based PropMem reference
• it scores 0.4981 on LoCoMo versus 0.5360 for that same reference

This is not a claim that a 7B model beats GPT-4.1 everywhere. It is a claim that a 7B model can take over the memory-writing step and stay competitive on the held-out evaluation surface.

Why That Matters

If the memory-writing step is weak, retrieval never gets a clean chance. Important details stay buried inside noisy chat turns.

PRISM-Memory is useful when later questions depend on things like:

• a hard operational limit: 20 GitHub Actions jobs
• a durable preference: aggregated Slack alerts instead of noisy ones
• a status distinction: mTLS is not live yet, it is planned for phase two
• a dated fact: Sam took up painting in May 2023
• a refusal case: the system should answer None instead of inventing a reason for an unsupported guitar story

Those are memory problems, not style problems.

How The System Works

The released system has three pieces.

1. A learned extractor based on Qwen/Qwen2.5-7B-Instruct with LoRA.
2. Post-processing that cleans speaker references and resolves relative time.
3. Hybrid retrieval with BM25, dense retrieval, and reranking.

The extracted propositions are the important interface. They are the memory records the retriever indexes. That keeps the memory store inspectable instead of opaque.

What The Training Data Actually Was

The release data is synthetic.

• 2,329 synthetic training conversations
• 584 held-out synthetic conversations
• 100,427 supervised extraction examples derived from those conversations
• 20,000 supervised examples used for the released adapter

The conversations were designed to stress real memory behaviors:

• new facts introduced in one session and used later
• updated details that should overwrite stale ones
• deleted or invalidated facts that should stop influencing answers
• mixtures of personal details, project facts, preferences, dates, and plans

The labels were GPT-4.1-derived memory-writing targets. No real user chat logs are part of the public release.

What Worked

1. The clean supervised base mattered more than clever add-ons

The release model came from a stable 20,000-example synthetic supervision base. That base was more valuable than trying to patch the model later with many narrow benchmark-specific additions.

2. Hybrid retrieval was part of the result

The release is not just a model story. It is a model-plus-retrieval story. Sparse retrieval kept lexical anchors, dense retrieval recovered semantically close memories, and reranking cleaned the shortlist.

3. Explicit time anchoring helped

The model improved when the memory records carried explicit dates and the system resolved relative references like yesterday or last weekend into normalized anchors.

4. Turn-local extraction was enough

Feeding long recent-context windows into the extractor made it worse. The stronger pattern was local extraction at write time and cross-turn composition later through retrieval.

5. Adversarial precision mattered

The release model kept the best adversarial behavior among the runs considered for public release. That mattered because a memory system that answers unsupported questions confidently is worse than one that refuses.

What Did Not Work

1. Benchmark-style formatting tricks

Trying to train the model toward benchmark-style relative-date outputs hurt more than it helped. It optimized the look of answers instead of the quality of the stored memory.

2. Narrow LoCoMo-style add-ons

Adding targeted benchmark-domain data often bought a small gain in one slice of LoCoMo and then lost balance somewhere else.

3. More noisy supervision was not automatically better

Scaling up original noisy temporal supervision amplified the wrong lesson. The model became more specialized and less balanced.

4. Overtraining past the local optimum

Several follow-on variants nearly matched the final release on one metric, but they usually gave back LongMemEval performance, adversarial precision, or both.

Why Only One Public Model Ships

The repo tried multiple follow-on variants. The nearest internal runner-up nearly tied the released model on overall LoCoMo and disagreed on 152 of the 400 held-out LoCoMo questions, which means the comparison was real.

But the public release decision is simpler than the internal ablation story. One model ships because it had the best overall release profile:

• strongest LongMemEval score
• strongest adversarial behavior
• best total balance across the held-out surface

That is a better public story than shipping several near-tied variants with internal names nobody else should care about.

What Ships

The public release surface is intentionally narrow:

1. PRISM-Memory
2. one released model
3. one extraction skill
4. one Space demo
5. one set of release docs and benchmark artifacts

The broader frontier_memory harness stays in the repo for ongoing research, but the release story stays focused on the memory-writing component that proved worth shipping.