MemAnchor - parametric memory adapters

LoRA adapters for Qwen/Qwen3-4B-Base. Each one is a memory: a corpus of 759 Wikipedia paragraphs written into the weights, so that asking a question makes the model generate the relevant paragraph. There is no index and no retrieved text at question time - the paragraph is regenerated.

Every adapter here stores the same 759 paragraphs with the same trainer (r=64, alpha=128, 30 epochs, lr 2e-4, max_len 384; the reader is r=16/alpha=32). The only thing that differs between the memory/ folders is what (question -> paragraph) training pairs were written, which is what the study varies.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from huggingface_hub import snapshot_download

path = snapshot_download("0010SS/MemAnchor", allow_patterns="memory/questions-9subject-9mention/*")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Base")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Base", dtype="bfloat16")
mem = PeftModel.from_pretrained(base, f"{path}/memory/questions-9subject-9mention")

ids = tok("Question: Who distributed the film UHF?\nAnswer:", return_tensors="pt")
print(tok.decode(mem.generate(**ids, max_new_tokens=200)[0]))   # recites the paragraph

Disable the adapter (with mem.disable_adapter():) to get the frozen base back - that is how the planner and the untrained reader run in our pipeline.

Contents

folder	what was written per paragraph
memory/questions-9subject-9mention	Generated questions, 9 subject-anchored + 9 mention-anchored per paragraph (12,762 pairs).
memory/questions-5subject-5mention	Same generator and prompts at 5 of each (9,733 pairs) - the matched-budget comparator.
memory/keys-subject	Extracted 'subject ; relation ; object' keys as addresses, 10 per paragraph (7,307 pairs).
memory/keys-mention	Same keys with half anchored on mentioned entities instead of the subject (6,971 pairs).
memory/graph-titleset	Entity -> paragraph-title-set graph written into the weights (23,812 pairs), frozen-test filtered.
memory/graph-titleset-unfiltered	Same graph with node addresses exempt from the frozen-test filter (23,985 pairs).
memory/questions-plus-templated-keys	Row 1's questions plus every extracted key rendered into two templates (25,709 pairs).
memory/memo-synthesis	MeMo's four-stage question synthesis, our storage target (5,984 pairs, 624/759 covered).
reader/v2-hard-distractors	Span-extraction reader, 2,000 train-split items, hard distractors, 4-doc contexts (6,406 pairs).
reader/v1-random-distractors	Earlier reader revision, 2 randomly sampled distractors - the closed-book end-to-end numbers use this.

Evaluation

Two tests, both over the same 759-passage corpus:

• Retrieval (168 already-decomposed sub-questions): does the generated paragraph match the correct one, by best word-overlap over all 759. 84 direct questions name the paragraph's own subject; 84 bridge questions name an intermediate entity.
• End to end (42 original two-hop questions): the model decomposes, queries its own memory twice, and answers. Exact match and contains.

Known caveats

Read these before quoting numbers.

1. The graph-titleset adapters were scored by a different evaluator than the other memories (title-space argmax with set membership over up to 6 fetched passages, versus passage-text top-1 over 759). Their retrieval figures are not directly comparable to the other folders'.
2. questions-9subject-9mention was selected as the best of a ten-point sweep evaluated on the same 42-item test set. Treat its end-to-end number as an optimistic point estimate, not an unbiased one; the full sweep is in the repo.
3. Run-to-run variance on the 42-item test is roughly +/-0.05 (about two items). Differences smaller than that are not differences.
4. Neither questions-* nor memo-synthesis had its training questions filtered against the test questions; the keys-* and graph-* arms did. An audit found 85 of 12,475 generated questions would be flagged, concentrated on 36 paragraphs.

Not included

The supervised upper bound and the frozen-test-filtered arm were trained on a collaborator's cluster and are not in this snapshot. The JANUS memory matrices, SCoL consolidation checkpoints, librarian writer-RL arms, legacy meta-RL per-episode adapters, and LoCoMo per-conversation memories are excluded as out of scope.

Code, training data and all result files: https://github.com/0010SS/MemAnchor

Attribution

Built from MuSiQue (CC-BY-4.0) over Wikipedia text (CC-BY-SA). Training questions were generated with Qwen2.5-7B-Instruct; adapters are LoRA weights over Qwen/Qwen3-4B-Base.