Lngram v2 memory models (ANLP project, Morbius)
Seven 24-layer, ~400M-parameter Llama-style decoders trained on the same 2.0B ClimbMix tokens, and eight continued-pre-training (CPT) runs on 20M PubMed tokens. Five of the seven pre-trained models contain an Lngram v2 memory (an n-gram memory whose table rows are addressed directly by the quantized router codes of the last 2–3 tokens); two are iso-parameter dense baselines.
Code, evaluation and full reports: MorbiusPlexus/project_ablation_code, branch eval-suite (KEY_FINDINGS.md, FINAL_REPORT.md).
Files
pretrained/ (step 3815, 2.0B tokens)
| file | id | memory | FFN / attention |
|---|---|---|---|
R1_w1_mem_full.pt |
R1 | layers 1 and 11 | full d_ff 2816, 16q/2kv |
R2_w1_mem_cut_near.pt |
R2 | layers 1 and 11 | d_ff cut in layers 2–5, 12–15; 24q/4kv |
R3_w1_mem_cut_far.pt |
R3 | layers 1 and 11 | d_ff cut in layers 6–9, 16–19; 24q/4kv |
R4_w1_dense_full.pt |
R4 | none | d_ff 4256 (iso-parameter dense baseline) |
R5_w1_dense_cut_near.pt |
R5 | none | R4 with R2's cut; 24q/4kv |
A_w2_mem_rank1.pt |
A | R1 + rank-1 path at layer 1 | as R1 |
S1_w2_mem_super1.pt |
S1 | layer 1 only, 184 routes | as R2 |
cpt/ (300 steps on PubMed abstracts, 65k tokens/step)
| file | base | trainable |
|---|---|---|
CPT_R2_all.pt, CPT_R4_all.pt, CPT_S1_all.pt |
R2 / R4 / S1 | everything |
All CPT runs use peak lr 1e-4 except CPT_R2_memory_lr5e-4.
| CPT_R2_ffn12.pt | R2 | FFNs of 12 layers (104M) |
| CPT_R4_ffn8.pt | R4 | FFNs of 8 layers (105M) |
| CPT_R2_memory.pt | R2 | memory only (105M) |
| CPT_R2_memory_lr5e-4.pt | R2 | memory only, lr 5e-4 |
| CPT_S1_memory.pt | S1 | memory only (105M) |
Format and loading
Each file is a torch.save dict with model (fp32 state dict) and config (the full training config). They need the ffnmem package from the repository above:
from ffnmem.hooks import load_model
model, state = load_model("pretrained/R2_w1_mem_cut_near.pt", device="cuda")
The files are pickles (weights_only=False); only load them from this repository. Tokenizer: byte-level BPE, 32,768 tokens, at data/tokenizer/ in the repository.
Headline results
- At equal parameters, memory neither helps nor hurts general zero/few-shot ability (means within ~1 point of R4).
- The models rely heavily on layer-1 memory (zeroing it raises val loss by 0.11–0.64 nats); layer-11 memory is nearly unused.
- Memory is local and lexical: it carries the last 1–3 tokens' identity forward without attention.
- Memory-only fine-tuning and memory-only CPT are weaker than tuning an equally sized FFN subset.