RidgeFill maps for Qwen3.5-9B: the reference study's maps at four split depths.
Fitted once in closed form. Retrieval non-inferior to exact with attention-selected rows.
Data and code · All maps · llama.cpp fork · Citation
What this is
The ridge maps of the RidgeFill paper's reference study for Qwen3.5-9B (Qwen/Qwen3.5-9B, revision c2022362, run on
its text path only). These are the files the paper's runs loaded. They load in the reference implementation (code/kva
in the data repository); no serving engine is claimed for them.
Files
| File | Split | Bytes | SHA256 |
|---|---|---|---|
qwen3.5-9b-sweep-s8.safetensors |
8 of 32 | 805,505,368 | 39c6e75cb9da7304b494b9e93adba18c13b5cbbd4eea0092cb7ae74a178ee2f1 |
qwen3.5-9b-sweep-s12.safetensors |
12 of 32 (primary) | 671,254,536 | 75bb7254fd6f655867928e37f0892c928960920a16dfa143945b0d74f801a311 |
qwen3.5-9b-sweep-s16.safetensors |
16 of 32 | 537,003,696 | 3233973a9dee5efc716278f65099476181b42fbed4826696da85910d31f2b078 |
qwen3.5-9b-sweep-s20.safetensors |
20 of 32 | 402,752,864 | a40471edef1615ff22dbcd9791110207a3a52e734916d87a05a3fda41695f9fc |
Each file holds one bf16 tensor layer.L of shape [4096, 4097] (weights, then bias) for every layer L above the split,
fitted in closed form on 839,652 rows from 71 calibration documents with λ = 0.03. Files with the same names elsewhere
may be earlier fits; check the sha256.
Results
From the paper's reference study, with a 4,096-token exact tail. Qwen3.5-9B is a hybrid of attention and Gated DeltaNet layers, and its fill error sits in the values and the recurrent state.
| Measure | Value |
|---|---|
| Filled-region RULER, RidgeFill − identity fill, split 12 (219 samples, 8K to 64K) | +3.3 [−1.3, +7.8] points |
| Filled-region RULER, RidgeFill + attention-selected rows − exact, split 12 | 0.0 points, lower bound −1.36 (non-inferior at 2 points) |
| Key-token loss, RidgeFill − identity fill, split 12 | −0.347 [−0.523, −0.196] nats |
| Novel-token loss vs exact, splits 8 / 12 / 16 / 20 | +0.033 / +0.024 / +0.017 / +0.008 nats |
How RidgeFill works
The lower layers run exactly up to a split. For every prompt row except the exact ones, a ridge map per later layer predicts that layer's input from the split layer's state, and the layer writes its cache from the prediction (b) instead of running in full: its own key and value projections run unchanged, and on a Gated DeltaNet layer so does its state update. The maps are fitted once, in closed form, from one pass over calibration text. The last few thousand prompt tokens, any selected exact rows and every decoding step run the full model. Nothing is trained and no token is dropped. The design comes from DeepSeek-V4.1-Flash, where the projection is trained into the model; (a) is kishida's Q3-8B-KVA-Projector, which retrofitted it onto Qwen3-8B with trained key and value heads.
Citation
The paper is RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill by Dylan Johnston and Robert P Smith (preprint, 2026, doi:10.5281/zenodo.23890247). Cite it as:
@misc{johnston2026ridgefill,
author = {Johnston, Dylan and Smith, Robert P},
title = {{RidgeFill}: Training-Free Late-Layer Cache Fill for Faster Prefill},
year = {2026},
publisher = {Zenodo},
note = {Preprint},
doi = {10.5281/zenodo.23890247},
url = {https://doi.org/10.5281/zenodo.23890247}
}
To cite the data and code themselves, use the data release's citation (Approximate-Science/ridgefill-data).
An earlier technical note is archived at doi:10.5281/zenodo.23179168. The
paper replaces some of its numbers; SUPERSEDED.md in the data repository lists which.