TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector

RidgeFill maps for Llama-3.1-8B-Instruct: the reference study's maps at four split depths.

Fitted once in closed form. 39.7 more filled-region RULER points than the identity fill at the same split.

Data and code · All maps · llama.cpp fork · Citation

Built with Llama.

What this is

The ridge maps of the RidgeFill paper's reference study for Llama-3.1-8B-Instruct (unsloth/Llama-3.1-8B-Instruct, revision 4699cc75), plus the split-12 maps converted to GGUF for the llama.cpp fork. The maps are fitted on Llama 3.1 hidden states and are distributed under the Llama 3.1 Community License (LICENSE, USE_POLICY.md).

Files

File	Split	Bytes	SHA256
safetensors/llama-3.1-8b-instruct-sweep-s8.safetensors	8 of 32	805,505,392	911b01565af665931c1e2cb01dd6fc6a7951f120117e1327281245c51a75b335
safetensors/llama-3.1-8b-instruct-sweep-s12.safetensors	12 of 32 (primary)	671,254,560	89bf425556ca2ab5455b2ab76b686e2a29b1bdf9716bb882cb902f2bd6c18008
safetensors/llama-3.1-8b-instruct-sweep-s16.safetensors	16 of 32	537,003,728	b076e73dc8c852f7bb27b37bc1b13cbac2bf6e9db769d095cf046b88de80358e
safetensors/llama-3.1-8b-instruct-sweep-s20.safetensors	20 of 32	402,752,888	086afd8767eaf4d207968ca9a4572151d21f710e02c7734b084c9c9eeeeb1212
llama-3.1-8b-instruct-s12-bf16.gguf	12 of 32, bf16	671,419,904	83b893b7da619dbf280071f6d5a7e2fb1655eef4b7b5f288a90003d3c612fd34
llama-3.1-8b-instruct-s12-q8_0.gguf	12 of 32, Q8_0	356,847,104	625cffeb57d7a26659f720bd05931cbaa6cd0ef526f93bb3d77623cb452797a9

The safetensors files are the bf16 maps the paper's runs loaded, in the reference implementation (code/kva in the data repository). They were fitted in closed form on 834,541 rows from 71 calibration documents with λ = 0.03.

Which split

Llama-3.1-8B has no key normalization, and its fill error sits in the keys. At split 12 (3/8 depth) the fitted fill's added loss on novel tokens is larger than the identity fill's, +1.006 against +0.505 nats; at split 16 and 20 it is far smaller, +0.085 and +0.030 against +0.563 and +0.406. For loss, use split 16 or deeper. At split 12, attention-selected exact rows remove the failure and restore retrieval. The GGUF files in this repository are split 12.

Results

From the paper's reference study, with a 4,096-token exact tail.

Measure	Value
Filled-region RULER, RidgeFill − identity fill, split 12 (216 samples, 8K to 64K)	+39.7 [+32.6, +46.5] points
Filled-region RULER, RidgeFill + attention-selected rows − exact, split 12	+0.2 points, lower bound −1.38 (non-inferior at 2 points)
Novel-token loss vs exact, split 16	+0.085 nats (identity fill +0.563)
Released SwiftKV checkpoint − RidgeFill, novel-token loss, split 16	+0.010 [−0.004, +0.025] nats

Use in llama.cpp

hf download TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector llama-3.1-8b-instruct-s12-q8_0.gguf --local-dir models
llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 32768 -ngl 99 -fa on \
    --ridgefill models/llama-3.1-8b-instruct-s12-q8_0.gguf --ridgefill-tail 8192

Build the fork as described in its RIDGEFILL.md. Only the plain fill with an exact tail applies to this model; the selection modes of the Qwen3.8-27B port do not.

Measured in llama.cpp

Earlier measurements of the split-12 GGUF, separate from the paper. One RTX 4090 24 GB, CUDA 12.4, flash attention on, -b 2048 -ub 512, F16 KV cache, fork commit fa8061c4b, wikitext-2 test text. Stock is the same build without --ridgefill. Tail T = 4096 unless stated.

Prompt processing, llama-bench -p N -n 0 -r 3, speedup over stock:

Model	8K	16K	32K
F16	1.26×	1.51×	1.74×
Q4_K_M	1.26×	1.52×	1.73×

Time to first token through llama-server, Q4_K_M, mean of 3 prompts, seconds (stock → RidgeFill): 8K 0.77 → 0.60; 16K 1.73 → 1.14; 32K 4.30 → 2.47. Generation speed (tg128) is the same with and without RidgeFill.

Tail NLL (mean NLL of the last 4,095 tokens of each window, nats per token, stock → RidgeFill), Q4_K_M:

Prompt	T = 4096	T = 8192	T = 16384
8K (35 windows), stock 1.846	2.031	n/a	n/a
16K (17 windows), stock 1.811	2.047 (1.52×)	1.927 (1.27×)	nothing filled
32K (8 windows), stock 1.843	2.681 (1.73×)	2.050 (1.52×)	1.938 (1.23×)

llama-perplexity (-c N; effective tail max(T, N/2)), stock → RidgeFill:

Model	-c 8192	-c 16384	-c 32768
F16	6.1775 → 7.5485	5.8588 → 7.0083	6.7350 → 7.8851
Q4_K_M	6.3350 → 7.6229	6.0108 → 7.0517	6.9105 → 7.9416
F16, Q8_0 maps	6.1775 → 7.5446	5.8588 → 7.0046	6.7350 → 7.8886

Needle in a haystack (wikitext-2 haystack, synthetic needle, depths 10 to 90 %, 4 needles per depth and length, greedy, T = 4096), needles found:

Arm	8K	16K	32K	Needles inside the filled region
F16 stock	20/20	20/20	20/20	n/a
F16 RidgeFill	19/20	15/20	15/20	35/44
Q4_K_M stock	20/20	20/20	20/20	n/a
Q4_K_M RidgeFill	18/20	17/20	14/20	34/44

How RidgeFill works

The lower layers run exactly up to a split. For every prompt row except the exact ones, a ridge map per later layer predicts that layer's input from the split layer's state, and the layer writes its cache from the prediction (b) instead of running in full: its own key and value projections run unchanged, and on a Gated DeltaNet layer so does its state update. The maps are fitted once, in closed form, from one pass over calibration text. The last few thousand prompt tokens, any selected exact rows and every decoding step run the full model. Nothing is trained and no token is dropped. The design comes from DeepSeek-V4.1-Flash, where the projection is trained into the model; (a) is kishida's Q3-8B-KVA-Projector, which retrofitted it onto Qwen3-8B with trained key and value heads.

Citation

The paper, RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill by Dylan Johnston and Robert P Smith, is in preparation. Until it is out, cite the data release:

@misc{johnston2026ridgefilldata,
  author  = {Johnston, Dylan and Smith, Robert P},
  title   = {{RidgeFill}: data and code for ``Training-Free Late-Layer Cache Fill for Faster Prefill''},
  year    = {2026},
  version = {1.1},
  url     = {https://github.com/Approximate-Science/ridgefill-data}
}

An earlier technical note is archived at doi:10.5281/zenodo.23179168. The paper replaces some of its numbers; SUPERSEDED.md in the data repository lists which.