ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller

Qwen3.8-27B — NVFP4 prefiller

Prefill and decode do not have to run the same weights. This repo holds the prefill half: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired with a resident low-bit GGUF for decode.

The point is that the two phases want different things. Decode is memory-bound and spends its budget on being small. Prefill is compute-bound, so it can afford better weights — as long as they never have to be resident.

Better prefill weights recover most of what aggressive decode quantization gives up, at no resident-memory cost — and past a few thousand tokens the streaming disappears under compute, so it is not paid for in latency either.

How it works

Only the transformer stack's linear projections are disaggregated. The embedding, output head, norms and the gated-DeltaNet convolution/state parameters are shared with decode and come from the GGUF.

What it costs

The checkpoint is 12.8 GiB and never becomes resident: it is read through a two-slot ring carved out of the output head's own memory, so the ring itself adds zero device memory. Reads overlap the previous layer's compute.

Variants

One directory of blocks per decode quantization, named for the GGUF it was trained against.

variant	decoder it pairs with	status
IQ1_S	Qwen3.8-27B-UD-IQ1_S.gguf	available
IQ1_M	Qwen3.8-27B-UD-IQ1_M.gguf	available
IQ2_XXS	Qwen3.8-27B-UD-IQ2_XXS.gguf	available
IQ2_S	Qwen3.8-27B-UD-IQ2_S.gguf	available

A prefill checkpoint belongs to one decode checkpoint. Pairing the wrong two is silent — the shapes agree and the model still speaks, it is just worse — so the engine selects the variant from the decoder rather than letting you guess.

Running it

Needs a Blackwell consumer GPU — GB10 (sm_121) or RTX 50-series (sm_120). The build picks the right architecture on its own; if you pin one, it must be 120a/121a, because the block-scaled FP4 MMA is architecture-specific PTX and plain 120/121 or the 120f family target fail to assemble.

git clone https://github.com/IST-DASLab/disaggregated-llama.cpp.git llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Both halves come from the Hub, so there is nothing to download by hand. Pass the decoder and this repo; the prefill variant follows from the decoder:

./build/bin/llama-server -ngl 99 \
    -hf unsloth/Qwen3.8-27B-GGUF:IQ1_S \
    --odp-blocks ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill

A local decoder works the same way — swap -hf ... for -m path/to/decoder.gguf.

The log tells you what it picked:

odp: taking variant 'IQ1_S' from the decoder (filename)
odp: prefill variant 'IQ1_S' from ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill (64 parts)

Flags

flag	what it does
--odp-blocks <repo>	this repo, or <repo>:<VARIANT> to pin one, or a local directory
-ub N	micro-batch. Left alone it is raised to 8192 so the checkpoint streams once per batch; lower it to trade time-to-first-token for device memory (2084 MiB at 8192, 521 MiB at 2048)
-cram 0 -ctxcp 0	for benchmarking. The server otherwise splits a prompt to snapshot recurrent state, and ODP re-streams for each split

If your build has no TLS (configure -DLLAMA_OPENSSL=ON if you can), populate the cache out of band and llama.cpp reads it from there:

hf download unsloth/Qwen3.8-27B-GGUF --include 'Qwen3.8-27B-UD-IQ1_S.gguf'
hf download ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill

Format

Each block is a real GGUF holding one layer's projections as GGML_TYPE_NVFP4, so gguf_dump and the viewer above show what is inside. The trained 4-bit values and their E4M3 block scales are copied verbatim from the compressed-tensors export — the conversion is bit-identical, verified over every weight of a projection. NVFP4's per-tensor global scale has no slot in ggml's block layout and needs none: a scalar commutes with the matmul, so it rides in the KV as odp.scale.<name> and is applied to the result.

Files use llama.cpp's split-file convention, which is what lets one repo hold every variant:

Qwen3.8-27B-ODP-IQ1_S-00001-of-00064.gguf ... -00064-of-00064.gguf
Qwen3.8-27B-ODP-IQ1_M-00001-of-00064.gguf ...