benjzzz/DeepSeek-V4-Flash-GALE-1bit

DeepSeek-V4-Flash — GALE 1-bit experts

A 1-bit re-quantization of DeepSeek-V4-Flash's routed experts, built to fit entirely inside consumer VRAM. On the machine it was made on — 2×RTX 3090 + 2×RTX 2080 Ti, 70 GB of VRAM total — all 11,008 experts are pinned in VRAM with no CPU tier, no overflow tier and no disk reads during decode. That residency, not the kernels, is what makes it usable.

159.6 GB → 55.0 GB, and the same quantizer that scores 0.52–0.54 expert output error on V4-Pro scores 0.357 here: Flash's smaller experts take 1-bit roughly twice as well.

What is in the pack

Weights	Format on disk	Size	Bits/weight
Routed experts (43 layers × 256 experts × w1/w3 [2048,4096] + w2 [4096,2048])	GALE 1-bit: sign bit per weight under a blocked FWHT-1024 rotation, fp16 scale per 128-weight block, 1% outliers as int4 with a per-row fp16 scale	46.0 GB	1.329
Attention, shared experts, norms, routers, HC-mixing, embedding, head	fp8 (block-scaled) and fp16 on disk; fp8 entries are quantized to 4-bit at load and kept 4-bit in VRAM	8.9 GB	8 on disk / 4 in VRAM

277.0 B routed-expert parameters across 33,024 expert matrices, plus 1,228 non-expert tensors.

How the experts were quantized

Same v4 recipe as the V4-Pro pack: each sign is chosen with GPTQ-style error feedback in output space, weighted by the inverse Hessian of real activations, with a least-squares per-row gain folded into the block scales at zero extra bits.

What differs here is the calibration data. V4-Pro's card names "re-calibrate on code" as its next lever, so this pack was calibrated on 49,152 tokens (24 × 2048) of ~64 % Python source plus technical English, captured layer by layer during a real forward pass with the original weights, with each layer's hidden state chained through the quantized reconstruction of the layer before it.

The whole 43-layer run took 7 h 07 min on one RTX 3090 (~10 min/layer, 256 experts each).

Accuracy

Held-out expert output relative error, measured per expert during the run (33,024 matrices):

	expert_out_rel
whole model (mean)	0.3571
median	0.3537
best layer (layer 0)	0.244
worst layer (layer 27)	0.413
V4-Pro, same quantizer and setting	0.52–0.54

Error grows with depth, because each layer is calibrated on activations that already carry the previous layers' quantization error:

L0 0.244 · L7 0.343 · L14 0.355 · L21 0.344 · L28 0.389 · L35 0.397 · L42 0.338

Do not quote the layer-0 number as the model's error. A single-layer probe reads 0.244 and is what the bit-width decision was made on; the finished model is 0.357.

Why 1 bit and 1 % outliers

Both were chosen from measurements on real Flash experts, not assumed. The outlier budget has sharp diminishing returns, and everything above 1 % costs more VRAM than the accuracy is worth once the goal is full residency (layer-0 probe):

outlier budget	expert_out_rel	bits/weight	experts total	fits 70 GB VRAM
1 %	0.248	1.33	46 GB	yes, with room for KV cache
2 %	0.236	1.52	53 GB	yes
3 %	0.227	1.73	60 GB	tight
5 %	0.211	2.12	74 GB	no

A 2-bit/4-bit alternative (BenQ quantizer, n_tiers 2 and 8) is genuinely more accurate — held-out out_relerr 0.128 / 0.135 / 0.217 for w1/w3/w2 at 2-bit, 0.049 / 0.052 / 0.090 at 4-bit — but lands at ~96 GB, which spills ~27 % of experts back onto a CPU tier and costs more speed than the accuracy buys. It also has no serving kernel here. 1-bit was chosen for residency, not because it is more accurate.

Running it

pip install "torch>=2.5" triton transformers tokenizers fastapi uvicorn jsonschema
ulimit -n 65536
./serve.sh                      # edit D= at the top to point at the clone

serve.sh carries every setting that matters. Three are not optional:

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — full residency fills each GPU to ~90 %, where the default allocator fragments and OOMs with GBs still free.
• DSV4_ONESHOT / DSV4_ONESHOT_TOOLS — see Prompting below. These are load-bearing.
• DSV4_TIER_BIAS= (empty) — the tiering knobs are meaningless when nothing is on a slow tier.

The server plans placement itself from what it finds: it enumerates every visible GPU, reads each one's free memory, and splits the layers in contiguous blocks proportional to each device's budget. On this mixed set that yields 16/6/15/6 layers and 20.6 / 7.4 / 18.4 / 7.6 GB. If the model does not fit resident, it falls back to a tiered plan with a CPU expert tier — the same code serves both.

Serving this pack surfaced four V4-Pro constants hardcoded in that "generic" server, all fixed in server/deepseek_v4pro_server_v2.py: expert-slot sizing looped range(61) (Pro's layer count, an instant KeyError on 43); the cache warmers defaulted to 384 experts (Flash has 256); full-residency placement filled the largest GPU first and left the last card empty; and RESERVE was a literal 0.85 despite being documented as DSV4_RESERVE.

Measured performance

2×RTX 3090 + 2×RTX 2080 Ti, Ryzen 9 3900X, 62 GB RAM, weights on NVMe:

	V4-Flash 1-bit (this pack)	V4-Pro 1-bit, same box
plan	full residency, 11,008 experts pinned, no CPU tier	tiered, ~60–90 of 366 experts/token on the CPU tier
load	80 s	130 s
decode	3.4–4.1 tok/s	1.2–1.8 tok/s
experts touched per token	258 (43 × 6), all from VRAM	366, most off CPU/NVMe
agent task: write hello.py, run it, report output	196 s, completed	250 s
VRAM	20.6 / 7.4 / 18.4 / 7.6 GB	23.4 / 21.7 / 8.8 / 8.8 GB

Prefix caching is on by default: the unchanging preamble of a conversation is prefilled once, snapshotted to kv_snapshots/ and reused, and a rolling checkpoint every 8 decode tokens lets a follow-up turn resume mid-answer.

Prompting (read this before judging the quality)

This model follows instructions weakly enough that the one-shot example in the preamble decides what it does, and getting it wrong looks like a broken model. Measured, on the same prompts:

one-shot shown	behaviour
greeting only	echoes the user's prompt back instead of answering
greeting + a markdown code answer	good code, but it describes tool work instead of calling tools
a tool call only	calls a tool for everything — tries to read_file a file that does not exist when merely asked to show a snippet
ends after one tool result	does the first step of a task and stops
greeting + terse answer + code answer (no tools) and + a two-step tool example (tools)	what ships: answers directly, calls tools when asked, and chains steps

The shipped DSV4_ONESHOT_TOOLS demonstrates a two-step task whose second step inspects a file. An otherwise identical example whose second step runs the file made the model fabricate the run's output instead of calling anything — so the bridge also strips <tool_result> blocks and stray Assistant: markers from model output, terminated or not.

Files

• packed_v4/ — expert pack, layer_LLL.h0.pack (43 files) + manifest_mixed_full.jsonl (34,252 records: every expert matrix and every non-expert tensor, with kind, shape, sections, outlier count, pack file and byte offset).
• misc_v3/ — non-expert tensors, misc_model-000NN-of-00046.pack (46 files).
• server/ — everything needed to run standalone: the engine (deepseek_v4pro_server_v2.py) and its kernels (dsv4_kernels.py, dsv4_w4.py, dsv4_cpu_expert.c + compiled .so, dsv4_cpu_tier.py, dsv4_device_plan.py), the reference implementation (reference/, from deepseek-ai/DeepSeek-V4-Flash; byte-identical to V4-Pro's, the reference code is genuinely config-driven), the tokenizer, the quantizer used to build this pack (v4pro_gptq1bit.py, v4pro_stream_quantize_v3.py), and the tool-calling bridge under server/agent_bridge/.
• serve.sh — the launcher, including the one-shot examples above. It resolves its own directory, so a fresh clone runs with no edits: ./serve.sh.
• routing_counts.json — uniform placeholder. Full residency pins everything, so hot-set ranking is unused here; it matters only if you run this pack tiered on a smaller machine.

Honest limits

• It is not a strong coding model. It writes a correct iterative Fibonacci where V4-Pro produced prose, answers arithmetic and shell-command questions correctly, and drives a real write-then-run tool loop end to end. It also still gets things wrong that a good model would not — asked for a one-liner counting lines in .py files it returns ls *.py | wc -l, which counts files.
• It fabricates tool results if allowed to keep generating past a tool call. The bridge ends the turn at the first closed tool block and strips invented results; do not remove that.
• Context is 4096 tokens, and a tool-schema preamble costs ~1,100 of them. Prefix caching makes that cheap in time, not in context.
• Each agent step is a full round trip, so multi-step tasks run in minutes, not seconds.
• Quality was measured on a handful of prompts and per-expert reconstruction error, not on a standard benchmark suite. Treat every quality claim here as indicative.
• Kernels are Triton (any CUDA GPU); the optional fast paths verify themselves at startup and fall back to a portable implementation if they cannot run. Nothing runs on Apple Silicon or ARM as-is.
• The expert files were built from the official deepseek-ai/DeepSeek-V4-Flash weights streamed shard by shard; the original weights are not included here.