Prefill reads ~2x the experts it needs — fixing it is 1.7x on one SSD, 2.7x on three

#6
by konstantinnikol - opened

If you're streaming this Q4 from NVMe with ds4 because it doesn't fit in RAM, there's
free performance in the prefill path.

The layer-major prefill sweep reads all 384 experts per layer. A 512-token chunk only
routes to 187 of them — 48.8%. So it reads roughly twice what the model actually touches.
Measured on unmodified ds4 (bd66c40), single internal SSD: 301.9 GB read where the
prompt touches 149.2 GB.

Running the router first and staging only the selected experts fixes it. Numbers on an
M5 Max 128 GB, 512-token prompt / 200 generated, interleaved against upstream, identical
output SHA-256 on every arm:

upstream ours
prompt processing, one drive, no replicas 16.50 28.04 1.70×
steady decode, one drive 10.44 14.38 1.38×
prompt processing, three drives 16.23 43.62 2.69×
steady decode, three drives 10.61 17.38 1.64×

The one-drive rows are the point: that gain needs no extra hardware, just not reading half
the bytes. The three-drive rows add weighted split reads across byte-identical replicas,
which is what more devices buy on top.

One caveat — it's adaptive by prompt length. At 512 tokens the routed union is 48.8% and it
wins clearly; above ~1024 tokens the union passes 60% and full staging with read-ahead is
better. The threshold is a tunable.

Branch: https://github.com/argonautlabsai/ds4-argodrive/tree/argonaut-v41-benchmark
A fresh clone builds ds4 and ds4-bench with the multi-source reader compiled in.

Method, every arm and the raw sampler data:
https://argonautlabs.ai/research/deepseek-2026-09-15.html

Reported to antirez by email before posting.

Awesome optimization! Will integrate it into DwarfStar soon.

Awesome optimization! Will integrate it into DwarfStar soon.

Thank you!

Everything's on the argonaut-v41-benchmark branch of argonautlabsai/ds4-argodrive.

If you only pick one commit, 38e200a: the selective prefill path was gated behind "two or more sources", and with the gate at one it runs on a single drive. On the internal SSD alone that's 16.5 → 28 tok/s prefill and 10.4 → 14.4 decode, same output SHA. Only kicks in for prompts over 256 tokens.

One question: any plans for a DSpark file for V4.1 Flash?
Decode is GPU-bound for us now, so a drafter is the one thing left that could move it a lot.

Happy to help on the DwarfStar side if useful.

New result: 20.115 generation-inclusive tok/s, or 20.375 steady.

Setup: V4.1 Flash Q4, M5 Max 128 GB, internal SSD plus two TB5 enclosures.
Method: 512 prompt tokens, 512 generated tokens, two runs per configuration in BAAB order.

https://github.com/argonautlabsai/ds4-argodrive/releases/tag/v41-router-qualified-20260921

Sign up or log in to comment