kern-deepseek-v41-flash-sm103

Kernel artifacts and manifest for serving DeepSeek-V4.1-Flash (with its DSpark speculative draft) on one tray of four GB300 (sm_103a) with kern, a model-agnostic GPU runtime. The runtime contains no model; everything it needs to know about this one is in the manifest.

Status: preview. The manifest serves requests end to end; its numerics and speculative acceptance have not yet been certified against the released reference. The 2026-09-11 upload replaces the first working kernel set with the tuned one described under Tuned set; the previous manifest is kept as manifests/deepseek-v41-flash-2026-09-10.json and every cubin it pins is still in cubins/.

Layout

• cubins/<module>-<sha12>.cubin — every device module the manifest pins, one flat content-addressed directory. The runtime resolves modules by sha256 only; the name is a label and the sha prefix keeps every version of a kernel side by side. New builds are added, nothing is renamed or removed.
• manifests/deepseek-v41-flash.json — manifest schema 5, topology ep: 4 (one rank per GPU, 96 routed experts each, dense / attention / indexer / shared-expert weights replicated), generated for a 1M-token context and 256 sequences per rank. Programs: load (once: DeepGEMM scale packing, MegaMoE weight prep, the Q-B / O-A permutation the fused attention kernel needs, RoPE tables and the Engram hash constants computed on the device), prefill (up to 2048 tokens a chunk), decode_batch (up to 256 sequences per rank, captured CUDA graph per bucket) and round (the 6-row DSpark draft / verify step, captured CUDA graph per bucket).
• manifests/deepseek-v41-flash-32k.json — the same kernels generated for a 32k context, 16 sequences per rank and 128-token prefill chunks; it needs 29 GiB less HBM for workspaces and is the one to use when the tray must leave room for something else.
• manifests/deepseek-v41-flash-hbm.json — the same model for a device that cannot read host memory at speed, an HGX B300 (x86 host, no NVLink C2C): the two Engram tables are sharded into HBM across the four ranks (equal row slices, exported, read through peer arrays by dsv41_engram_peers) instead of being mapped once in host memory, about 47 GiB more HBM per GPU. Generated for a 1M-token context, 256 sequences per rank and 2048-token prefill chunks by tools/dsv41/gen.py --engram device --sms 148. It pins the same tuned set as the other two, but the 148-SM instances of the three persistent Mega kernels, because a B300 SXM6 has 148 SMs where a GB300 has 152 (see Kernels and the SM count). See HGX B300.
• licenses/ — the upstream licenses named in the provenance table.

The other two manifests are generated with the default --sms 152 and run on a GB300. Use -hbm on a B300 and either of the others on a GB300; a manifest whose Mega kernels ask for more SMs than the device has hangs on their first grid barrier, which the driver ends as a grid sync timeout.

• patches/ — the changes applied to the upstream DeepGEMM and FlashMLA sources before instantiating the tuned modules (see below).

The manifest is generated by tools/dsv41/gen.py in the kern repository from the checkpoint's own safetensors headers; it is byte-identical to what that generator produces at the kern commit named in this upload's commit message. Every manifest here needs kern v0.2.1 or newer (manifest schema 5, the per-state chunk budget and state pointers the per-sequence window rings need, host weights kept NUMA-local, and peer memory without fabric handles for the -hbm manifest). v0.2.0 refuses to load them ("68 chunks ... hold 15 sequence slots, not the 19 asked for"); v0.1.0 refuses the file.

Not hosted here

• Weights and tokenizer. They are the model's own Hugging Face checkpoint, unmodified: deepseek-ai/DeepSeek-V4.1-Flash. Each weight buffer names the checkpoint tensors that fill it; routed experts are selected per rank at load time from the same shards.
• The two Engram hash tables (layers.{1,14}.engram.embed.*, 2 × 384M rows of FP8 + E8M0 scales, about 189 GiB) are declared placement: host: the runtime maps them once per tray in pinned host memory shared by the four ranks and the lookup kernel reads them over C2C. The host memory must be backed by 512 MiB transparent huge pages (madvise mode is enough); the runtime refuses to start otherwise, because 64 KiB pages make the random lookups about ten times slower. The -hbm manifest binds the same checkpoint tensors as four row slices per table, one per rank, so its weights are 47 GiB per GPU larger and it needs no host memory for them.

Run

Requirements: 4 × GB300 in one tray (135 GiB of weights plus 29 GiB of workspaces per GPU with the 1M manifest; the remaining ~100 GiB is the state pool — about 18 sequences of 1M tokens or 70 of 256k per rank), at least 200 GiB of free host memory, and kern v0.2.1 or newer (curl -fsSL https://kern-baa.pages.dev/install.sh | sh installs the released kern and kern-serve together). State memory per rank: 904 B per token of paged state (compressed KV, index keys, Engram history) plus 5.8 MB per sequence slot (the 43 sliding-window rings of 256 tokens, a slot per active sequence and per kept checkpoint).

hf download Pegainfer/kern-deepseek-v41-flash-sm103 --local-dir kern-dsv41
hf download deepseek-ai/DeepSeek-V4.1-Flash --local-dir deepseek-v41-flash   # the weights, as published
kern-serve --manifest kern-dsv41/manifests/deepseek-v41-flash.json \
           --kernels  kern-dsv41/cubins \
           --weights  deepseek-v41-flash \
           --gpus 0,1,2,3 --chunk 2048 --max-seqs 256 \
           --rows 6 --port 8000

--rows 6 runs the DSpark round (one anchor plus five draft tokens) as the step; --rows 1 serves plain decode. Without --capacity the runtime sizes the state pool from the HBM left after weights and workspaces; --chunk 2048 is the prefill chunk the manifest allows (128-token chunks prefill at about 6k tokens/s per rank, 2048-token chunks at 9-10k). The 1M manifest's plain-decode step at 16 rows per rank is 11.1 ms with --chunk 128 and 12.2 ms with --chunk 2048, against 9.8 ms for the 32k manifest: the graph itself is only 0.2 ms longer (two glue kernels that clear the context-wide score rows), the rest is host-side work in kern-serve's step that grows with the chunk and the context, still to be taken out. For the 32k manifest use --capacity 32768 --chunk 128 --max-seqs 16. Every manifest shares its buffers between the four ranks through peer memory: on a GB300 tray over fabric handles, which need the IMEX channel; on a device without fabric support over the allocation handles themselves, which is why the four ranks of a group are one kern-serve process.

Kernels and the SM count

dsv41_mega_moe, dsv41_mhc and the 128-expert (draft) dsv41_gate are persistent kernels: one CTA per SM, and their dispatch and split reductions barrier over every one of them, so the SM count is both a template parameter and the launch grid. A grid wider than the device leaves CTAs that never become resident and the first barrier never completes. Each of those cubins therefore carries one instance per SM count kern serves — 152 for GB300, 148 for B300 SXM6 — and the manifest pins the pair its device needs, since the entry name and the grid are static in it. The 384-expert gate's 144 and the dsv41_dense / dsv41_oa GEMMs (persistent schedulers, no grid barrier) run on either. The two manifests differ in nothing else: same buffers, states, programs and slab geometry, three ops' entry and grid apart.

HGX B300

The -hbm manifest serves one EP4 group per four B300 (sm_103a, the same cubins), so an eight-GPU node runs two independent servers:

kern-serve --manifest kern-dsv41/manifests/deepseek-v41-flash-hbm.json \
           --kernels  kern-dsv41/cubins --weights deepseek-v41-flash \
           --gpus 0,1,2,3 --capacity 2097152 --chunk 2048 --max-seqs 256 \
           --rows 6 --port 8000
kern-serve ... --gpus 4,5,6,7 --port 8001

--capacity is the paged state per rank in tokens (904 B each): a request can be up to 1M tokens, and 2M holds one such request plus the checkpoints of others; raise it for more concurrent long contexts, or leave it out to let the runtime size the pool from the HBM left. Requirements: about 190 GiB of HBM per GPU plus 0.9 GiB per million tokens of capacity, no IMEX channel, no host-memory requirement beyond loading, kern v0.2.1 or newer. On a GB300 tray the same manifest with KERN_NO_FABRIC=1 exercises this path end to end, which is how it was gated — the 148-SM kernels are resident on a 152-SM device, so its grids and its peer path are both covered there: greedy outputs identical to the host-memory manifest over 16 prompts at --rows 1 and --rows 6, and a passphrase planted 10% into 128k / 1M-token prompts recalled each time (time to first token 12 / 102 s, about 10k prefill tokens/s at chunk 2048; plain decode 7.6 ms per step after). The chat renderer is not a flag: the frontend picks it from the checkpoint's model_type, so this checkpoint gets the V4.1 text encoding (system messages, multi-turn history, thinking with reasoning_effort); its tool-call output parser is not adapted yet. With a kern.toml target the same runs as kern server <target> -- ....

Provenance and licenses

artifacts	origin	license
dsv41_mhc, dsv41_gate, dsv41_dense, dsv41_mega_moe, dsv41_paged_indexer, dsv41_sparse_indexer	DeepGEMM at ab69f76be5bb9ea3499bc755002b1a876cb0b3d9 (PR #432: mHC, fused router, MXFP8 GEMM, FP8/FP4 MegaMoE, MXFP4 index scorers), instantiated by kern's tools/dsv41/moe/*.cu and tools/dsv41/attention/*_indexer.cu against the upstream headers; dsv41_mega_moe is a fork of the upstream MegaMoE kernel for the EP4 slab layout	MIT (licenses/LICENSE.deepgemm.txt)
dsv41_fused_split	the same FlashMLA kernel with patches/flashmla-split-kv.patch: a DecodeWithSplitKV mode that runs several CTAs per row over disjoint KV block ranges into fp32 partials, and a combine kernel (lse merge, inverse O RoPE, MXFP8 cast) that produces the one-CTA path's output layout	MIT (licenses/LICENSE.flashmla.txt)
dsv41_fused_decode	FlashMLA at 4f38f29ef6793c228363e4af5be66d44e81167ba (PR #221: the fused V4.1 decode kernel — Q RoPE, sparse attention over FP8 window + FP4 compressed paged KV, inverse O RoPE and the MXFP8 output cast in one launch, 64 heads), extracted with cuobjdump -xelf from a build of the upstream sources	MIT (licenses/LICENSE.flashmla.txt)
dsv41_deepselect	DeepSelect at 8e70df71d2a4b0c969ef96dc3b8998efa09a3315 (top-512 token and top-2048 candidate selection), extracted the same way	MIT (licenses/LICENSE.deepselect.txt)
dsv41_auxiliary	kern's handwritten kernels (tools/dsv41/auxiliary/auxiliary.cu): Engram history / hash / lookup / inject, compressor, window ring slots and cache indices, FP8/FP4 cache inserts, RoPE, index quantization, and the fused RMSNorm+MXFP8 (dsv41_norm_quant) and RMSNorm+RoPE (dsv41_norm_rope) attention-input kernels. Semantics follow the checkpoint's inference/ code and vLLM PR #56201 (61140208c5940195b5970fc44de77a9695d0baf7); no upstream code is copied	same as kern
dsv41_engram_peers	kern's handwritten kernel (tools/dsv41/auxiliary/engram_peers.cu): the Engram lookup over tables sharded across the EP group, the same per-element arithmetic as dsv41_lookup reading each row from the rank that holds it; pinned only by the -hbm manifest	same as kern
dsv41_engram_constants	generated by tools/dsv41/auxiliary/engram_constants.py from the checkpoint's tokenizer and Engram layout: the compressed token map, hash multipliers, primes and offsets, and 32 inverse RoPE frequencies per family; the full tables are computed on the device by load	same as kern
dsv41_moe_boundary, dsv41_moe_stage, dsv41_weight_prep, dsv41_fused_prep, dsv41_candidate, dsv41_head, dsv41_spec_round, copy_rows	kern's handwritten kernels (tools/dsv41/moe/*.cu, tools/dsv41/attention/candidate.cu, tools/dsv41/moe/fused_prep.cu, tools/dsv41/head.cu, tools/kernels-src/*.cu), nvcc -cubin -arch=sm_103a	same as kern

Tuned set (2026-09-11)

The manifest now pins the kernel set that came out of a decode / DSpark tuning pass on one tray (4 x GB300, DP4 / EP4). Measured on 10k-token prompts, 3 concurrent requests: plain decode TPOT 10.3 -> 7.3 ms, DSpark round step 12.2 -> 9.4 ms (TPOT 3.7 -> 3.2 ms). Plain decode throughput per tray with 1k-token prompts (steady state, every request decoding):

concurrent requests (rows per rank)	tokens/s	step ms	manifest
4 (1)	480	8.3	32k
16 (4)	1960	8.1	32k
64 (16)	6500	9.8	32k
256 (64)	14600	17.5	1M
512 (128)	19100	26.8	1M
1024 (256)	22200	45	1M

A single 200k-token prompt prefills at 9.4k tokens/s and then decodes at 7.9 ms per token (10k-token prompts: 7.3 ms), so the step is nearly flat in context length; the sparse attention reads a 128-token window plus 512 selected tokens, and only the indexer's key scan grows with the context. Plain decode output is bit-identical to the previous set for the first ~1.3k generated tokens of a 10k prompt and diverges afterwards at near-tie tokens, the same class of difference the previous set shows between its DSpark and plain paths. No change is specific to a batch size; the DSpark acceptance rate at these settings reads within text-to-text noise of the previous set.

What changed, module by module (sha prefixes are the new files in cubins/):

• dsv41_dense (c115e88d76d2): per-shape DeepGEMM MXFP8 tile instantiations chosen from a sweep at M = 1..128 (wqkv, wq_b, wo_a, wo_b, indexer query, compressor and Engram projections), built from the upstream sm100_fp8_fp4_gemm_1d1d with patches/deepgemm-pdl-window.patch (programmatic dependent launch: the dependent grid is released as the producer drains; the consumer's own griddepcontrol.wait still orders the data).
• dsv41_mhc (9ffd59d22dd3): Mega mHC instantiated with 40 K-splits instead of 4 (the reduction over the 4 x 5120 residual is spread over more CTAs); patches/deepgemm-mhc-gate-early-trigger.patch issues the PDL trigger right after the grid-dependency wait, so the successor's prologue overlaps the whole kernel (its own wait still covers this grid's completion).
• dsv41_gate (faef4c0ef8c5): the 384-expert router instantiated with split-K 8 and 12 pipeline stages; the split partial scores are merged in a fixed order before scoring, so routing is deterministic and matches the single-split kernel (same top-6, weights within 5e-7).
• dsv41_fused_split (94a00fbb9cc6): split-KV fused attention, 10 parts per row in both the decode and the verify programs (the two must split identically, or the draft and target reductions differ and DSpark acceptance drops).
• dsv41_auxiliary (58c73025afa7, then 5b763b24174f, the build from the kern tree once the changes landed) and dsv41_engram_peers (590c505e63fc): every auxiliary kernel and the peer lookup open with griddepcontrol.wait and trigger launch_dependents immediately after, so the whole layer chain (auxiliary glue, DeepGEMM, attention) is launched with programmatic dependent launch and each node's prologue overlaps its predecessor.

Kernel time per DSpark round after the pass (nsys, one 10k-context sequence): MegaMoE 40%, dense GEMMs 21% (about 4 TB/s on the large projections), mHC 11%, attention 9%, launch-bound glue 9%, lm_head 4%, gate 4%. The remaining gap to the weight-stream floor is inside MegaMoE at small M, which is a kernel rewrite rather than a tuning change.

Every manifest also shares the indexer's full-context score workspace ([rows, context] fp32 per index layer) between the prefill, decode and verify programs, which never run concurrently; without that the three copies cost 65 GiB per GPU at the 1M context.

The generator that produces this set is in the kern repository since dde9eb3 (tools/dsv41, the DeepGEMM / FlashMLA patches under tools/dsv41/patches/), and the three manifests here are its output at that commit. Its gate (tray05, 4 x GB300, kern v0.2.1 binaries, 16 greedy prompts at concurrency 1): the regenerated 1M manifest is byte-identical in output to the previous upload at --rows 1, and --rows 6, the -hbm manifest (fabric handles off) at both row counts, and the -hbm needles above all match it; plain decode step 7.6 ms, DSpark round 9.6 ms with 2.6 to 2.7 tokens accepted per step. The 32k manifest flips near-tie tokens late in 9 of the 16 prompts against the 1M manifest; the serving shape is not the cause (the 1M manifest at --chunk 128 --max-seqs 16 is byte-identical to itself at --chunk 2048 --max-seqs 256), the buffer shapes are, which the tuned kernels' work partition follows. It is the same class of difference as between the tuned and previous sets.

Toolchain: CUDA 13.0 nvcc (cuda_13.0.r13.0/compiler.36424714_0) for the attention, selection, auxiliary, head, speculative and constant modules; CUDA 13.1 nvcc (cuda_13.1.r13.1/compiler.37061995_0) for the DeepGEMM-derived MoE / mHC / dense modules and copy_rows; dsv41_auxiliary-58c73025afa7 is a CUDA 13.1 build and dsv41_auxiliary-5b763b24174f / dsv41_engram_peers-590c505e63fc are CUDA 13.0 builds of the same source. A different nvcc yields a different cubin and therefore a different sha; the manifest pins the build that was present when it was generated, and whether two builds agree numerically is kern test's verdict, not the name's.