0xTank/DeepSeek-V4.1-Flash-FP8-R37-4x-GB10

DeepSeek-V4.1-Flash FP8 — R37 on four GB10 systems

An FP8 deployment recipe for either standard DeepSeek weights or dealignai FP8 weights. The measurements on this page use dealignai FP8. FP8 identifies the checkpoint variant; the model also has MXFP4 routed experts. R37 identifies the runtime.

Single-stream prose: 50.50 tokens/s. Eight-request aggregate: 229.28 tokens/s. 600,000-token context, eight active slots, K3 speculation, and multimodal input.

Current benchmark

Benchmark credit: Tony / Tech2Wild on Hugging Face (Tech2Wild/Kai), authors of the upstream benchmark suite. The results below were measured by 0xTank on this recipe.

Tony's fixed prompt set v1, temperature zero, thinking explicitly off. Each concurrency cell is one synchronized batch. Aggregate includes batch wall time; per-stream decode excludes TTFT. The headline averages eight categories and excludes counting. These are measurements of this deployment, not guaranteed performance on other hardware or the standard checkpoint.

Concurrency	Aggregate tok/s	Mean per-stream tok/s	Mean TTFT (s)
C1	57.45	63.2	0.231
C2	82.51	45.83	0.268
C3	113.76	43.36	0.3
C4	145.91	41.29	0.312
C5	160.12	37.41	0.355
C6	187.13	35.33	0.374
C7	203.01	33.39	0.433
C8	229.28	33.12	0.426

Per-stream decode tokens/s

Category	C1	C2	C3	C4	C5	C6	C7	C8
coding	76.62	58.31	58.35	58.23	49.65	48.79	47.0	51.36
json	65.84	46.02	44.56	42.23	37.77	36.6	34.32	31.6
narrative	42.54	34.62	27.91	25.23	23.31	20.05	20.06	21.67
prose	50.5	36.22	32.73	30.14	28.27	24.82	25.05	22.0
math	78.55	56.17	52.79	54.68	42.84	45.32	42.64	41.06
reasoning	66.99	46.98	46.15	38.69	37.9	34.65	33.92	32.95
summary	44.03	30.57	30.81	28.0	27.64	25.61	22.42	20.88
format	80.55	57.75	53.56	53.11	51.88	46.81	41.68	43.44
ceiling_count	84.58	60.43	66.44	65.45	60.47	56.46	54.1	55.6

Prefill

Actual input tokens	Elapsed seconds	Input tokens/s
2950	0.967	3050.6
11592	3.49	3321.5
46810	14.763	3170.8
93335	30.702	3040.0

Needle retrieval

Actual input tokens	TTFT seconds	Input tokens/s	Result
130258	46.9	2777.6	PASS
260119	110.5	2354.8	PASS
520921	257.1	2026.0	PASS

Needle depth was 50%; this is a single-fact retrieval check.

Vision, tools, and idle

Vision passed all three checks. Automatic tool selection, round-trip tool results and parallel calls passed. Forced named tool selection failed: get_weather was returned alongside forced get_time. Overall: 6/7. Offering only the selected tool worked in three diagnostic repetitions; the API still returned finish_reason: stop. No enforcement fix is claimed.

After the controlled 45-second idle, counting was 83.8 tokens/s and coding 75.1 tokens/s. The first idle-probe request followed other tests, not a long idle period. Stream-chunk step estimates in Tony's probe are diagnostic and should not be treated as exact kernel-step measurements.

All raw records are in bench/results/current. The vision/tool and idle helpers were run with an explicit thinking-off override to match Tony's original server default. The normal serving default is thinking on, xhigh, temperature 1.0, top_p 0.95.

Exact runtime configuration

• Four 128 GB GB10 systems; TP4 over RoCE; eight active requests.
• Context 600000, max batched tokens 4096, GPU memory utilization 0.81.
• B12X attention, linear and MoE; FP8 KV, block size 256.
• K3, greedy draft, standard rejection, adaptive verification enabled.
• FULL_AND_PIECEWISE CUDA graphs; disk Engram with local worker row overlays.
• Up to four images per request, 1 GiB multimodal processor cache.
• vLLM c687594b8a8082e18af9fe2f64eb5f9ee442e251.
• B12X f1c4e9dd5b1d841c10cda70ee49c3190cf3db0be.
• Runtime image ID sha256:638d63b0f4bb9741c4afd4936e184659053e8a5ec466d6e7a8994ae0094e58fa.

Download the FP8 weights

Choose either checkpoint:

• Standard DeepSeek: deepseek-ai/DeepSeek-V4.1-Flash
• dealignai FP8: dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8

Download the complete repository, including tokenizer, configuration, index, and all weight shards. Use the same checkpoint on every rank and prepare its own Engram overlays. The benchmarks on this card used the dealignai checkpoint.

# Standard checkpoint:
hf download deepseek-ai/DeepSeek-V4.1-Flash --local-dir /srv/models/DeepSeek-V4.1-Flash

# Or dealignai FP8, in its own directory:
hf download dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8 --local-dir /srv/models/DeepSeek-V4.1-Flash-dealignai

Set MODEL_PATH in cluster.env to your chosen checkpoint directory.

Installation status

The parameterized launcher is in launch/node.sh. Copy launch/cluster.env.example to cluster.env and configure the RoCE addresses, interface, HCA, model, cache and Engram paths on each node. Start ranks 3, 2, 1, then 0 using bash launch/node.sh RANK cluster.env. Existing containers are never overwritten. The API listens on port 8000 without authentication; use a trusted network or authenticated gateway.

Each node needs the same complete checkpoint, either locally or through read-only NFS. Select standard deepseek-ai/DeepSeek-V4.1-Flash or dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8 and use that checkpoint consistently across all ranks. Do not reuse sparse Engram overlays from a different checkpoint. On each worker prepare fresh local rows:

python3 tools/prepare-engram.py RANK /srv/models/DeepSeek-V4.1-Flash /srv/engram-local/DeepSeek-V4.1-Flash

Allow approximately 510 GB for the complete checkpoint, roughly 48 GB allocated per worker for sparse Engram rows, plus runtime and cache storage. Use a sparse-file-capable filesystem. The supplied preparation helper refuses to overwrite destination shards.

Load the exact runtime

Download the R37 Docker archive: 40,716,629,504 bytes. This includes the compiled runtime, pinned dependencies, and original image layers. Model weights and node-local Engram overlays are downloaded/prepared separately. No separate private base image is needed to load this archive. The build/ files document its construction; they are not a standalone public-base rebuild procedure.

hf download 0xTank/DeepSeek-V4.1-Flash-FP8-R37-4x-GB10 runtime/ds41-r37-runtime.tar --local-dir ./r37
echo '984e4e5ea68c3bac50c8fa2144dddc353b755d869ac734007b5614bfe87316d1  r37/runtime/ds41-r37-runtime.tar' | sha256sum -c -
docker load -i r37/runtime/ds41-r37-runtime.tar
docker image inspect local/ds41-r37-spark:runtime --format '{{.Id}}'

The image ID must be sha256:638d63b0f4bb9741c4afd4936e184659053e8a5ec466d6e7a8994ae0094e58fa. Load the same archive on all four nodes. It requires ARM64 Linux, NVIDIA GPU container support, a CUDA-13-compatible driver, and working RoCE. Allow at least 85 GB of additional free storage per node for the downloaded archive and unpacked image, plus checkpoint, Engram and cache space.

The archive was inspected across all 63 saved layers, including hidden earlier files. No user HF credentials, private SSH keys, or private cluster identifiers were found by the release checks. Matches were reviewed as upstream library code/test fixtures and cached dependency wheels. This is a targeted release privacy review, not a guarantee that every third-party dependency is secure.

Agent/operator walkthrough

Follow these steps in order. Commands are run on the four Linux Spark hosts, not on the machine reading this page. Use ranks 0, 1, 2, 3, one per host. This is a documented reproduction procedure; it has not yet been validated as a clean-room installation of the public package.

1. Check the hosts and private fabric

Each host needs ARM64 Linux, a 128 GB GB10 GPU, Docker with NVIDIA GPU-container support, a CUDA-13-compatible NVIDIA driver, Python 3, the Hugging Face CLI, and sufficient local storage. Budget approximately 510 GB for a complete checkpoint, another roughly 48 GB allocated on each worker for sparse Engram rows, and at least 85 GB for the runtime archive plus loaded image, with additional cache and operating-system headroom.

Run on every host:

uname -m
nvidia-smi
docker info
df -h /srv
ip -br address
rdma link show
ibv_devinfo

Expect aarch64, a visible GB10 GPU, working Docker, and an active RDMA device. RDMA tools must be installed by the host administrator if absent. Determine each host's private fabric IP, interface name, HCA and appropriate RoCE GID index; example values are not hardware discovery. Confirm private-IP reachability between all ranks. Ping alone does not prove RDMA works: validate the fabric with your site's RDMA/NCCL connectivity test before launching. Do not automatically reconfigure networking, disable a firewall, or stop an existing model to make this recipe work. The head uses coordination port 29543; distributed workers also need private inter-node connectivity. Keep the fabric private rather than exposing these ports to the internet.

2. Download the recipe and choose one checkpoint

On every host, use a writable working directory:

mkdir -p "$HOME/r37-recipe"
cd "$HOME/r37-recipe"
hf download 0xTank/DeepSeek-V4.1-Flash-FP8-R37-4x-GB10 \
  --include 'README.md' 'launch/*' 'tools/*' 'bench/*' \
  --local-dir .

Choose exactly one checkpoint for the whole cluster. Obtain any required access through HF; do not put tokens in cluster.env, scripts or published logs. Have the administrator provide writable model, Engram and cache directories, then download to the selected model directory:

# Choose ONE:
hf download deepseek-ai/DeepSeek-V4.1-Flash --local-dir /srv/models/DeepSeek-V4.1-Flash
# OR:
hf download dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8 --local-dir /srv/models/DeepSeek-V4.1-Flash-dealignai

Use the same checkpoint revision on all four hosts; for a pinned deployment, pass the same --revision COMMIT to each download. The complete checkpoint includes tokenizer/configuration files, the safetensors index and every referenced shard. A read-only shared model mount is an alternative, but worker Engram overlays must remain local. Do not substitute another quantization or mix shard sets. The measurements above used dealignai FP8, not a paired validation of both checkpoints.

3. Obtain and verify the exact runtime — availability gate

The verified runtime archive is now published at runtime/ds41-r37-runtime.tar. Download it and verify the checksum below before loading. If the file cannot be downloaded or verification fails, stop and investigate; do not substitute an unrelated image or attempt to rebuild from the private base named in the provenance files.

hf download 0xTank/DeepSeek-V4.1-Flash-FP8-R37-4x-GB10 \
  runtime/ds41-r37-runtime.tar --local-dir .
echo '984e4e5ea68c3bac50c8fa2144dddc353b755d869ac734007b5614bfe87316d1  runtime/ds41-r37-runtime.tar' | sha256sum -c -
docker load -i runtime/ds41-r37-runtime.tar
docker image inspect local/ds41-r37-spark:runtime --format '{{.Id}}'
docker run --rm --gpus all --entrypoint nvidia-smi local/ds41-r37-spark:runtime

Expected archive size: 40,716,629,504 bytes. Expected image ID: sha256:638d63b0f4bb9741c4afd4936e184659053e8a5ec466d6e7a8994ae0094e58fa. Verify and load this same image on every rank. A private-LAN copy of the verified archive is fine; verify the destination checksum too. The archive does not contain the model checkpoint or your worker overlays.

4. Configure each host

cp launch/cluster.env.example cluster.env

Edit cluster.env before proceeding. Set the same ordered NODE_IPS array on every host. Set FABRIC_IFACE, IB_HCA and IB_GID_INDEX to that host's actual fabric devices. Set MODEL_PATH to the checkpoint selected above and ENGRAM_PATH and CACHE_PATH to writable local directories. Retain MAXLEN=600000, SEQS=8, GMU=0.81, R37_USE_LOCAL_ENGRAM_EDGES=1 and the exact R37 image tag to reproduce this configuration. Leave K3 unchanged. This environment file is sourced as shell code; only use a file you trust.

5. Prepare local Engram rows on workers only

Run once on ranks 1, 2 and 3, substituting that worker's actual rank:

cd "$HOME/r37-recipe"
source ./cluster.env
RANK=1  # use 2 on rank 2; use 3 on rank 3
python3 tools/prepare-engram.py "$RANK" "$MODEL_PATH" "$ENGRAM_PATH"
du -h "$ENGRAM_PATH"

Rank 0 reads the complete checkpoint and does not run this helper. The helper writes sparse overlays for shards 47 and 48 plus r37-row-coverage.json; their apparent size is larger than allocated disk usage. Use a sparse-file-capable local filesystem. It refuses to overwrite existing destination shards. If preparation fails, inspect the error and use a fresh destination after correcting the cause; do not blindly delete or reuse partial/different-checkpoint overlays. The helper is specific to the expected 48-shard checkpoint layout and does not validate an arbitrary replacement model.

6. Review and launch one rank per host

On each host first inspect the generated command, using its own rank:

DRY_RUN=1 bash launch/node.sh 1 cluster.env

Then execute the following on the indicated hosts, in order:

# On rank 3:
bash launch/node.sh 3 cluster.env
# On rank 2:
bash launch/node.sh 2 cluster.env
# On rank 1:
bash launch/node.sh 1 cluster.env
# On rank 0 LAST:
bash launch/node.sh 0 cluster.env

These are four separate host commands, not four commands to run on the head. The default container name is ds41_r37. The launcher refuses an existing container of that name; inspect it rather than removing or replacing it automatically. Workers may wait for rank 0. Startup includes weight loading and compilation; a returned container ID is not proof the API is ready.

7. Verify readiness and make a request

Inspect every rank for startup failures:

docker logs --tail 100 ds41_r37
docker ps --filter name=ds41_r37

On rank 0, wait for successful health and model-list responses:

curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/v1/models
curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"Write two sentences about the ocean."}],"max_tokens":128,"temperature":0,"chat_template_kwargs":{"thinking":false}}'

The smoke test explicitly disables thinking to check visible text with a small token budget. Normal serving defaults remain thinking on/xhigh, temperature 1.0 and top-p 0.95. 127.0.0.1 means the machine running the command; remote clients must use the head's reachable address. Port 8000 has no built-in authentication in this launcher: use a trusted network or an authenticated gateway. Do not expose it publicly as-is.

Do not benchmark until all ranks are healthy and the API is idle. Benchmark settings must match the methodology above, including thinking off; normal agent requests with xhigh are not directly comparable to those tables. Multimodal input is enabled, but the forced-named-tool limitation documented above still applies. If startup fails, collect the relevant error from each rank and verify image ID, checkpoint completeness, paths and fabric configuration before changing settings. Do not publish credentials or private logs when requesting help.

Credit

This is a derivative deployment recipe. The model, inference runtime, benchmark suite, and deployment adaptations have different authors:

• DeepSeek: created the original DeepSeek-V4.1-Flash model, checkpoint, and embedded DSpark draft components. DeepSeek retains credit for the model itself.
• Local Inference Lab: provided the R37 runtime source composition used here, including its vLLM implementation and B12X attention, linear, and MoE execution paths. This is the upstream runtime foundation for our Spark adaptation, not an inference engine authored by 0xTank. See their DeepSeek-V4.1-Flash implementation and deployment notes and Blackwell deployment repository. This recipe pins vLLM source c687594b8a8082e18af9fe2f64eb5f9ee442e251 and B12X source f1c4e9dd5b1d841c10cda70ee49c3190cf3db0be; the upstream documentation may describe newer releases. Our published measurements are for this R37-based Spark build, not upstream R38 results.
• Tony / Tech2Wild and Kai: provided the original four-Spark deployment recipe and the benchmark suite. Credit for the fixed workloads, prompt set, and benchmark timing methodology belongs to them. The tables on this page are measurements collected by 0xTank on this deployment, not Tony's hardware results.
• dealignai: provided the modified FP8 checkpoint used for the measurements on this page. The recipe also supports the standard DeepSeek FP8 checkpoint; supporting both does not mean both were used for these reported measurements.
• PyTorch, NVIDIA, FlashInfer, Triton, vLLM, and other upstream contributors: provided the underlying framework, GPU libraries, compiler infrastructure, and serving components. Their original licenses and attribution continue to apply.

0xTank's contribution: adapted and packaged the R37 build for ARM64/SM121 on four DGX Sparks; installed required CUDA development headers; corrected CUDA driver linkage; pinned FlashInfer 0.6.17 and InstantTensor 0.2.0; adapted local Engram row boundaries; configured the four-node deployment; and ran and published the measurements and reproduction instructions. These deployment changes do not constitute authorship of DeepSeek's model, Local Inference Lab's runtime, or Tony's benchmark suite.

Original component licenses continue to apply. This recipe does not claim ownership of upstream model or runtime code, and linking an upstream project does not imply its authors endorse this deployment.