ProprietaryLegal/DeepSeek-V4-Flash-V100

DeepSeek-V4-Flash on V100 — working serving recipe, bug fix, and measured numbers

PLI Labs. This repository documents (and ships the patches for) running DeepSeek-V4-Flash (284B/~13B-active MoE) correctly on eight 2017-era NVIDIA V100-SXM2-32GB GPUs with llama.cpp — to our knowledge the first published Volta datapoint — including the root cause and fix for a silent-output-corruption bug that affects every backend when the K-cache is quantized (upstream: ggml-org/llama.cpp#25382).

No model weights here: use unsloth's GGUF quants (unsloth/DeepSeek-V4-Flash-GGUF, UD-Q4_K_XL, 144 GiB) and verify shard sha256s against the LFS manifest.

Measured (8× V100, layer split, FA on, power-capped 245 W/GPU)

metric	value
generation	12.9 t/s (-b 2048 -ub 1024)
prefill @ 8K	205–209 t/s
scaling	8 GPUs optimal — 10 ≈ par, 14 halves throughput (PCIe island hops)
quality	legal-drafting screen 6/6 + verbatim needle retrieval at ~7K tokens

The one thing you must know

With --cache-type-k q8_0, this model emits confident gibberish behind a healthy /health — on CPU and CUDA alike. Cause: llama.cpp's quantized-KV Hadamard rotation diverts DeepSeek-V4 off its sparse attention paths into a fallback with broken rotation math. Either run --cache-type-k f16 (zero-cost, MLA's cache is tiny) or apply the patches here (also on the ds4-volta-fix branch), which disable the rotation for this architecture — verified with a full K/V-type matrix on CPU and GPU.

Quickstart

git clone -b ds4-volta-fix https://github.com/Mermiges/llama.cpp
cmake -S llama.cpp -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j
build/bin/llama-server -m DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf \
  -ngl 999 -sm layer -fa 1 -b 2048 -ub 1024 -c 8192 \
  --cache-type-k f16 --cache-type-v f16

Update: the proper fix (rotation-aware attention) is also available

Beyond the minimal fix (patches in the repo root), patches-rotation-aware/ (branch ds4-rotation-aware) makes DeepSeek-V4's attention paths rotation-aware and re-enables the quantized-KV incoherence rotation — verified coherent across the full K/V-type matrix, ~6.8% decode cost, and in a ~6000-token needle screen the rotation-ON build retrieved a needle that plain quantized-K missed. patches-tensor-parallel/ carries the experimental de-blacklisting of DeepSeek-V4 for llama.cpp's tensor-parallel mode (loads and reaches decode on 8× V100; one scheduler assert remains).

Full research record (evidence matrices, raw benchmarks, negative results): https://github.com/ProprietaryLegal/deepseek-v4-flash-v100

PLI Labs — Proprietary Legal Intelligence, LLC. Research artifacts for lawyer-supervised, on-premises legal AI; nothing here is legal advice.