Run on DGX Spark cluster
Hi I`m planning on runnig this version, normal dspark and regular flash on a 2 dgx spark cluster to test precision and speed. Do you have any suggestions on the vllm config?
You have to change the code of vllm for deepseek v4 .
Required vLLM patch
Stock vLLM validates that every loaded tensor matches the model's parameter shape. Because this checkpoint keeps the original 6-column tid2eid routing table while the config now declares 4 experts-per-token, loading fails with:
AssertionError: Attempted to load weight (torch.Size([129280, 6]))
into parameter (torch.Size([129280, 4]))
Fix (one line). In vLLM's DeepSeek-V4 loader — for the build used here …/site-packages/vllm/models/deepseek_v4/nvidia/model.py (path may vary by version; search for def load_weights) — inside the generic else branch, before the weight_loader(param, loaded_weight) call, slice the routing table to the configured expert count:
else:
if is_pp_missing_parameter(name, self):
continue
param = params_dict[name]
# --- top-k=4 patch: slice the 6-column tid2eid routing
# table down to num_experts_per_tok columns ---
if "tid2eid" in name and loaded_weight.shape != param.shape:
loaded_weight = loaded_weight[:, : param.shape[1]].contiguous()
weight_loader = getattr(
param, "weight_loader", default_weight_loader
)
weight_loader(param, loaded_weight)
loaded_params.add(name)
continue
This keeps the top-4 hash-routing entries per token and discards the unused 5th/6th columns, matching num_experts_per_tok = 4 in config.json.
Auto-apply (copy-paste). Idempotent script that locates your installed vLLM and inserts the patch in the right place:
python - <<'PY'
import pathlib, vllm
p = pathlib.Path(vllm.file).parent / "models/deepseek_v4/nvidia/model.py"
src = p.read_text()
if "tid2eid" in src:
print("already patched:", p); raise SystemExit
needle = (" param = params_dict[name]\n"
" weight_loader = getattr(\n")
patch = (" param = params_dict[name]\n"
" if "tid2eid" in name and loaded_weight.shape != param.shape:\n"
" loaded_weight = loaded_weight[:, : param.shape[1]].contiguous()\n"
" weight_loader = getattr(\n")
assert needle in src, "loader block not found -- check your vLLM version / file path"
p.write_text(src.replace(needle, patch, 1))
print("patched:", p)
PY
The exact file path can differ between vLLM builds (e.g. vllm/model_executor/models/deepseek_v4.py in some releases). The script resolves it from the installed vllm package; if the loader block isn't found, open the file, find def load_weights, and add the two tid2eid lines before the generic weight_loader(param, loaded_weight) call.