One-command deployment on a single DGX Spark (34-42 tok/s, prefix caching, 262K)
For anyone pulling this checkpoint for a DGX Spark (GB10, 128GB unified): it runs well on a single box. SGLang plus two sha256-pinned patches that mmap the 51B n-gram table from NVMe (it never needs to be resident, credit hashd1ve), MTP head active, full 262K context, vision, tool calls, and working prefix caching. 34 to 42 tok/s decode measured.
One-command install, revision-locked to this repo's snapshots, with a systemd service and clean uninstall: https://github.com/hasso5703/dgx-spark-qwen38 (target flash). Benchmarks and GB10-specific traps: https://forums.developer.nvidia.com/t/qwen3-8-flash-next/381228/194
For anyone pulling this checkpoint for a DGX Spark (GB10, 128GB unified): it runs well on a single box. SGLang plus two sha256-pinned patches that mmap the 51B n-gram table from NVMe (it never needs to be resident, credit hashd1ve), MTP head active, full 262K context, vision, tool calls, and working prefix caching. 34 to 42 tok/s decode measured.
One-command install, revision-locked to this repo's snapshots, with a systemd service and clean uninstall: https://github.com/hasso5703/dgx-spark-qwen38 (target flash). Benchmarks and GB10-specific traps: https://forums.developer.nvidia.com/t/qwen3-8-flash-next/381228/194
Is it possible to do single docker-compose.yml file only? Not even referencing chat template or requiring an entire repo or anything.
Thanks @gossi23 but your Docker compose doesn't have any flags about PLE. So wouldn't that try to load the Ngram on to the memory, therefore make the machine run out of memory and hang?
Apologies I don't know what PLE is (>_<)
I basically got the config from the SGLang cookbook
https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B#hw=dgx-spark&variant=default&quant=nvfp4-bf16-head&nodes=single&spec=dspark&tier=low-latency&ssmDtype=float32
Then added / tweaked the context length until it was able to start the model without memory issues
After a cold start, according to btop it's using 106GB of 121GB total RAM
I'm getting ~15 - 30tps
That's for the 27B model, not Flash Next. Flash Next model is 5x bigger, with 2x bigger Ngram model (that's supposed to be loaded on to storage, therefore PLE function is needed).
You're in the wrong discussion thread. I probably should have mentioned it in your previous post. Plus, what you posted was Docker run command, not a Docker compose.
For 27B model, you can go as low as mem-fraction-static 0.36 with the max 262k context (I see you're using yarn with 524k, unsure if that's ideal for your use), which is only ~40GB of memory instead of using 106GB. With the extra memory, you can even run ComfyUI (allocating maybe max 60GB including LoRAs etc.) which the Qwen3.8 27B model can control and generate multimedia for you. Or TTS. Or music generator. Etc.