YTan2000/Qwen3.8-Flash-Next-tqLLM

Qwen3.8 Flash-Next for tqLLM

The extra files tqLLM needs to run Qwen3.8 Flash-Next (125B mixture of experts) on one 24 GB GPU, on top of the ISTA-DASLab GSQ-RCO IQ3_S GGUF. The routed experts stay in system memory and a cache of them lives on the GPU. At start-up tqLLM rearranges the experts in memory for that cache (under a minute; the server is ready in about two), so the GGUF is the only large download.

On one RTX 3090 with tqLLM 1.2, against Strata 0.2.2 on the same GGUF: 1.63x faster decode after a 128K-token prompt (77.1 against 47.2 tok/s) and about 3x sooner first token on short chats, measured with tqLLM 1.2 on the same card. Method: tqllm.com/performance.

File	Size	What it is
mtp-blk48-iq4.gguf	1.46 GB	the model's own multi-token-prediction layer, IQ4, for exact speculative decoding
flashnext-tokenizer.gguf	11 MB	tokenizer and chat template
atx_65536.txt	0.4 MB	draft vocabulary for the MTP drafter
flashnext.tqfn	1 KB	the manifest tqLLM opens: lists the files above (paths relative to it) and the per-layer expert cache plan

Run

Needs a 24 GB NVIDIA GPU (RTX 30 or newer) or a GB10 / DGX Spark, 64 GB of system memory or more (51 GB of experts live there; the numbers above used 128 GB) and tqLLM 1.2.3 or newer (the engine sizes its expert cache on the device). Put everything in one folder:

hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include "IQ3_S/*" --local-dir fn
mv fn/IQ3_S/* fn/
hf download YTan2000/Qwen3.8-Flash-Next-tqLLM --local-dir fn
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/fn:/fn:ro" --entrypoint tqserve \
  ytan2000/tqllm:1.2.3 --model /fn/flashnext.tqfn --tokenizer /fn/flashnext-tokenizer.gguf \
  --context 32768 --host 0.0.0.0 --port 8086

Then curl -s http://localhost:8086/health and any OpenAI-compatible client at http://localhost:8086/v1.

The server is ready after about two minutes.

Coming next

Further compression is on the way.

Credits and licence

Apache-2.0, as the sources. Model: Qwen team (Qwen/Qwen3.8-Flash-Next). Quantization of the main weights: IST Austria DAS Lab (GSQ + RCO). The files here are derived from those weights; see NOTICE.