boxwrench/Ling-3.0-flash-int4-generic

Ling-3.0-flash — Wrench INT4 (generic)

An INT4 compressed-tensors S4/g32 quant of inclusionAI/Ling-3.0-flash, using the official INT4 tensor layout, configuration and tokenizer. Calibrated on public corpora only.

Base model	Ling-3.0-flash (MIT); architecture
Runs with	vLLM; measured build 0.26.1rc1.dev467+g838616875 (ROCm)
Tested fit	8060S / Max+ 395, 128 GB unified memory; check HF file sizes before downloading
Full results, method, reproduce	Ling-3.0 page

Results

Model	Status	Prompt	Prefill tok/s	Decode tok/s	TTFT (s)
Generic	SERVES OK	1K	953.4 [386.7-956.8]	29.49 [28.11-29.54]	1.10 [1.09-2.71]
Generic	SERVES OK	8K	1089.2 [1083.6-1089.5]	27.74 [25.43-28.07]	7.54 [7.54-7.58]

Measured 2026-10-17 on Radeon 8060S, Ryzen AI Max+ 395, 128 GB unified; Linux/ROCm, vLLM build above. Greedy, 64 generated tokens, MTP enabled; three-run medians [min–max], nonce per request. Keep ranges beside comparisons. Serving is not a quality measurement; no task-accuracy claim. Receipt and limits.

Quality

Metric	Official INT4	Generic
KL to FP8 ↓	0.1402	0.1281
Top-1 agreement with FP8 ↑	92.61%	93.00%

Against Ling-3.0-flash-fp8 on a fixed 92 × 4096-token corpus; not task accuracy. Per-domain table.

Run it

Prepare a compatible vLLM installation with INSTALL. Measured on weight revision 0b2bc30f5fe4. The vLLM build was a ROCm build, not an upstream release, so compare checkpoints on one server rather than absolute numbers across builds. Untested download and launch example; set MODEL to a local directory and review checkpoint code before trusting it:

hf download boxwrench/Ling-3.0-flash-int4-generic --local-dir "$MODEL"
vllm serve "$MODEL" --host 127.0.0.1 --port 8000 \
  --attention-backend TRITON_MLA --moe-backend triton \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 6 \
  --max-num-batched-tokens 8192 --enable-chunked-prefill --enable-prefix-caching \
  --mamba-cache-mode align --trust-remote-code

Discrete GPUs, SGLang and CUDA are untested. Agent setup: AI_SETUP.

Your numbers

How to measure and report.

Credits

inclusionAI for the base model and official INT4 (MIT); vLLM for serving (Apache-2.0). Licence notices.