Qwen3.5-9B-FP8-Dynamic

English | 日本語

Quantized from Qwen/Qwen3.5-9B, revision c202236235762e1c871ad0ccb60c8ee5ba337b9a, by InfiniCloud.

FP8 weights with static per-channel scales and dynamic per-token FP8 activations. No calibration dataset.

This is a mixed-precision multimodal checkpoint. Quantization covers 128 Linear modules, containing 5,301,600,256 weight elements out of 9,409,813,744 loaded model parameters. Vision modules, Gated DeltaNet linear_attn projections, lm_head, embeddings, and other non-Linear parameters remain unquantized. This exclusion is a conservative recipe choice. The original vision processor is retained; calibration for INT4 used text only.

MTP weights are not included because the quantization model class does not load them. KV-cache quantization is not part of this recipe. Source weights include MTP tensors, so source/candidate file-size reductions also include that omission.

Intended VRAM: one L4 24GB

The deployment target is one dedicated NVIDIA L4 with 24GB VRAM, tensor parallel size 1, with no CPU weight offload. For initial L4 validation, plan for an 8,192-token total context and one concurrent request, retaining BF16 KV, prefix caching disabled, and at most one image / no video per prompt. L4 has not yet been tested; this is a capacity-planning target, not a verified minimum VRAM requirement or a guarantee of compatibility.

The existing Blackwell startup logs provide the following reference values. These are GPU-memory measurements, separate from weight file sizes.

Model	GPU memory reported for model loading	Sum of profiled non-cache components
FP8	12.76 GiB	approximately 15.10 GiB
INT4 G128	10.35 GiB	approximately 12.70 GiB

The last column adds weights plus non-PyTorch consumption, peak activation, and actual CUDA graph allocation: 13.06 + 1.98 + 0.06 GiB for FP8, 10.66 + 1.98 + 0.06 GiB for INT4. It excludes the attention KV / hybrid-state cache pool. These sums use rounded startup diagnostics; they are neither whole-run peak measurements nor minimum requirements. The reference setup used vLLM 0.29.0, context 16,384, maximum 8 sequences, batched-token limit 8,192, non-eager execution, and at most one image / no video per prompt. Encoder profiling included one image at the maximum feature size; the performance requests themselves were text-only.

Cache demand also depends on context length and concurrency. From the stored model configuration, the eight full-attention layers use 32 KiB per token of BF16 KV content across the model. The 24 GatedDeltaNet layers add 49.125 MiB per active request of raw state with BF16 convolution state and FP32 recurrent state. Under tensor parallel size 1, prefix caching disabled, and no speculative decoding:

Total tokens per request	Concurrent requests	Attention KV content	KV plus GatedDeltaNet state content
8,192	1	0.250 GiB	approximately 0.298 GiB
8,192	8	2.000 GiB	approximately 2.384 GiB
16,384	1	0.500 GiB	approximately 0.548 GiB
16,384	8	4.000 GiB	approximately 4.384 GiB

Total context includes input, generated tokens, and any image tokens. This table describes tensor contents, not the allocated cache pool or total VRAM; block rounding, padding, workspaces, and runtime headroom are additional. Prefix caching can also retain additional states. FP8/INT4 weight quantization does not reduce the BF16 KV sizes in this table.

For the intended dedicated L4 setup, use --gpu-memory-utilization 0.90 as an initial planning setting: the executor budget is 0.90 × the total memory reported by CUDA. For illustration only, if CUDA reports 22.5 GiB, the budget is 20.25 GiB, leaving approximately 5.15 GiB for FP8 or 7.55 GiB for INT4 after subtracting the Blackwell non-cache reference. This is an allowance for the cache and further overhead, not a proven usable KV capacity. L4's advertised 24GB must not be substituted as 24 GiB; check its actual reported capacity. Different L4 kernels, image sizes, and serving settings can change these figures.

vLLM normally uses the remaining executor budget to preallocate cache. Consequently, nvidia-smi usage can be much larger than the weights and live request contents. The 60.92 / 63.33 GiB cache allocations in our 96GB Blackwell runs are automatic reservations, not model requirements. See vLLM memory tuning. Re-profile on L4; start with --max-model-len 8192 --max-num-seqs 1, then increase context or concurrency after checking startup and peak memory with representative text and images. The 16K × 8 rows above are sizing examples, not validated L4 operating points.

The log excerpts, source hashes, assumptions, and calculations are included in VRAM planning evidence. This guidance covers inference, not quantization or fine-tuning.

Artifact and runtime verification

• Weight file: 13,520,203,008 bytes (13.52 GB; 12.59 GiB).
• SHA256: a928657692dbbad12dc248c51f82db70c6ccf868a486b25395028d223eb14768.
• All 888 floating tensors are finite; all 128 quantization scales are positive and finite. See numerical evidence.
• vLLM loaded each checkpoint and returned HTTP 200 responses with normal completion for the three fixed smoke requests. Per-case correctness and formatting outcomes are reported below, including failures.
• Generated using llmcompressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.14.1, PyTorch 2.13.0+cu130 on one RTX PRO 6000 Blackwell Max-Q 96GB.
• Inference/evaluation used vLLM 0.29.0 with compressed-tensors 0.17.0, Transformers 5.14.1, and driver 580.159.04. The generation and inference environments are separate.

Smoke outcomes and limitations

Model	Japanese text	Arithmetic JSON	Image shapes / strict JSON
BF16	Pass	Pass	Pass
FP8	Pass	Pass	Pass
INT4	Pass	Pass	Fail (strict_json, expected_answer)

INT4 image_shapes returned Markdown-fenced output and failed the strict JSON requirement. Raw content and the original checks are preserved in results.json; no Markdown stripping or outcome reclassification is applied. These three cases do not establish quantitative image or application-quality performance. Smoke servers were explicitly launched with enforce_eager=True.

Quality comparison

Measured on 2026-09-12 using lm-eval 0.4.13: full GSM8K test set (1,319 questions) and Japanese MGSM native chain-of-thought test set (250 questions), both 5-shot. The original tokenizer/revision and chat template were shared across all three models, with multi-turn few-shot formatting, thinking disabled, temperature 0, and at most 2,048 generated tokens. Context limit was 8,192; all formatted prompts plus the output budget fit. Task prompts, stop sequences, filters, and dataset revisions are bundled in evaluation.

Model	GSM8K strict / flexible (%)	Japanese MGSM strict / flexible (%)
BF16	93.40 / 93.56	2.80 / 79.60
FP8	93.10 / 93.78	2.80 / 78.80
INT4	92.27 / 92.95	2.40 / 76.80

Strict and flexible answer extraction are both shown. The upstream Japanese strict extractor requires the exact answer-marker format: spaces or intervening units, such as 答えは 3 です。 or 答えは18ドルです。, can fail strict extraction despite containing a correct numeric answer. A low strict score therefore combines answer correctness with formatting sensitivity; inspect the flexible metric and sample outputs alongside it. The metrics, standard errors, sample counts, and protocol are in results.json; no statistical significance or automatic acceptance threshold is asserted. Japanese MGSM is a translated mathematics benchmark and does not establish broad Japanese application quality.

Exploratory serving performance on Blackwell

These measurements use RTX PRO 6000 Blackwell Max-Q 96GB, not L4. Each cell contains 64 synthetic requests with fixed token lengths, seed 42, ignored EOS, and unlimited request arrival rate under the stated concurrency cap. The server used BF16 KV cache, a 16,384-token context limit, maximum 8 sequences, chunked prefill, prefix caching disabled, and GPU memory utilization 0.8. Recorded serving configuration used enforce_eager=False; this setting alone is not evidence of which CUDA graphs actually executed. Short-run p95 values are exploratory.

The benchmark used zero request warmups and disabled its initial readiness-test request (ready_check_timeout_sec=0). Server startup is outside measurement, but kernel JIT during measured requests is included; these are not warmed steady-state results. Source inspection indicates that performance requests inherit vLLM's temperature 1.0 with generation-config=vllm; request payloads were not captured. The quality evaluation above explicitly uses temperature 0.

Model	Input / output tokens	Concurrency	TTFT p50 / p95 (ms)	TPOT p50 / p95 (ms)	Output tok/s	Completed
BF16	512 / 256	1	66.03 / 70.38	12.16 / 12.24	81.33	64 / 64
BF16	512 / 256	8	291.90 / 302.90	12.42 / 13.11	593.28	64 / 64
BF16	8192 / 256	1	609.92 / 620.76	12.38 / 12.40	68.26	64 / 64
BF16	8192 / 256	8	2817.49 / 3918.75	19.96 / 26.71	257.39	64 / 64
FP8	512 / 256	1	69.84 / 75.03	8.49 / 8.56	114.76	64 / 64
FP8	512 / 256	8	226.32 / 261.48	9.01 / 9.37	810.10	64 / 64
FP8	8192 / 256	1	444.19 / 454.46	8.73 / 8.75	96.24	64 / 64
FP8	8192 / 256	8	1995.29 / 3123.13	14.76 / 19.46	354.38	64 / 64
INT4	512 / 256	1	66.60 / 70.98	6.60 / 6.67	146.55	64 / 64
INT4	512 / 256	8	296.54 / 320.68	7.10 / 7.85	969.56	64 / 64
INT4	8192 / 256	1	637.14 / 644.84	6.82 / 6.85	108.16	64 / 64
INT4	8192 / 256	8	3215.99 / 4635.18	13.88 / 21.73	300.84	64 / 64

No L4 hardware measurement has been performed. File size and the Blackwell results do not establish L4 runtime memory fit, throughput, latency, or concurrency limits.

Use and reproduce

Use a separate environment with vllm==0.29.0 and transformers==5.14.1. The tested checkpoint loads its stored compressed-tensors quantization configuration automatically. Follow evaluation commands and environment for the exact smoke, quality, and performance settings, and standalone quantization code to regenerate either recipe.

For INT4, the calibration dataset is HuggingFaceH4/ultrachat_200k, immutable revision 8049631c405ae6576f93f445c6b8166f76f5505a. Selected row IDs and token hashes are in the INT4 quantization manifest. The corpus is English-focused; Japanese-specific calibration, quantitative vision evaluation, tool calling, MTP, and L4 deployment are not validated here.

A normalized exact-text check found none of the 1,569 complete evaluation questions within the 512 actual truncated INT4 calibration samples. The check reproduced the manifest token hash and used Unicode NFKC, casefolding, and whitespace normalization. It does not assess paraphrases, semantic overlap, or base-model pretraining contamination; its scope and evidence are recorded in results.json.

The quantization manifest records the original structural validation and generation provenance; later inference results are recorded separately. GPU reruns are not promised to be bit-identical. Original model weights retain the Qwen Apache-2.0 license; bundled tools and evaluation sources retain their included licenses and attribution.