SJTU-compression/qwen-3-vl-4b-instruct-int2

VL-QAT — Qwen3-VL-4B-Instruct at 2 bits

A 2-bit weight-quantized Qwen3-VL-4B-Instruct that keeps 88.7% of the bf16 model's average score across GSM8K, MMStar and RealWorldQA. The same quantization format applied with GPTQ keeps 21.9%.

Quantization format

tensor group	format
language-model Linear weights	INT2, asymmetric, group size 32
vision-tower Linear weights	FP4 (E2M1), asymmetric, group size 32
tied embed_tokens / lm_head	FP4 (E2M1), asymmetric, group size 32
LayerNorm, 1-D biases, patch Conv3d	bf16

Activations are not quantized. Per group, storage is bits + 16/32 for the scale plus bits/32 for the asymmetric zero point.

• Decode-path effective bpw: 2.76 — averaged over the tensors read on every generated token (language Linear plus the tied embedding). The vision tower runs once during prefill and is excluded.
• Packed model size: 1.63 GB, against 8.88 GB at bf16 — a 5.4× reduction.

Results

Measured with EvalScope on the full test sets, non-thinking mode. keep ratio is the average score divided by the bf16 average.

method	decode bpw	GSM8K	MMStar	RealWorldQA	avg	keep ratio
bf16	16.00	94.62	65.73	66.01	75.45	100.0%
4-bit RTN	4.50	93.71	62.20	66.54	74.15	98.3%
3-bit GPTQ	3.69	92.65	61.27	66.93	73.62	97.6%
2-bit GPTQ	2.76	1.52	28.27	19.87	16.55	21.9%
2-bit QAT-SFT only	2.76	86.50	47.33	25.88	53.24	70.6%
2-bit this model	2.76	87.41	53.20	60.26	66.96	88.7%

At 2 bits, post-training quantization collapses: GPTQ scores 1.52 on GSM8K. The recovery comes in two stages, and both are needed — the quantization-aware cold start restores language ability, and the on-policy stage restores vision, taking RealWorldQA from 25.88 to 60.26.

Method

Stage 1 — quantization-aware SFT. Weights are fake-quantized to the frozen format on every forward pass while the model is distilled from the original bf16 checkpoint over one epoch of a 1.35M-row text corpus (14,000 steps, lr 1e-5 cosine to 1e-6, sequence length 8192). The vision tower is frozen; unfreezing it cost 18.3 MMStar points in ablation.

Stage 2 — on-policy distillation. 400 steps at batch 64, rollout capped at 2048 tokens, lr 1e-6. The student generates, and the original bf16 model scores those generations as the teacher; the loss is a top-64 forward KL over the student's own distribution. The prompt pool is 25,600 image-text items with no text-only rows, 75% of them rewritten to ask for a worked answer ending in ANSWER:.

The quantization format is identical in both stages and in the released weights.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "chizhongw/VL-QAT", dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("chizhongw/VL-QAT")

Requires a transformers version with Qwen3-VL support; the export was written by transformers 5.3.0.dev0 and uses the consolidated processor_config.json format rather than the older split preprocessor_config.json.

Sampling settings matter

Use a presence penalty. These were the settings used for every number above:

temperature 0.7 · top_p 0.8 · top_k 20 · presence_penalty 1.5

Without a presence penalty the model falls into repetition loops on roughly a quarter of open-ended prompts; with presence_penalty 1.5 that drops to about 5%. Greedy decoding is the worst case — do not use temperature 0 with no penalty.

What this checkpoint is, and is not

The weights are stored dequantized in bf16: every value sits on the INT2 or FP4 grid it was quantized to, but each is written as a bf16 number. The repository is therefore 8.3 GB, not 1.63 GB, and loading it saves no memory over the bf16 base model.

This format exists so the accuracy above is reproducible through the standard transformers and vLLM paths with no custom kernel. The 1.63 GB figure is what the format implies once weights are bit-packed; producing a packed artifact requires a separate exporter and an inference kernel that reads it.

Known limitations

• MMStar is the weakest axis at 81% of bf16, against 92% for GSM8K and 91% for RealWorldQA. Roughly 5 of the missing points are inherent to weight quantization at any precision — RTN FP4, which is untrained, loses 4.2 points on MMStar while losing almost nothing on GSM8K.
• Non-termination. On about 10% of MMStar items the model generates to the context cap instead of stopping. Set an explicit max_tokens.
• Numeric degeneration. Prompts that invite a long digit string (for example "what is 2 to the power of 10") can produce a correct prefix followed by a run of repeated digits. This is present at 2 bits and absent in the bf16 base model.

License

Apache 2.0, inherited from Qwen/Qwen3-VL-4B-Instruct.