Sage-1.5-W4A16

W4A16 compressed-tensors quantization of InternScience/Sage-A1.5 for fast text serving on Intel Arc GPUs. Follows the base model license (Apache-2.0).

• Quant-first: int4 symmetric weights (group 128), fp16 compute, packed compressed-tensors export — tuned for bandwidth-bound single-stream decode on a single 32 GB card.
• Server-validated: every number below was measured on the same live serve, not estimated.
• Faithful: unquantizable tensors (MTP, vision, lm_head, shared-expert gates) ship alongside as raw BF16 instead of being silently dropped.

Model details

Property	Specification
Base checkpoint	InternScience/Sage-A1.5, text use (--language-model-only)
Parameters	~36B (Qwen3_5MoeForConditionalGeneration: 40 layers, 256 experts, 8 active)
Architecture	MoE decoder (hidden 2048, tied word embeddings), plus MTP and a vision tower (both BF16, text serve unused)
Context length	8192 tokens at serve time (--max-model-len 8192)
Quantization	W4A16 compressed-tensors, pack-quantized (int4 symmetric, group 128, no activation quantization)
License	Apache-2.0 (follows the base model)

Inference

Serve with vLLM (XPU fork on Intel GPUs; upstream CUDA vLLM also reads compressed-tensors):

pip install "vllm>=0.32"   # XPU fork (vllm-xpu) on Intel GPUs
vllm serve LeoHyperlink/Sage-1.5-W4A16 --trust-remote-code \
  --max-model-len 8192 --gpu-memory-utilization 0.9 --language-model-only

Transformers can load it for offline use (compressed-tensors + llmcompressor packages), but expect CPU-class throughput there; the fused path above is the vLLM one.

Sampling parameters (the validated settings — all numbers below used these):

• General (chat, structured output, evals): temperature=0, thinking disabled via --default-chat-template-kwargs '{"enable_thinking": false}'.
• Thinking must be off for valid string-match and structured-output evals: raw completions with thinking scored 9.9% GSM8K / 26.6% IFEval on this same checkpoint, vs 92.0% / 81.9% with thinking off and the chat template applied. That gap measures the harness, not the quant.

Performance

Speed (measured, not estimated)

Single-stream decode, 128 streamed tokens (temperature 0, prompt: "Explain why the sky is blue in one short paragraph"), OpenAI-compatible /v1/chat/completions streaming, tokens counted after time-to-first-token:

Checkpoint	Decode	TTFT	Backend
This repo (Sage W4A16-CT)	104.3 t/s	0.06 s	XPUwNa16LinearKernel (fused)
intel-ai/Qwen3.6-35B-A3B-MXFP4-Mixed-FP8Attn-CT-AutoRound (same box and protocol, reference)	60.0 t/s	0.08 s	XPUExpertsMxFp4 + XPUMxFp8LinearKernel

Correctness spot-checks on this checkpoint: 8/8 concurrent prompts coherent, plus an 80-request 4-way-concurrency soak with zero garbage outputs. A/B notes: FP8 KV cache + prefix caching did not move single-stream decode (102.4 vs 104.3 t/s, noise); the tuner-suggested VLLM_XPU_LM_HEAD_INT8=1 does not exist in the serving build below and was rejected.

Benchmarks

Three sound-harness proxy scores — every one measured on the same live serve as the speed numbers above (thinking off, temperature 0):

Benchmark	Sage-1.5 W4A16	Setup	Hub card
Math
GSM8K	92.0% flex / 91.5% strict	lm-eval, 1319 prompts, chat + template	.eval_results/gsm8k.yaml
Instruction following
IFEval	81.9% prompt-strict / 87.7% inst-strict	lm-eval, 541 prompts, chat + template, thinking off	README only (unregistered)
Structured output
IFStruct v1.0	72.8% pass (1456/2000)	upstream harness, 2000 prompts	.eval_results/ifstruct.yaml
Not run here
HLE	n/a	no HLE task in lm-eval 0.4.13	—

Evaluation details

All proxy scores were measured against the live vLLM-XPU 0.32.0 endpoint (single Arc Pro B70, thinking disabled, temperature 0). GSM8K: lm-eval 0.4.13 local-chat-completions, 1319 prompts, 8-way concurrency (loose scores: 86.0% prompt / 90.5% instance). IFEval: lm-eval, 541 prompts, chat completions + template, thinking off. IFStruct: upstream harness (github.com/Liquid4All/ifstruct) with defaults (temperature 0, max_tokens 2000) — JSON 73.3%, YAML 72.3%.

The base model card's 12 agentic benchmarks (SWE-Pro 61.2, SWE-Multilingual 72.7, Scicode 44.3, MLE-Lite 65.2, FS-Olympiad 81.0, HLE w/ tools 55.1, CritPt 20.0, CMT 80.0, ALE-CLI 19.1, PinchBench 90.0, BrowseComp 82.8, IFBench 81.6) need tool harnesses (terminals, browsers, multi-turn agents) this box does not run, so they cannot be re-run here. Do not place proxy numbers next to the base chart as recovery percentages — different benchmarks, different harnesses. String-match benchmarks on a thinking model measure the harness unless thinking is off and the chat template is applied: raw completions with thinking scored 9.9% GSM8K / 26.6% IFEval on this same checkpoint.

Recipe

Setting	Value
Tool	Intel auto-round 0.15.1, auto_round:llm_compressor (CT pack-quantized) export
Weights	int4 symmetric, group 128
Activations	none quantized (fp16 compute)
Calibration	pile-10k default (128 samples x 2048 tokens), auto-round default optimizer settings
torch.compile	explicitly disabled (Triton make_ttgir inductor failure on XPU)
Quantized	31030 / 31181 layers
Kept in BF16	lm_head, per-layer mlp.shared_expert_gate, visual tower
MTP	NOT quantized: 785 raw BF16 tensors passed through to model_extra_tensors.safetensors
Quant time	~8.7 h on one Intel Arc Pro B70

Tested hardware / software

• GPU: Intel Arc Pro B70 32 GB (Battlemage Xe2, 256 XMX engines, 608 GB/s), single GPU, ~90% VRAM in use at serve time
• Host: 64-core CPU, 251 GB RAM
• Serving stack: vLLM-XPU 0.32.0 fork (vllm-xpu-kernels with libgrouped_gemm_xe_2 and fused_moe_interface), torch 2.x +XPU, oneCCL
• Serve flags: --trust-remote-code --max-model-len 8192 --gpu-memory-utilization 0.9 --language-model-only --kv-cache-dtype fp8 --enable-prefix-caching (text-only; the visual tower ships BF16 and was not exercised), GPU pinned via ONEAPI_DEVICE_SELECTOR=level_zero:N. Note: ZE_AFFINITY_MASK=0.0 crashes engine init on this build (int parsing of "0.0"); this build binds localhost only.
• Eval harnesses: lm-eval 0.4.13 local-chat-completions against the live endpoint (8-way concurrency), IFStruct upstream harness against the same endpoint.

Files

• model-0000[1-4]-of-00004.safetensors (20 GB) + model.safetensors.index.json — quantized weights
• model_extra_tensors.safetensors (1.7 GB) — raw BF16 MTP/vision passthrough tensors, kept for faithfulness. vLLM 0.32.0 cannot load them alongside the CT main checkpoint; text-only serving does not need them.
• quantization_config.json — compressed-tensors pack-quantized spec (int4 symmetric g128, no activation quant)
• config.json, tokenizer files, preprocessor/processor configs, chat_template.jinja, generation_config.json
• .eval_results/gsm8k.yaml, .eval_results/ifstruct.yaml — machine-readable scores (Hub eval-results schema)

Limitations

• The base chart's agentic benchmarks were not re-run (see Benchmarks); proxy numbers above are not recovery claims.
• Multimodal (vision) path ships BF16 and was not benchmarked here.
• The 80-sample soak is a smoke test, not a proof: the same Qwen family has a known unquantized-lm_head sustained-concurrency corruption trap on some builds, so soak your own workload before production.
• MTP speculative decoding is NOT enabled by this checkpoint (MTP weights are raw; the loader requires packed CT).

Citation

No new training was done for this quant — if you use it, cite the base model (InternScience/Sage-A1.5) and link back to this repo for the quantization recipe and serving numbers.