Swift-1.5-FP8-AWQ-MTP

FP8 block-scaled quantization of ukisai/Swift-1.5-Qwen3.8-27b (27.8B, multimodal qwen3_5, 256K context, hybrid attention: 48 linear-attention + 16 full-attention layers) with three quality features on top of the base FP8_BLOCK format:

• AWQ smoothing pre-pass on BF16 weights before quantization (method of TelperionAI/Qwen3.8-27B-FP8-block-AWQ);
• calibrated FP8 KV-cache scales (per-layer k_scale/v_scale) — halves KV-cache memory (0.00% drift on a validated build of this recipe);
• quantized MTP head (parity layout) — the model is ready for MTP speculative decoding.

Size	~30 GB (vs 55.6 GB BF16)
Format	compressed-tensors FP8 block-scaled [128,128] weights (e4m3, RTN) + dynamic per-token×128 activations
Quality	statistically indistinguishable from BF16 (see Evaluation)
Tested runtime	vLLM ≥ 0.31.0 (compressed-tensors is auto-detected, no --quantization flag)

Quantization recipe

Component	Format
LM Linear weights	FP8 e4m3, block-scaled [128,128] (RTN, data-free)
LM input activations	FP8 e4m3, dynamic per-token×128 at inference
KV cache (16 full-attention layers)	FP8 e4m3, static per-layer k_scale/v_scale (calibrated, static_minmax observer, 256 sequences × 512 tokens)
MTP head: 8 Linear layers (self_attn.{q,k,v,o}_proj and mlp projections)	FP8 e4m3, block-scaled [128,128] (RTN)
MTP head: mtp.fc (5120→10240 input projection) + norms	BF16
Visual tower, embed_tokens, lm_head, all LayerNorms, q_norm/k_norm, mlp.gate, shared_expert_gate, linear-attention aux modules (A_log, conv1d, dt_bias, in_proj_a/b/ba, norm)	BF16

Quantized with llm-compressor 0.14.0 (oneshot, AWQModifier(duo_scaling="both") + QuantizationModifier(scheme="FP8_BLOCK"), calibration on perfectblend 256×512), with the MTP head post-quantized using the native compressed-tensors RTN code path. The smoothing is a function-preserving transform: 1/s is folded into the preceding norm and s into the input dimension of the next Linear, so inference incurs no extra compute (empty transform_config in the checkpoint).

Evaluation

Teacher-forced metrics vs the BF16 base: same 131,008-token corpus (wikitext-2-raw test, 64×2048), same vLLM engine and parallelism for both models, prompt_logprobs. Disagreements are bucketed by the base model's confidence margin (top1 − top2 logprob); confident/certain (margin ≥ 2) are the damage-relevant buckets.

Metric	BF16	This model	Δ
PPL (wikitext-2 style)	6.7745	6.7741	−0.0004
Top-1 token agreement	—	96.36%
Top-1 of quant in base's top-5	—	99.97%
Confident + certain disagreements	—	0.02% of positions
Coarsened KL (top-5, lower bound)	—	0.0055 nats/token

94% of the disagreement mass sits in the near-tie bucket (margin < 0.5): coin flips between nearly-equal tokens, semantically harmless. A BF16↔BF16 control run gives 0.00% on every metric, so the numbers above are signal, not engine noise.

gsm8k (full set, 1319 questions, 5-shot CoT, max_gen_toks=2048; accuracy by unified last-number extraction, exact McNemar test vs BF16 on the same questions):

Model	acc	McNemar p vs BF16
BF16 base	93.71%	—
This model (FP8 + AWQ)	94.47%	0.18 (not significant)
Same recipe without AWQ smoothing	94.16%	0.46 (not significant)
External FP8_BLOCK build without smoothing	92.34%	0.013 (significantly worse)

Capabilities are preserved: with the generation limit saturated (≤1 of 1319 questions hits the 2048-token cap), the quant is statistically indistinguishable from the BF16 base. The AWQ pass is what keeps the distribution closest to base — without it the same recipe drifts (PPL Δ+0.028 vs Δ−0.0004 here), and the external no-smoothing build is significantly worse on gsm8k as well.

FP8 KV cache: on a validated build of this recipe, serving with --kv-cache-dtype fp8 against the auto (BF16) KV cache measured 0.00% drift on all teacher-forced metrics (PPL, top-1, all buckets) — the calibrated scales are lossless in practice and halve the KV-cache footprint (32 vs 64 KiB/token).

MTP head: the head is not invoked in ordinary forward passes, so quantizing it does not change any generation metric (verified on a build of this recipe: teacher-forced metrics identical before/after MTP post-quantization). It matters for MTP speculative decoding, where the parity layout (8 FP8 Linear layers, mtp.fc + norms in BF16) matches the reference build.

Usage

vLLM (recommended)

compressed-tensors checkpoints are detected automatically — do not pass --quantization. --kv-cache-dtype fp8 is required to apply the calibrated KV scales (with the auto dtype they are not used and the KV cache stays BF16).

Production configuration (4x 16 GB GPUs)

Full-featured serving on 4x RTX 5070 Ti (16 GB, PCIe, no NVLink): TP4 fits the budget (~7.5 GB weights + ~2 GB of 256K FP8 KV per card), and tensor parallelism is required here — vLLM 0.31.0 rejects pipeline parallelism combined with --async-scheduling and structured output (tool parsers). The validated no-tools configuration on this hardware uses PP4.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True    # vLLM-recommended allocator for 16 GB cards
export NCCL_P2P_DISABLE=1                                  # no NVLink: NCCL over shared memory (PCIe P2P is flaky)
export VLLM_USE_DEEP_GEMM=0                                # DeepGEMM targets Hopper (SM90); not needed on SM120
export VLLM_USE_FASTOKENS=1                                # fast detokenization backend
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0          # do not reserve an estimated CUDA-graphs budget in the profiler

vllm serve MegaPSV/Swift-1.5-FP8-AWQ-MTP \
  --kv-cache-memory 5000000000 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --attention-backend FLASHINFER \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --max-num-batched-tokens 10240 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-num-seqs 4 \
  --no-use-tqdm-on-load \
  --enable-chunked-prefill \
  --async-scheduling \
  --disable-custom-all-reduce \
  --default-chat-template-kwargs '{"preserve_thinking":true,"enable_thinking":true,"reasoning_effort":"xhigh"}' \
  --trust-remote-code \
  --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
  --port 8000

• --kv-cache-memory 5000000000 — 5 GB per GPU (20 GB total with TP4); this flag takes precedence over --gpu-memory-utilization. Comfortably covers 256K with fp8 KV (≈8 GiB total).
• --attention-backend FLASHINFER — supported on SM120 (RTX 50 series) in vLLM ≥ 0.31; drop the flag to fall back to vLLM's default backend.
• --disable-custom-all-reduce — for PCIe interconnect (no NVLink).
• The export lines are for this consumer PCIe box (no NVLink); on datacenter hardware with NVLink most of them can be dropped.
• MTP speculative decoding uses the quantized head; start with num_speculative_tokens: 3 and fall back to 1 if acceptance is poor.
• If the engine fails to allocate memory at startup, lower --max-num-batched-tokens (e.g. 2048), --max-num-seqs, or --kv-cache-memory.

from vllm import LLM, SamplingParams

llm = LLM(
    model="MegaPSV/Swift-1.5-FP8-AWQ-MTP",
    kv_cache_dtype="fp8",
    max_model_len=262144,
)
output = llm.generate("What is 84 * 3 / 2?", SamplingParams(temperature=0.0))
print(output[0].texts[0])

Memory footprint

• Weights: ~30 GB.
• Verified configuration: 4× 16 GB GPUs (no NVLink) with tensor parallelism (--tensor-parallel-size 4); the full 256K context fits with the FP8 KV cache. The agentic configuration (tool calling + --async-scheduling + MTP spec-decode) runs on the same hardware with TP4 — see Production configuration.

Limitations

• Evaluation covered teacher-forced distribution metrics and gsm8k (math reasoning). A VLM (image) smoke test was not part of the validation suite; the visual tower is BF16, but AWQ smoothing touches the vision→LM input projections.
• AWQ calibration is inherently non-deterministic (CUDA reduction noise in the activation statistics feeds the smoothing grid-search): re-running the recipe does not reproduce this exact checkpoint. This artifact is the validated build — use it as-is rather than re-quantizing.
• Quality claims are relative to the specific BF16 base ukisai/Swift-1.5-Qwen3.8-27b; they do not transfer to other Qwen3.8 variants.

Acknowledgments

• Base model: ukisai/Swift-1.5-Qwen3.8-27b (UkisAI, Swift Open License v1.0)
• Quantization format reference: Qwen/Qwen3.8-27B-FP8
• AWQ+FP8-block method: TelperionAI/Qwen3.8-27B-FP8-block-AWQ
• MTP parity layout: realderpz/Swift-1.5-Qwen3.8-27B-FP8
• Quantization tooling: llm-compressor / compressed-tensors, serving with vLLM