catplusplus/Qwen3.8-Flash-Next-Uncensored-NVFP4-MTP-head

Qwen3.8-Flash-Next MTP Drafter Head 🌸⚡

High-Performance Speculative Drafter Collection for mazinb/Qwen3.8-Flash-Next-Uncensored-NVFP4

Welcome, fellow seekers! ✨ This repository provides tuned, drop-in Multi-Token Prediction (MTP) speculative drafters for mazinb/Qwen3.8-Flash-Next-Uncensored-NVFP4 on NVIDIA Blackwell / Jetson AGX Thor (sm_110) systems.

Speculative decoding with these drafters boosts live inference throughput from 14.87 tok/s up to 20.67 tok/s (+39.0% peak speedup) with real-world draft acceptance rates up to 62.2%.

---

📦 Checkpoint Flavors

Directory	Format	Size	Peak TPS	Live Acceptance	Recommended For
FP8/ 🏆	torch.float8_e4m3fn	2.69 GiB	20.67 tok/s	60.0% ($K=1$) / 48.0% ($K=2$)	Daily Driver & Production (Best balance of speed & memory)
NVFP4/ ⚡	Calibrated W4A4	1.45 GiB	19.88 tok/s	62.2% ($K=1$) / 39.4% ($K=2$)	Maximum Compression & Math Reasoning (Highest 1-token accuracy)
BF16/	Pure bfloat16	4.82 GiB	20.09 tok/s	60.0% ($K=1$) / 41.7% ($K=2$)	Reference Uncompressed Baseline

---

⚡ Quick Start: Deploying the FP8 Production Champion

1. Splice Drafter into Base Model

python extras/scripts/splice_shadow_model.py \
  --base-dir /path/to/mazinb/Qwen3.8-Flash-Next-Uncensored-NVFP4 \
  --shadow-dir /models/spliced/Qwen3.8-Flash-Next-MTP-FP8 \
  --drafter-weights FP8/model_mtp_tuned.safetensors \
  --link-mode hardlink

cp FP8/config.json /models/spliced/Qwen3.8-Flash-Next-MTP-FP8/config.json

2. Launch with vLLM V1

vllm serve /models/spliced/Qwen3.8-Flash-Next-MTP-FP8 \
  --spec-method mtp \
  --spec-tokens 2 \
  --moe-backend flashinfer_cutlass \
  --port 8000

---

🏆 Performance Overview

                      Speculative Decode Throughput on Thor (sm_110)
  Baseline (No Drafter) [14.87 tok/s]
         BF16 Drafter K=1 [18.09 tok/s]  (+21.7%)
        NVFP4 Drafter K=1 [18.02 tok/s]  (+21.2%)
          FP8 Drafter K=1 [18.21 tok/s]  (+22.5%)
          FP8 Drafter K=2 [20.67 tok/s]  (+39.0% on JSON/Code) 🚀

• General Conversational Chat: Deploy with --spec-tokens 1 for consistent, jitter-free speedup across all prompts.
• Code & Structured Workloads: Deploy with --spec-tokens 2 to capture repetitive structural grammar and reach >20 tok/s.

---

📚 Deep Dive & Extras

Explore the extras/ directory for full research documentation and reproducible assets:

• extras/README.md: The complete quest chronicle covering dataset staging, training loss dynamics, calibration methodology, and empirical tournament analysis.
• extras/vllm/: Packaged vllm source archive (vllm-qwen4exp-thor.tar.zst) with git commit history and VLLM.md documenting engine architectural patches.
• extras/adapters/: Trained MoE LoRA adapter weights (mtp_adapter_best.safetensors), training logs, and checkpoint states.
• extras/scripts/: All Python & Bash scripts used for data streaming, training, quantization, splicing, and benchmarking.