Eggchad/mango-artifacts

MANGO1.5 Thai-tuned MTP head (mtpft6)

This repo holds a fine-tuned MTP (multi-token prediction) draft head for CMKL/MANGO1.5-Qwen3.5-9B, plus GGUF builds that include it. MANGO's shipped MTP head was trained before the Thai fine-tune, so it guesses Thai text poorly: only 18–34 % of its draft tokens are accepted. We retrained only the mtp.* block (~0.25 B parameters) on MANGO's own answers, with the trunk, embeddings and lm_head frozen. Thai acceptance roughly doubles, and English stays the same.

Speculative decoding verifies every draft token with the full model, so a better head changes speed only. The output is the same as without speculation, up to floating-point near-ties.

By team ThunderPermanent (GreenMind LLM Inference Optimization Hackathon, CMKL, Oct 2026): Eakkachat Sathaphon, Paradee Pimoleakaksorn, Pongsura Pongpisan, Thanakrit Samoena.

Results

vLLM 0.32, A100-SXM4-40GB, AWQ W4A16 trunk + MTP (results/A100_VLLM_MTPFT_2026-10-22.md):

head	k	Thai accept	Thai tok/s (c=1)	English accept	English tok/s (c=1)
original CMKL	1	0.339	145.0	0.818	191.6
ours (mtpft6)	1	0.749	186.6 (+29 %)	0.819	192.2
original CMKL	2	0.214	126.8	0.693	204.0
ours (mtpft6)	2	0.631	196.0 (+55 %)	0.708	206.1

At 16 concurrent requests, Thai throughput rises from 1035 to 1537 tok/s (k=2) and tok/J from 4.03 to 6.31.

Offline chained accuracy on held-out prompts (step 1/2/3, results/K8_FINDINGS.md, results/KF_FINDINGS.md):

	Thai	English
original head	39.7 / 28.0 / 22.0	80.3 / 67.2 / 59.3
mtpft6	86.6 / 82.5 / 79.5	81.4 / 70.6 / 64.4

Files

path	what it is
vllm/mango-awq-mtpft6-mtp-only/mtp_ft.safetensors	the head alone, HF tensor names (mtp.*), bf16, 0.49 GB
vllm/mango-awq-mtpft6-mtp-only/build_awq_mtpft.py	extracts the head from GGUF and merges it into a vLLM checkpoint
models/mango-mtpft6-bf16-dh32k.gguf	BF16 trunk + our MTP + 32k draft head (18.7 GB)
models/mango-mtpft6-q8_0-dh32k.gguf	Q8_0 (9.9 GB)
models/mango-mtpft6-iq4xs-dh32k.gguf	IQ4_XS (5.4 GB)
models/mango-mtpft6i-q4km-dh32k.gguf	imatrix Q4_K_M (5.9 GB)
training/train_mtp.py	PyTorch MTP re-implementation and trainer (vLLM/HF semantics)
training/dump_h.cpp, training/pipeline.sh	dump trunk hidden states, then the data/eval pipeline
training/topk_eval.py, training/pmin_sweep.sh	evaluation helpers
patches/llama.cpp-mango/	llama.cpp patches: MTP drafting, trimmed draft head, rollback, device argmax
tools/gguf_add_draft_head.py	builds the trimmed 32k-vocab draft head (dh32k)
data/imatrix/	imatrix calibration text and imatrix.dat for the Q4_K_M build
data/mango-draft/ngram_{th,en}.mdt	n-gram drafting tables used by our engine
data/mango-k8/	held-out prompt lists and indices used for evaluation (see Data)
results/	findings reports and raw vLLM benchmark JSON

dh32k is a draft-only lm_head trimmed to the 32,768 most frequent tokens. It makes each draft step cheaper. Because the full head still verifies every token, the trimmed head is lossless.

Usage

vLLM (any MANGO1.5 checkpoint that ships mtp.* tensors in bf16, for example the CMKL AWQ -mtp repo):

python build_awq_mtpft.py merge --base <checkpoint_dir> --mtp mtp_ft.safetensors --out <out_dir>
vllm serve <out_dir> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Run python build_awq_mtpft.py -h for the exact arguments. k=2 was best in our A100 tests.

llama.cpp: the GGUFs keep the MTP block as blk.32.nextn.* and add the dh32k draft head. Stock llama.cpp does not use them for drafting. They need our patches in patches/llama.cpp-mango/ (apply them to llama.cpp with git am).

Training recipe

1. Prompts. WangchanThaiInstruct (Thai, human-written) and alpaca-cleaned (English). They are deduplicated by text against every test and held-out set. Round 1 used 3k Thai and 1k English, round 2 used 10k, and round 3 added 12k Thai and 8k English, for about 30k sequences in total.
2. Answers. MANGO generated them itself (greedy, thinking off, ≤ 256 tokens). This is self-distillation, so the head learns MANGO's own output distribution.
3. Hidden states. dump_h saves the trunk hidden state that the MTP head receives when it drafts.
4. Training. train_mtp.py trains mtp.* only. It uses AdamW at lr 1e-5 for 1 epoch, a 3-step recursive loss (FastMTP weights 0.51/0.31/0.18), and English oversampled ×2–3. mtpft6 continues from mtpft5 for 40k steps on one consumer GPU (RTX 4070).
5. Check. With the original weights, the PyTorch re-implementation matches live llama.cpp acceptance (Thai 39.7 vs 39.5 %, English 80.3 vs 81.2 %).

A negative result: adding a JEPA-style latent-alignment loss had no effect.

Data

data/mango-k8/ contains only prompt lists and indices taken from the public datasets above. It has no competition or grader content. Check the upstream dataset licenses before you redistribute it: WangchanThaiInstruct is CC BY-SA 4.0 and alpaca-cleaned is CC BY 4.0.

Relation to the hackathon submission

The submitted INT8 model, Eggchad/MANGO1.5-Qwen3.5-9B-W8A8-INT8, uses CMKL's original MTP head, unchanged. This head was not part of the graded server. Combining it with the INT8 build is future work.

License

The weights are derived from CMKL/MANGO1.5-Qwen3.5-9B and Qwen3.5-9B and are released under Apache-2.0. Code in this repo is Apache-2.0.