RadixArk/Kimi-K3-DSpark

Kimi K3 DSpark speculator

Overview

A DSpark speculator for the Kimi K3 target, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. This checkpoint was trained with SpecForge using hidden states from a live SGLang target engine.

Model Specifications

• Base model: moonshotai/Kimi-K3.
• Format: Safetensors (single-file BF16, 2,249,289,601 parameters).
• Draft: 5 layers (Qwen3-style GQA), hidden 7168, 64 heads / 16 KV heads, head_dim 64, FFN 14336, rope_theta 10000, block_size=7.
• Vocabulary: 163,840 rows; mask_token_id=163824, bos=163584, eos=163586, pad=163839.
• DSpark heads: Markov rank 256 (vanilla) and confidence head (with-Markov).
• Aux hidden-state layers: [7, 23, 51, 67, 83] (full-attention layers of the 93-layer target).
• Trained context: sequence length 4096.
• Target weights: target embedding and unembedding weights are not included in this checkpoint.

Evaluation Results

Dataset	Accept length
GSM8K	5.6660
MATH500	4.0803
HumanEval	5.3613
MBPP	4.9220
MT-Bench	3.3425
AIME26	2.6326

Serving with SGLang

sglang serve \
  --trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 8 \
  --dcp-size 8 \
  --mem-fraction-static 0.85 \
  --max-mamba-cache-size 160 \
  --max-running-requests 32 \
  --cuda-graph-max-bs-decode 32 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
  --speculative-dspark-block-size 7

Training Details

• Framework: SpecForge online distillation, with hidden states captured from a frozen Kimi K3 target served by a live SGLang engine. Draft trained from random initialization.
• Data: 700K regenerated OpenPerfectBlend samples. thinking_effort=max, right-truncated at 4096.
• Schedule: 10 epochs, AdamW (fp32 master weights), peak learning rate 6e-4, cosine decay to 0, 4% warmup, and gradient clipping at 1.0. Global batch 512, seed 42.
• Loss: 0.1 CE + 0.9 L1 distillation + 1.0 confidence BCE, decay gamma 4.0, with 512 sampled anchors per sequence and block_size=7.
• Topology: 4 nodes × 4 GB300 (16 ranks) — 2 × TP8 target replicas, DP2 sampler, FSDP16 SHARD_GRAD_OP on the draft, TP-batch scatter. Batch 8 per replica × 32 accumulation steps × 2 replicas = global batch 512.

Known Limitations

• This DSpark checkpoint was trained with a maximum context length of 4,096 tokens, which may lead to reduced acceptance lengths in extreme long-context and agentic use cases. Training for these scenarios is ongoing.