Muse-Spark-Open-DFlash2

Overview

A DFlash2 speculator for Muse Spark Open, designed to accelerate inference through speculative decoding with SGLang. The draft proposes multiple tokens in parallel, which the target model verifies.

This repository contains the BF16 draft weights and configuration. The target model supplies the embeddings, LM head, tokenizer, and native chat template. The draft is used alongside the target and is not a standalone language model.

Model Specifications

• Base model: meta-models/Muse-Spark-open-816B-A42B-NVFP4
• Format: Single-file BF16 Safetensors
• Draft: 5 Qwen3-style GQA layers, hidden size 8192, 64 query heads / 8 KV heads
• Attention: Sliding-window attention with a 2,048-token window
• Verification width: 8 tokens (1 current token + 7 draft tokens)
• Selector: Rank 256, top-k 16
• Auxiliary target layers: [1, 16, 30, 44, 59]
• Configured context length: 1,048,576 tokens, aligned with the target
• Target weights: Embedding and unembedding weights are loaded from the target

The configured context limit is distinct from the attention window. The config alignment does not establish draft acceptance quality across the full 1M context.

Evaluation Results

acc_len is the per-request acceptance length L = C / V, where C is the number of completion tokens and V is the number of verification steps, averaged equally across successful requests. Completion-token-weighted AL is sum(C * L) / sum(C); pooled AL is sum(C) / sum(V).

Dataset	Samples	acc_len	Completion-token-weighted AL	Pooled AL
GSM8K	1,319	5.370116	5.133438	5.067487
AIME26	240 (30 × 8)	4.194966	3.942215	3.891917
GPQA Diamond	198	3.879590	3.614902	3.587121
MindCube	1,050	3.253494	3.213286	3.206133
Video-MME short	900	3.743335	3.393189	3.340400

Measured with real DFlash8 and block verification enabled, checkpoint-default sampling (temperature=1.0, top_p=1.0, top_k=64), and xhigh reasoning. The run used two independent ARM64 GB300 replicas, each with TP4/EP4, FP8 target KV cache, and BF16 draft KV cache. Suites ran concurrently with 16–64 workers per suite; these results are not from a single-request-at-a-time evaluation.

All 3,707 samples completed without truncation. Acceptance statistics include two successful SDK retries; the two initial failed HTTP attempts had no model counters and are excluded from AL. Video-MME short used 96 frames per sample.

Serving with SGLang

The following example launches one TP4/EP4 replica with block verification:

python -m sglang.launch_server \
  --model-path meta-models/Muse-Spark-open-816B-A42B-NVFP4 \
  --quantization modelopt_fp4 \
  --moe-runner-backend flashinfer_trtllm \
  --tp-size 4 \
  --ep-size 4 \
  --attention-backend trtllm_mha \
  --page-size 128 \
  --mem-fraction-static 0.8 \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --enable-layernorm-sp \
  --enable-megamoe-for-layernorm-sp \
  --kv-cache-dtype fp8_e4m3 \
  --enable-tf32-matmul \
  --sampling-defaults model \
  --max-running-requests 128 \
  --context-length 1048576 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
  --speculative-draft-attention-backend fa4 \
  --speculative-num-draft-tokens 8 \
  --speculative-use-block-verification \
  --host 127.0.0.1 \
  --port 30000

Use the target's native chat template and xhigh reasoning for the evaluation settings above.