RedHatAI/Mistral-Small-4-119B-2603-speculator.dspark

This is a DSpark speculator model for mistralai/Mistral-Small-4-119B-2603.

Training Details

This model was warm-started from RedHatAI/Mistral-Small-4-119B-2603-speculator.dflash and trained using the Speculators library on inference-optimization/Mistral-Small-4-119B-Regenerated. The shared backbone (decoder layers, fc, norms, embeddings, lm_head) was copied from the DFlash checkpoint; the DSpark-specific heads (MarkovHead, ConfidenceHead) were randomly initialized and trained from scratch. DSpark extends DFlash with a Markov head (intra-block token dependency) and a confidence head (per-position acceptance prediction).

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model mistralai/Mistral-Small-4-119B-2603 \
  --data hf:inference-optimization/Mistral-Small-4-119B-Regenerated \
  --output ./output/dspark_mistral_small_119b_warmstart \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  mistralai/Mistral-Small-4-119B-2603 \
  --target-layer-ids 2 18 33 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 16386 \
  --enforce-eager

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path mistralai/Mistral-Small-4-119B-2603 \
  --speculator-type dspark \
  --from-pretrained shanjiaz/dflash-mistral-small-119b \
  --data-path ./output/dspark_mistral_small_119b_warmstart \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dspark_mistral_small_119b_warmstart/checkpoints \
  --epochs 3 \
  --lr 1e-4 \
  --backbone-lr-scale 0.1 \
  --total-seq-len 16384 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --markov-head-type vanilla \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --confidence-head-alpha 1.0 \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --fsdp-shard \
  --no-sample-from-anchor \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.1

Model Specifications

Base Model	mistralai/Mistral-Small-4-119B-2603
Chat Template	mistralai/Mistral-Small-4-119B-2603 (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia B200

Deployment

# Deploy with speculative decoding
vllm serve mistralai/Mistral-Small-4-119B-2603 \
    --tensor-parallel-size 4 \
    --speculative-config '{
        "model": "RedHatAI/Mistral-Small-4-119B-2603-speculator.dspark",
        "num_speculative_tokens": 7,
        "method": "dspark"
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	79.3%	61.4%	47.3%	36.2%	27.8%	21.2%	15.6%	3.89
math_reasoning	89.5%	77.7%	66.8%	58.0%	49.5%	40.9%	33.2%	5.16
qa	59.0%	34.9%	20.0%	11.6%	6.4%	3.6%	2.1%	2.37
question	66.1%	43.5%	29.2%	20.2%	14.1%	9.6%	6.3%	2.89
rag	64.6%	40.7%	25.0%	15.6%	9.4%	5.7%	3.4%	2.64
summarization	62.6%	36.2%	20.8%	11.8%	6.1%	3.2%	1.3%	2.42
tool_call	64.3%	41.6%	27.5%	18.2%	12.4%	8.3%	5.3%	2.77
translation	73.1%	43.8%	22.6%	11.5%	6.2%	2.9%	1.6%	2.62
writing	65.9%	43.3%	29.3%	20.4%	14.8%	10.3%	7.2%	2.91