Zhangdanyang
/

Kimi_K25_eagle3_v2_phase1_Lumen

speculative-decoding

Model card Files Files and versions

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Kimi-K2.5 Eagle3 Draft Model v2 (LumenRL, MI350)

Eagle3 MTP draft model for accelerating inference of Kimi-K2.5, trained with LumenRL on 8x AMD MI350 GPUs.

Architecture aligned with lightseekorg/kimi-k2.5-eagle3.

Training Setup

Hardware: 8x AMD MI350 (4 train + 4 inference)
Training: FSDP2 (float16), GPUs 0-3
Inference: vLLM (BF16), GPUs 4-7, TP=4
Transfer: Mooncake TCP
Draft: Eagle3, 1-layer Transformer (hidden_size=7168, intermediate_size=12288, num_heads=64, head_dim=128)

Training Phases

Phase 1 (Foundation): perfectblend subset, 111K steps (3 epochs)
Phase 2 (Mixed Domain): VL + Chinese + tool-call + agent + writing, 67,826 steps (3 epochs)

Final checkpoint at step 37000.

Architecture (matches reference)

attention_bias: false
rms_norm_eps: 1e-6
rope_theta: 1000000.0
rope_scaling: none
dtype: float16 (trained weights incl. lm_head), bfloat16 (embed_tokens from base model)

Downloads last month: 13

Safetensors

Model size

3B params

Tensor type

BF16

·

F16

·

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Zhangdanyang/Kimi_K25_eagle3_v2_phase1_Lumen

Base model

moonshotai/Kimi-K2.5

Finetuned

(43)

this model