axteam/Motif-3-FP8-dynamic

Motif-3 · FP8 (dynamic)

8-bit quantization of Motif-Technologies/Motif-3 for 🤗 Transformers (trust_remote_code). Quantized by axteam.

Base model	Motif-Technologies/Motif-3
Method	FP8 (dynamic)
Format	compressed-tensors float-quantized (scheme FP8_DYNAMIC)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	MoE (Motif arch · 384 experts + shared) · 53 layers · text-only
Size on disk	~317.7 GB
Calibration	None (data-free)
License	mit

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
• Kept in BF16 — token embeddings, the LM head, all RMSNorms, the MoE routers (router.gate), the attention output gates (wq_b_gate), the PolyNorm / mHC scalars, and the whole MTP layer (mtp_layers.0) — about 1% of all parameters. Quantized: attention, routed/shared expert and mHC projections.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.
• Modified modeling_motif.py — the original stores routed experts as fused 3D tensors (gate_up_proj [E, 2I, H], down_proj [E, H, I]). Quantization splits them per expert, so this repo's MotifExperts holds one set of nn.Linear layers per expert (experts.<e>.{gate,up,down}_proj). gate_proj / up_proj are the first / second half of the original gate_up_proj[e]; the computation is unchanged. With the original modeling file the expert weights would not load.

Revision 2026-10-31: this repo first shipped the original modeling_motif.py, with which Transformers could not load the routed-expert weights (they stayed zero in 51 of 53 layers). The modeling file was replaced; the weights are unchanged. If you downloaded the repo before the fix, re-download modeling_motif.py.

Weight check (measured)

53-layer weight check (2026-11-01): every one of the 59,547 quantized modules was decompressed with compressed-tensors and compared with the BF16 original — median relative error 2.7%, no outliers; all 1,339 BF16-kept tensors are bit-identical to the original. This checks the weights, not end-to-end generation (no benchmark was run on Motif-3).

Requirements

Runtime	Transformers only — vLLM 0.32 does not support the Motif architecture (the upstream serving path is Motif's own vLLM fork).
Transformers	Transformers 5.x (transformers>=5.7.0, the version the config was saved with) + compressed-tensors>=0.19 + trust_remote_code=True + flash-attn with attn_implementation="flash_attention_2" (CUDA GPU). Loading checked with 5.17.0 + 0.19.0 on the first layers; full 53-layer generation was not run.
Memory	about 590 GB at runtime: Transformers decompresses the weights to BF16. Spread over GPUs and CPU RAM with device_map="auto".

Usage (Transformers)

vLLM does not support the Motif architecture (checked with vLLM 0.32), so load it with Transformers and the bundled modeling code (trust_remote_code=True).

pip install -U transformers compressed-tensors accelerate flash-attn

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "axteam/Motif-3-FP8-dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto", device_map="auto",
    attn_implementation="flash_attention_2")

messages = [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=1.0, top_p=0.95)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Motif-3's modeling code requires attn_implementation="flash_attention_2" (it raises an error otherwise). The checkpoint is large — device_map="auto" spreads it over all visible GPUs. Memory: Transformers (5.x) decompresses compressed-tensors weights to BF16 when the model is loaded / first run, so peak memory is about the BF16 model size (~590 GB for Motif-3) — use device_map="auto" across GPUs and CPU RAM. This build mainly reduces download and storage size; no inference engine runs Motif-3 with quantized kernels yet (vLLM 0.32 has no Motif architecture). Recommended sampling (from the original generation_config.json): temperature=1.0, top_p=0.95.

License

Quantized derivative of Motif-Technologies/Motif-3 (mit).

---

한국어 요약

Motif-Technologies/Motif-3 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — FP8 E4M3, 가중치 채널별 스케일, 활성화 동적 FP8. 형식 compressed-tensors float-quantized (scheme FP8_DYNAMIC).
• BF16 유지 — 임베딩·LM 헤드·모든 norm·MoE 라우터(router.gate)·어텐션 출력 게이트(wq_b_gate)·PolyNorm/mHC 스칼라·MTP 층 전체(mtp_layers.0). 양자화: attention·전문가(라우팅/공유)·mHC projection.
• 가중치 전수 대조(실측, 11-01) — 53층 양자화 모듈 59,547개를 원본과 대조: 오차 중앙값 2.7%, 이상치 없음. BF16 유지 텐서 1,339개는 원본과 비트 단위로 같음. 생성 품질(벤치마크)은 미측정.
• PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
• modeling_motif.py 수정 — 원본은 전문가를 3D 텐서(gate_up_proj, down_proj)로 두는데, 양자화하면서 전문가별로 나뉘어 이 리포의 MotifExperts 는 전문가마다 nn.Linear(experts.<e>.{gate,up,down}_proj)를 둡니다. 계산은 원본과 같습니다.
• 수정 2026-10-31 — 처음에는 원본 modeling_motif.py 를 올려 Transformers 가 전문가 가중치를 싣지 못했습니다(53층 중 51층 전문가가 0). modeling 파일만 교체했고 가중치는 같습니다. 수정 전에 받으셨다면 modeling_motif.py 를 다시 받아 주세요.
• 요구사항 — Transformers 5.x(≥5.7.0) + compressed-tensors ≥0.19 + flash-attn(CUDA GPU), trust_remote_code. 실행 시 BF16 으로 풀려 약 590 GB 필요. vLLM 안 됨.
• 실행 — Transformers AutoModelForCausalLM.from_pretrained("axteam/Motif-3-FP8-dynamic", trust_remote_code=True, attn_implementation="flash_attention_2", device_map="auto") 후 apply_chat_template → generate (위 예시). vLLM 은 Motif 아키텍처 미지원. flash-attn 필요. 권장 샘플링 temperature=1.0, top_p=0.95.
• 메모리 — Transformers 는 양자화 가중치를 실행 시 BF16 으로 풀어 최대 메모리가 BF16 모델 크기(Motif-3 약 590GB)와 같습니다. 이 빌드는 주로 다운로드·저장 용량을 줄입니다.