Motif-Technologies

/

Motif-3-Beta

like 175

Follow

Motif Technologies 339

Update — independent corroboration & ablation-based localization

Since posting, I found the community FP8 quantization (0ppxnhximxr/Motif-3-Beta-FP8) independently reached the same conclusion, stated in its README warning: the base checkpoint "produces incoherent output in every configuration we tested (teacher-forced NLL ~20 vs. 2-4 expected), before any quantization is applied", and "the published modeling code and weights most likely belong to different architecture revisions." This matches my measurements (NLL ≈ 22 nats; confidently-wrong output from position 0) from two further independent implementations.

추가로 제가 그 사이 배제 완료한 항목(모두 실측):

• mHC: arXiv 2512.24880 수식과 조문 대조 — 2σ post-gate 포함 구현이 논문과 정합 (용의 제외)
• 텐서 레이아웃 전체: 가중치 통계 포렌식(gate/up의 CV·왜도 시그니처, wkv_b k/v의 q_nope 부분공간 정렬, rope 주기-32 쌍 상관, group-major noise 슬롯 아웃라이어)이 배포 코드의 view/split 가정과 일치 — 익스포트 손상이 아니라 의미론 리비전 차이라는 미러 제작자의 결론을 지지
• 임베딩 입력 스케일(×√D 류): 없음 확인 + 기능 테스트 음성

컴포넌트 절제(ablation) 매트릭스 — 리비전 델타의 위치 특정에 도움이 되길 바랍니다 (풀 모델, 20개 절제, held-out NLL 768토큰 EN/KO, 음수=개선):

즉 routed 경로에는 실제 신호가 있고, shared expert의 무가중 합산(y + shared(x))과 라우팅 가중의 normalize-then-×2.0 관례가 학습 리비전과 다른 것으로 보입니다 (diff-attn 결합·mHC 게이트는 3군 기여). 가장 빠른 해결은 이 가중치와 정합하는 modeling 코드 리비전 공개(또는 재익스포트) 입니다. 재현 스크립트는 준비돼 있으니 필요하시면 말씀해 주세요.

I've also been working on this with AI to figure out how to run inference on this and hit the same roadblock. Here's where we got to (AI summary since it's doing the heavy lifting):

Follow-up: fault isolated to the shared-expert branch (evidence from systematic ablations)

We ran systematic ablations on an independent MLX port (which matches the patched reference implementation). Key results, using the depth-wise residual RMS and position-0 logits as signals:

• Zeroing only the shared expert at MoE layers: final-layer RMS drops 15.9 → 3.7, the position-0 junk token (id 26857) disappears, and generation recovers real words with loose grammar. This is the strongest single-change rescue we found.
• Zeroing the routed experts instead makes output worse (junk-token confidence rises 0.24 → 0.83) — the routed experts appear to partially compensate for the shared branch.
• Not the fix (all tested): shared-output scaling ×0.5/×0.25/×0.125 (monotonic RMS improvement but still incoherent), gate/up swap, SiLU replacement, alternate input points (unnormed read, mean-of-streams, attention input), alternate write paths (uniform/single-stream/÷4), and ±1 layer weight shift. Export forensics show no duplicated or shifted shared tensors.

This suggests the shipped modeling_motif.py composes the shared expert differently from training. Could the team share:

- The exact training-time computation of the shared expert (input point, normalization, any scaling such as polynorm_output_scale, and how its output enters the mHC streams)?
- The delta of "Modified mHC" vs. the mHC paper (arXiv 2512.24880)?

Even a short code snippet of the shared-expert + mHC composition from the training stack, or one known-good input→logits fixture, would let the community fix inference immediately.

Thank you!

Hi @avlp12 ,

Thank you for this — it's an exceptionally thorough report, and we really appreciate the time and rigor behind it.

You're right about the issues in the shipped modeling code, and we're sorry for the trouble it caused you. The team is actively working on a fix, including modeling code that is consistent with the released checkpoint. We'll post an update here as soon as we have something concrete, and we'll come back to your specific questions as well.

Thanks again for caring enough about the model to dig this deep.

@dongseokmotif Thank you for your reply. I am looking forward to the amazing model!

Hi @avlp12 , @ahames

Sorry for the delay in addressing the issue.

We're currently working on the bug you reported, but it may take some time to fix.

In the meantime, we've made our vLLM Docker image public so you can use the model without relying on the current modeling_motif.py implementation.
Please refer to the README for instructions on how to run the model with vLLM.

Thank you for your patience, and we'll update you as soon as the fix is ready.

Hi @avlp12 , @ahames — update from the Motif team.

We've pushed fixes to the shipped modeling code (modeling_motif.py + configuration_motif.py) for the bugs you reported — notably the RoPE inv_freq init (meta-device → identity RoPE, the position-0 collapse) and the per-expert PolyNorm dispatch (BUG 3). HF .generate now runs and produces coherent output. For production/high-throughput serving, vLLM stays recommended.

Sorry for the trouble, and thank you for the exceptional, rigorous reports — they made this much faster to pin down. 🙏

A note on the mHC post-gate: our implementation uses a sigmoid gate (coefficient 1.0), not the 2·sigmoid in the paper (arXiv 2512.24880).
This is intentional — a deliberate design choice, not an export/versioning artifact. We'll document the reasoning behind it in our upcoming tech report.

Confirmed: the fix works, and it resolves the MLX port too — thank you @dongseokmotif @iamwyldecat ! 🎉

I pulled the updated modeling_motif.py (SHA d2c9ac6) and diffed it. The decisive change for the incoherence was the PolyNorm activation, not RoPE: the shipped GroupedPolyNorm/PolyNormTorch now (a) pass the 3 coefficients through sigmoid(weight) (sigmoid_weight=True), (b) apply output_scale=0.5 (polynorm_output_scale), and (c) clamp the routed bias to ±0.5 (polynorm_bias_clamp) — exactly the config keys that were present-but-unread before. Since PolyNorm runs on every token at every layer and the experts are 97.84% of params, raw-vs-sigmoid coefficients is a position-independent, dominant divergence (which is why I saw garbage even at position 0, and why my earlier ablation only dampened rather than fixed it).

I applied the same three changes to my independent MLX port and, on the existing checkpoint (no re-export needed on my side — it's a runtime-activation change), held-out NLL dropped from ~23 → ~2.1 nats, and generation is now fluent in both languages:

• "The capital of France is" → " Paris. True or False? ..."
• "대한민국의 수도는" → " 서울입니다. 서울은 대한민국의 수도로서 정치, 경제, 사회, 문화 …"
• "Once upon a time" → ", there was a young girl named Lily who lived in a small village. …"

The per-layer RoPE (YaRN on full-attention layers, plain swa_rope_theta on SWA layers) is a smaller refinement on top (≈neutral at short context, minor KO gain), and I've matched that too. This unblocks my MLX quantization work — I'll follow up with the quantized builds. Really appreciate the fast, precise fix and the vLLM image in the meantime. 🙏

Thanks for the mHC clarification, @dongseokmotif — that's really helpful. I see it in the fixed code: mhc_h_post_coeff = 1.0 + mhc_h_post_alpha_end, which is 1.0 for Motif‑3 (alpha_end absent → 0), so the post‑gate is a plain sigmoid, not the paper's 2·sigmoid. I'll match 1.0 in my port for exact parity, and I'll watch for the reasoning in the tech report.

One data point that might be useful for you and for others reading this thread, in case it helps triage similar reports: on my independent MLX port, the PolyNorm activation (sigmoid(weight) coefficients + output_scale=0.5 + routed bias_clamp=±0.5) was by far the dominant term for the incoherence — applying just that, on the unmodified checkpoint, took held‑out NLL from ~23 → ~2.1 and produced fluent output at position 0 onward. The per‑layer RoPE (YaRN on full‑attention layers, plain swa_rope_theta on SWA layers, layer_idx % period for the SWA assignment) and the mHC h_post coefficient were both ≈neutral refinements on top of that (each < ~0.1 nat at short context in my tests). So for anyone porting from the checkpoint, PolyNorm is the load‑bearing fix. Thanks again for the fast turnaround and for surfacing the mHC design intent. 🙏

Thanks, @avlp12 — exactly right. The shipped PolyNorm used the raw coefficients (weight[i] * …) and was missing the sigmoid on them; adding sigmoid(weight) (together with output_scale=0.5 and the bias_clamp) is the load-bearing fix, which is why it moved your NLL the most.

Thanks again for the numbers. 🙏

Thank you + MLX builds are up 🙏 (Apple Silicon)

First, a sincere thank-you to the Motif team (@dongseokmotif , @iamwyldecat ) — this would not have been possible without your remarkably fast turnaround on the modeling-code fixes (SHA d2c9ac6) and the mHC design clarification in discussion #6. Once the reference matched the checkpoint, my independent MLX port fell straight into place (KL ≈1e-7/token vs the fixed reference). The speed of your fix is the only reason these builds exist today — so I wanted to share them back with the community as a thank-you.

세 줄 요약 (KR): 팀의 빠른 modeling 코드 수정 덕분에 Motif-3-Beta를 Apple Silicon용 MLX로 포팅·양자화했습니다. 수정 레퍼런스와 수치 일치(KL ≈1e-7)를 확인했고, 세 체급으로 공개합니다. 한국어·영어·코드 생성 모두 유창합니다.

MLX builds (avlp12, non-commercial research — base license carried forward)

Collection: https://huggingface.co/collections/avlp12/motif-3-beta-alis-mlx-apple-silicon-6a60e77168abddcfb5f8c2d8

Run it:

pip install git+https://github.com/avlp12/mlx-lm.git@motif
mlx_lm.generate --model avlp12/Motif-3-Beta-Alis-MLX-Dynamic-2.3bpw \
  --prompt "대한민국의 수도와 특징을 알려줘." --temp 1.0 --top-p 0.95

(The Motif model class isn't in upstream mlx-lm yet — the @motif fork branch adds it; an upstream PR is planned.)

How they were made

Mixed-bit affine quantization + clip-search + layerwise DWQ distillation (4.5bpw teacher, 45% Korean calibration) via the alis-dwq pipeline. The 2.3bpw floor: raw quant → clip (fixes greedy degeneration) → DWQ (cuts KL 58–72% vs the 8-bit reference). Eval + build-ladder charts are on each model card.

Honest notes

• Non-commercial research license (carried forward from the base model). Please respect it.
• These are quantizations of the Beta checkpoint; I'll re-quantize when the final release lands.
• Very-long-tail factual recall can be weak at 2.3bpw (a base-model property, not a quant artifact).
• One MLX-core bug (mx.split >2³¹-element silent corruption, ml-explore/mlx#3836) is worked around in the port.

Thanks again for building a genuinely strong Korean-capable open model — and for the fast, precise fixes that made this port possible. Happy to answer any questions about the MLX side. 🙏

This is fantastic, @avlp12 — thank you. Bringing Motif-3 to Apple Silicon with this much care is a genuine gift to the community.

The kind words mean a lot, but the credit really runs the other way: your report is what made the fix fast and precise in the first place. We're really glad it enabled this.

You can re-quantize when the final checkpoint lands. Thanks again for building these, sharing them back with the community. 🙏

Thank you @dongseokmotif - I was also able to make progress and successfully run inference on this now. Thank you and your team for the quick turnaround!