Motif 2 12.7B technical report
Authors: name: Junghwan Lim; name: Sungmin Lee; name: Dongseok Kim; name: Taehyun Kim; name: Eunhwan Park; name: Jeesoo Lee; name: Jeongdoo Lee; name: Junhyeok Lee; name: Wai Ting Cheung; name: Dahye Choi; name: Jaeheui Her; name: Jaeyeon Huh; name: Hanbin Jung; name: Changjin Kang; name: Beomgyu Kim; name: Minjae Kim; name: Taewhan Kim; name: Youngrok Kim; name: Hyukjin Kweon; name: Haesol Lee; name: Kungyu Lee; name: Dongpin Oh; name: Yeongjae Park; name: Bokki Ryu; name: Dongjoo Weon
Published: 2025-11-07 10:32:16

Abstract: We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.