Motif 2 12.7B technical report
Authors: name: Junghwan Lim, name: Sungmin Lee, name: Dongseok Kim, name: Taehyun Kim, name: Eunhwan Park, name: Jeesoo Lee, name: Jeongdoo Lee, name: Junhyeok Lee, name: Wai Ting Cheung, name: Dahye Choi, name: Jaeheui Her, name: Jaeyeon Huh, name: Hanbin Jung, name: Changjin Kang, name: Beomgyu Kim, name: Minjae Kim, name: Taewhan Kim, name: Youngrok Kim, name: Hyukjin Kweon, name: Haesol Lee, name: Kungyu Lee, name: Dongpin Oh, name: Yeongjae Park, name: Bokki Ryu, name: Dongjoo Weon
Organization: Motif Technologies

We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.