Trillion 7B Technical Report
Authors: name: Sungjun Han; name: Juyoung Suk; name: Suyeong An; name: Hyungguk Kim; name: Kyuseok Kim; name: Wonsuk Yang; name: Seungtaek Choi; name: Jamin Shin
Published: 2025-04-21 20:54:44

Abstract: We introduce Trillion-7B, the most token-efficient Korean-centric
multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA)
mechanism enables highly efficient and effective knowledge transfer from
English to target languages like Korean and Japanese. Combined with optimized
data mixtures, language-specific filtering, and tailored tokenizer
construction, Trillion-7B achieves competitive performance while dedicating
only 10\% of its 2T training tokens to multilingual data and requiring just
59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations
across 27 benchmarks in four languages demonstrate Trillion-7B's robust
multilingual performance and exceptional cross-lingual consistency.