Kanana: Compute-efficient Bilingual Language Models
Authors: name: Kanana LLM Team; name: Yunju Bak; name: Hojin Lee; name: Minho Ryu; name: Jiyeon Ham; name: Seungjae Jung; name: Daniel Wontae Nam; name: Taegyeong Eo; name: Donghun Lee; name: Doohae Jung; name: Boseop Kim; name: Nayeon Kim; name: Jaesun Park; name: Hyunho Kim; name: Hyunwoong Ko; name: Changmin Lee; name: Kyoung-Woon On; name: Seulye Baeg; name: Junrae Cho; name: Sunghee Jung; name: Jieun Kang; name: EungGyun Kim; name: Eunhwa Kim; name: Byeongil Ko; name: Daniel Lee; name: Minchul Lee; name: Miok Lee; name: Shinbok Lee; name: Gaeun Seo
Published: 2025-02-26 08:36:20

Abstract: We introduce Kanana, a series of bilingual language models that demonstrate
exceeding performance in Korean and competitive performance in English. The
computational cost of Kanana is significantly lower than that of
state-of-the-art models of similar size. The report details the techniques
employed during pre-training to achieve compute-efficient yet competitive
models, including high quality data filtering, staged pre-training, depth
up-scaling, and pruning and distillation. Furthermore, the report outlines the
methodologies utilized during the post-training of the Kanana models,
encompassing supervised fine-tuning and preference optimization, aimed at
enhancing their capability for seamless interaction with users. Lastly, the
report elaborates on plausible approaches used for language model adaptation to
specific scenarios, such as embedding, retrieval augmented generation, and
function calling. The Kanana model series spans from 2.1B to 32.5B parameters
with 2.1B models (base, instruct, embedding) publicly released to promote
research on Korean language models.