A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean
  Language Models
Authors: name: Hyunwoong Ko, name: Kichang Yang, name: Minho Ryu, name: Taekyoon Choi, name: Seungmu Yang, name: jiwung Hyun, name: Sungho Park

Polyglot is a pioneering project aimed at enhancing the non-English language
performance of multilingual language models. Despite the availability of
various multilingual models such as mBERT (Devlin et al., 2019), XGLM (Lin et
al., 2022), and BLOOM (Scao et al., 2022), researchers and developers often
resort to building monolingual models in their respective languages due to the
dissatisfaction with the current multilingual models non-English language
capabilities. Addressing this gap, we seek to develop advanced multilingual
language models that offer improved performance in non-English languages. In
this paper, we introduce the Polyglot Korean models, which represent a specific
focus rather than being multilingual in nature. In collaboration with TUNiB,
our team collected 1.2TB of Korean data meticulously curated for our research
journey. We made a deliberate decision to prioritize the development of Korean
models before venturing into multilingual models. This choice was motivated by
multiple factors: firstly, the Korean models facilitated performance
comparisons with existing multilingual models; and finally, they catered to the
specific needs of Korean companies and researchers. This paper presents our
work in developing the Polyglot Korean models, which propose some steps towards
addressing the non-English language performance gap in multilingual language
models.