Orion-14B: Open-source Multilingual Large Language Models
Authors: name: Du Chen, name: Yi Huang, name: Xiaopu Li, name: Yongqiang Li, name: Yongqiang Liu, name: Haihui Pan, name: Leichao Xu, name: Dacheng Zhang, name: Zhipeng Zhang, name: Kun Han

In this study, we introduce Orion-14B, a collection of multilingual large
language models with 14 billion parameters. We utilize a data scheduling
approach to train a foundational model on a diverse corpus of 2.5 trillion
tokens, sourced from texts in English, Chinese, Japanese, Korean, and other
languages. Additionally, we fine-tuned a series of models tailored for
conversational applications and other specific use cases. Our evaluation
results demonstrate that Orion-14B achieves state-of-the-art performance across
a broad spectrum of tasks. We make the Orion-14B model family and its
associated code publicly accessible https://github.com/OrionStarAI/Orion,
aiming to inspire future research and practical applications in the field.