Orion-14B: Open-source Multilingual Large Language Models
Authors: name: Du Chen; name: Yi Huang; name: Xiaopu Li; name: Yongqiang Li; name: Yongqiang Liu; name: Haihui Pan; name: Leichao Xu; name: Dacheng Zhang; name: Zhipeng Zhang; name: Kun Han
Published: 2024-01-20 12:29:27

Abstract: In this study, we introduce Orion-14B, a collection of multilingual large
language models with 14 billion parameters. We utilize a data scheduling
approach to train a foundational model on a diverse corpus of 2.5 trillion
tokens, sourced from texts in English, Chinese, Japanese, Korean, and other
languages. Additionally, we fine-tuned a series of models tailored for
conversational applications and other specific use cases. Our evaluation
results demonstrate that Orion-14B achieves state-of-the-art performance across
a broad spectrum of tasks. We make the Orion-14B model family and its
associated code publicly accessible https://github.com/OrionStarAI/Orion,
aiming to inspire future research and practical applications in the field.