Marco-LLM: Bridging Languages via Massive Multilingual Training for
  Cross-Lingual Enhancement
Authors: name: Lingfeng Ming; name: Bo Zeng; name: Chenyang Lyu; name: Tianqi Shi; name: Yu Zhao; name: Xue Yang; name: Yefeng Liu; name: Yiyu Wang; name: Linlong Xu; name: Yangyang Liu; name: Xiaohu Zhao; name: Hao Wang; name: Heng Liu; name: Hao Zhou; name: Huifeng Yin; name: Zifu Shang; name: Haijun Li; name: Longyue Wang; name: Weihua Luo; name: Kaifu Zhang
Published: 2024-12-05 09:26:58

Abstract: Large Language Models (LLMs) have achieved remarkable progress in recent
years; however, their excellent performance is still largely limited to major
world languages, primarily English. Many LLMs continue to face challenges with
multilingual tasks, especially when it comes to low-resource languages. To
address this issue, we introduced Marco-LLM: Massive multilingual training for
cross-lingual enhancement LLM. We have collected a substantial amount of
multilingual data for several low-resource languages and conducted extensive
continual pre-training using the Qwen2 models. This effort has resulted in a
multilingual LLM named Marco-LLM. Through comprehensive evaluations on various
multilingual benchmarks, including MMMLU, AGIEval, Belebele, Flores-200, XCOPA
and many others, Marco-LLM has demonstrated substantial improvements over
state-of-the-art LLMs. Furthermore, Marco-LLM achieved substantial enhancements
in any-to-any machine translation tasks, showing the effectiveness of our
multilingual LLM. Marco-LLM is a pioneering multilingual LLM designed to not
only perform exceptionally well in multilingual tasks, including low-resource
languages, but also maintain strong performance in English and other major
languages, closing the performance gap between high- and low-resource language
capabilities. By bridging languages, this effort demonstrates our dedication to
ensuring LLMs work accurately across various languages.