Tele-FLM Technical Report
Authors: name: Xiang Li; name: Yiqun Yao; name: Xin Jiang; name: Xuezhi Fang; name: Chao Wang; name: Xinzhang Liu; name: Zihan Wang; name: Yu Zhao; name: Xin Wang; name: Yuyao Huang; name: Shuangyong Song; name: Yongxiang Li; name: Zheng Zhang; name: Bo Zhao; name: Aixin Sun; name: Yequan Wang; name: Zhongjiang He; name: Zhongyuan Wang; name: Xuelong Li; name: Tiejun Huang
Published: 2024-04-25 14:34:47

Abstract: Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.