Tele-FLM Technical Report
Authors: name: Xiang Li, name: Yiqun Yao, name: Xin Jiang, name: Xuezhi Fang, name: Chao Wang, name: Xinzhang Liu, name: Zihan Wang, name: Yu Zhao, name: Xin Wang, name: Yuyao Huang, name: Shuangyong Song, name: Yongxiang Li, name: Zheng Zhang, name: Bo Zhao, name: Aixin Sun, name: Yequan Wang, name: Zhongjiang He, name: Zhongyuan Wang, name: Xuelong Li, name: Tiejun Huang

Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.