TeleChat Technical Report
Authors: name: Zihan Wang; name: Xinzhang Liu; name: Shixuan Liu; name: Yitong Yao; name: Yuyao Huang; name: Zhongjiang He; name: Xuelong Li; name: Yongxiang Li; name: Zhonghao Che; name: Zhaoxi Zhang; name: Yan Wang; name: Xin Wang; name: Luwen Pu; name: Huihan Xu; name: Ruiyu Fang; name: Yu Zhao; name: Jie Zhang; name: Xiaomeng Huang; name: Zhilong Lu; name: Jiaxin Peng; name: Wenjun Zheng; name: Shiquan Wang; name: Bingkai Yang; name: Xuewei he; name: Zhuoru Jiang; name: Qiyi Xie; name: Yanhan Zhang; name: Zhongqiu Li; name: Lingling Shi; name: Weiwei Fu; name: Yin Zhang; name: Zilu Huang; name: Sishi Xiong; name: Yuxiang Zhang; name: Chao Wang; name: Shuangyong Song
Published: 2024-01-08 10:43:19

Abstract: In this technical report, we present TeleChat, a collection of large language
models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It
includes pretrained language models as well as fine-tuned chat models that is
aligned with human preferences. TeleChat is initially pretrained on an
extensive corpus containing a diverse collection of texts from both English and
Chinese languages, including trillions of tokens. Subsequently, the model
undergoes fine-tuning to align with human preferences, following a detailed
methodology that we describe. We evaluate the performance of TeleChat on
various tasks, including language understanding, mathematics, reasoning, code
generation, and knowledge-based question answering. Our findings indicate that
TeleChat achieves comparable performance to other open-source models of similar
size across a wide range of public benchmarks. To support future research and
applications utilizing LLMs, we release the fine-tuned model checkpoints of
TeleChat's 7B and 12B variant, along with code and a portion of our pretraining
data, to the public community.