FLM-101B: An Open LLM and How to Train It with $100K Budget
Authors: name: Xiang Li; name: Yiqun Yao; name: Xin Jiang; name: Xuezhi Fang; name: Xuying Meng; name: Siqi Fan; name: Peng Han; name: Jing Li; name: Li Du; name: Bowen Qin; name: Zheng Zhang; name: Aixin Sun; name: Yequan Wang
Published: 2023-09-07 17:07:36

Abstract: Large language models (LLMs) have achieved remarkable success in NLP and
multimodal tasks. Despite these successes, their development faces two main
challenges: (i) high computational cost; and (ii) difficulty in conducting fair
and objective evaluations. LLMs are prohibitively expensive, making it feasible
for only a few major players to undertake their training, thereby constraining
both research and application opportunities. This underscores the importance of
cost-effective LLM training. In this paper, we utilize a growth strategy to
significantly reduce LLM training cost. We demonstrate that an LLM with 101B
parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
existing evaluations that focus more on knowledge-oriented abilities. We
introduce our benchmark including evaluations on important aspects of
intelligence including symbolic mapping, itrule understanding, pattern mining,
and anti-interference. Such evaluations minimize the potential impact of
memorization. Experimental results show that our model FLM-101B, trained with a
budget of 100K, achieves comparable performance to powerful and well-known
models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
contexts unseen in training data. The checkpoint of FLM-101B will be
open-sourced at https://huggingface.co/CofeAI/FLM-101B.