Understanding LLMs: A Comprehensive Overview from Training to Inference
Authors: name: Yiheng Liu; name: Hao He; name: Tianle Han; name: Xu Zhang; name: Mengyuan Liu; name: Jiaming Tian; name: Yutong Zhang; name: Jiaqi Wang; name: Xiaohui Gao; name: Tianyang Zhong; name: Yi Pan; name: Shaochen Xu; name: Zihao Wu; name: Zhengliang Liu; name: Xin Zhang; name: Shu Zhang; name: Xintao Hu; name: Tuo Zhang; name: Ning Qiang; name: Tianming Liu; name: Bao Ge
Published: 2024-01-04 02:43:57

Abstract: The introduction of ChatGPT has led to a significant increase in the
utilization of Large Language Models (LLMs) for addressing downstream tasks.
There's an increasing focus on cost-efficient training and deployment within
this context. Low-cost training and deployment of LLMs represent the future
development trend. This paper reviews the evolution of large language model
training techniques and inference deployment technologies aligned with this
emerging trend. The discussion on training includes various aspects, including
data preprocessing, training architecture, pre-training tasks, parallel
training, and relevant content related to model fine-tuning. On the inference
side, the paper covers topics such as model compression, parallel computation,
memory scheduling, and structural optimization. It also explores LLMs'
utilization and provides insights into their future development.