Densing Law of LLMs
Authors: name: Chaojun Xiao; name: Jie Cai; name: Weilin Zhao; name: Guoyang Zeng; name: Xu Han; name: Zhiyuan Liu; name: Maosong Sun
Published: 2024-12-05 16:31:13

Abstract: Large Language Models (LLMs) have emerged as a milestone in artificial
intelligence, and their performance can improve as the model size increases.
However, this scaling brings great challenges to training and inference
efficiency, particularly for deploying LLMs in resource-constrained
environments, and the scaling trend is becoming increasingly unsustainable.
This paper introduces the concept of ``capacity density'' as a new
metric to evaluate the quality of the LLMs across different scales and
describes the trend of LLMs in terms of both effectiveness and efficiency. To
calculate the capacity density of a given target LLM, we first introduce a set
of reference models and develop a scaling law to predict the downstream
performance of these reference models based on their parameter sizes. We then
define the effective parameter size of the target LLM as the parameter
size required by a reference model to achieve equivalent performance, and
formalize the capacity density as the ratio of the effective parameter size to
the actual parameter size of the target LLM. Capacity density provides a
unified framework for assessing both model effectiveness and efficiency. Our
further analysis of recent open-source base LLMs reveals an empirical law (the
densing law)that the capacity density of LLMs grows exponentially over time.
More specifically, using some widely used benchmarks for evaluation, the
capacity density of LLMs doubles approximately every three months. The law
provides new perspectives to guide future LLM development, emphasizing the
importance of improving capacity density to achieve optimal results with
minimal computational overhead.