Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based
  Perspective
Authors: name: Chengyin Xu, name: Kaiyuan Chen, name: Xiao Li, name: Ke Shen, name: Chenggang Li

The rapid advancements in computing dramatically increase the scale and cost
of training Large Language Models (LLMs). Accurately predicting downstream task
performance prior to model training is crucial for efficient resource
allocation, yet remains challenging due to two primary constraints: (1) the
"emergence phenomenon", wherein downstream performance metrics become
meaningful only after extensive training, which limits the ability to use
smaller models for prediction; (2) Uneven task difficulty distributions and the
absence of consistent scaling laws, resulting in substantial metric
variability. Existing performance prediction methods suffer from limited
accuracy and reliability, thereby impeding the assessment of potential LLM
capabilities. To address these challenges, we propose a
Clustering-On-Difficulty (COD) downstream performance prediction framework. COD
first constructs a predictable support subset by clustering tasks based on
difficulty features, strategically excluding non-emergent and non-scalable
clusters. The scores on the selected subset serve as effective intermediate
predictors of downstream performance on the full evaluation set. With
theoretical support, we derive a mapping function that transforms performance
metrics from the predictable subset to the full evaluation set, thereby
ensuring accurate extrapolation of LLM downstream performance. The proposed
method has been applied to predict performance scaling for a 70B LLM, providing
actionable insights for training resource allocation and assisting in
monitoring the training process. Notably, COD achieves remarkable predictive
accuracy on the 70B LLM by leveraging an ensemble of small models,
demonstrating an absolute mean deviation of 1.36% across eight important LLM
evaluation benchmarks.