Establishing Task Scaling Laws via Compute-Efficient Model Ladders
Authors: name: Akshita Bhagia; name: Jiacheng Liu; name: Alexander Wettig; name: David Heineman; name: Oyvind Tafjord; name: Ananya Harsh Jha; name: Luca Soldaini; name: Noah A. Smith; name: Dirk Groeneveld; name: Pang Wei Koh; name: Jesse Dodge; name: Hannaneh Hajishirzi
Published: 2024-12-05 18:21:49

Abstract: We develop task scaling laws and model ladders to predict the individual task
performance of pretrained language models (LMs) in the overtrained setting.
Standard power laws for language modeling loss cannot accurately model task
performance. Therefore, we leverage a two-step prediction approach: first use
model and data size to predict a task-specific loss, and then use this task
loss to predict task performance. We train a set of small-scale "ladder"
models, collect data points to fit the parameterized functions of the two
prediction steps, and make predictions for two target models: a 7B model
trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
models only costs 1% of the compute used for the target models. On four
multiple-choice tasks written in ranked classification format, we can predict
the accuracy of both target models within 2 points of absolute error. We have
higher prediction error on four other tasks (average absolute error 6.9) and
find that these are often tasks with higher variance in task metrics. We also
find that using less compute to train fewer ladder models tends to deteriorate
predictions. Finally, we empirically show that our design choices and the
two-step approach lead to superior performance in establishing scaling laws.