Observational Scaling Laws and the Predictability of Language Model
  Performance
Authors: name: Yangjun Ruan; name: Chris J. Maddison; name: Tatsunori Hashimoto
Published: 2024-05-17 17:49:44

Abstract: Understanding how language model performance varies with scale is critical to
benchmark and algorithm development. Scaling laws are one approach to building
this understanding, but the requirement of training models across many
different scales has limited their use. We propose an alternative,
observational approach that bypasses model training and instead builds scaling
laws from ~80 publically available models. Building a single scaling law from
multiple model families is challenging due to large variations in their
training compute efficiencies and capabilities. However, we show that these
variations are consistent with a simple, generalized scaling law where language
model performance is a function of a low-dimensional capability space, and
model families only vary in their efficiency in converting training compute to
capabilities. Using this approach, we show the surprising predictability of
complex scaling phenomena: we show that several emergent phenomena follow a
smooth, sigmoidal behavior and are predictable from small models; we show that
the agent performance of models such as GPT-4 can be precisely predicted from
simpler non-agentic benchmarks; and we show how to predict the impact of
post-training interventions like Chain-of-Thought and Self-Consistency as
language model capabilities continue to improve.