ICLR 2025

Published as a conference paper at ICLR 2025
U-SHAPED AND INVERTED-U SCALING BEHIND
EMERGENT ABILITIES OF LARGE LANGUAGE MODELS
Tung-Yu Wu, Melody Lo
National Taiwan University
{b08901133, peiyulo}@ntu.edu.tw
ABSTRACT
Large language models (LLMs) have been shown to exhibit emergent abilities
in some downstream tasks, where model performance stagnates at first and then
improves sharply and unpredictably with scale beyond a threshold. In this work,
we investigate the phenomenon by grouping questions based on difficulty level
and provide a possible explanation for emergent abilities. Specifically, we observe
U-shaped scaling for hard questions and inverted-U scaling followed by steady
improvement for easy questions. The two scaling patterns initially offset each
other, causing stagnant overall performance. The performance starts to soar when
the scaling pattern of easy questions reverts from inverse to standard scaling,
leading to emergent abilities. Based on this finding, we propose a simple yet
effective pipeline, called Slice-and-Sandwich, to predict the emergence threshold
and model performance beyond the threshold. Our code is publicly available at
https://github.com/tony10101105/ExpEmergence.
1
INTRODUCTION
Large language models (LLMs) (Team et al., 2023; Achiam et al., 2023; Brown, 2020; Touvron
et al., 2023a;b; Workshop et al., 2022; Li et al., 2023; Jiang et al., 2024) have shown strong potential
in various downstream applications (Jumper et al., 2021; Fawzi et al., 2022; Naveed et al., 2023;
Kaddour et al., 2023). Though the training-loss scaling law has been well established (Kaplan
et al., 2020; Hoffmann et al., 2022), the literature is inconclusive regarding how performance on
downstream tasks scales. In particular, for certain downstream tasks (Srivastava et al., 2023; Lin
et al., 2022a; Pilehvar & Camacho-Collados, 2019), LLMs display emergent abilities: performance
stagnates even when model training compute scales up hundredfold, and then improves sharp...