Title: From Acceleration to Saturation: Scaling Behavior of Bootstrapped Language Model Pretraining
Primary area: foundation or frontier models, including LLMs
Keywords: language modeling, continual pretraining, model growth, scaling law

Abstract: Bootstrapped pretraining, i.e., the reuse of a pretrained base model for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch.
However, its effectiveness remains unclear, especially when applied to overtrained base models.
In this work, we empirically study the scaling behavior of bootstrapped pretraining and find that its scaling efficiency diminishes in a predictable manner: The scaling exponent with respect to second-stage pretraining tokens decreases logarithmically with the number of tokens used to pretrain the base model.
 The joint dependence on first- and second-stage tokens is accurately modeled by a simple scaling law.
 Such saturation effect reveals a fundamental trade-off in multi-stage pretraining strategies: the more extensively a model is pretrained, the less additional benefit bootstrapping provides.
 Our findings provide practical insights for efficient language model training and raise important considerations for the reuse of overtrained models.

TL;DR: We show and quantify how an overtrained base model negatively impacts the performance in multi-stage pretraining