CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
  Language Model Pre-training
Authors: name: Shizhe Diao, name: Yu Yang, name: Yonggan Fu, name: Xin Dong, name: Dan Su, name: Markus Kliegl, name: Zijia Chen, name: Peter Belcak, name: Yoshi Suhara, name: Hongxu Yin, name: Mostofa Patwary, name: Yingyan, name: Lin, name: Jan Kautz, name: Pavlo Molchanov

Pre-training datasets are typically collected from web content and lack
inherent domain divisions. For instance, widely used datasets like Common Crawl
do not include explicit domain labels, while manually curating labeled datasets
such as The Pile is labor-intensive. Consequently, identifying an optimal
pre-training data mixture remains a challenging problem, despite its
significant benefits for pre-training performance. To address these challenges,
we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
automated framework that discovers, evaluates, and refines data mixtures in a
pre-training setting. Specifically, CLIMB embeds and clusters large-scale
datasets in a semantic space and then iteratively searches for optimal mixtures
using a smaller proxy model and a predictor. When continuously trained on 400B
tokens with this mixture, our 1B model exceeds the state-of-the-art
Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
clusters as a research playground, and ClimbMix, a compact yet powerful
400-billion-token dataset designed for efficient pre-training that delivers
superior performance under an equal token budget. We analyze the final data
mixture, elucidating the characteristics of an optimal data mixture. Our data
is available at: https://research.nvidia.com/labs/lpr/climb/