QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM
  Pretraining
Authors: name: Fengze Liu, name: Weidong Zhou, name: Binbin Liu, name: Zhimiao Yu, name: Yifan Zhang, name: Haobin Lin, name: Yifeng Yu, name: Xiaohuan Zhou, name: Taifeng Wang, name: Yong Cao

Quality and diversity are two critical metrics for the training data of large
language models (LLMs), positively impacting performance. Existing studies
often optimize these metrics separately, typically by first applying quality
filtering and then adjusting data proportions. However, these approaches
overlook the inherent trade-off between quality and diversity, necessitating
their joint consideration. Given a fixed training quota, it is essential to
evaluate both the quality of each data point and its complementary effect on
the overall dataset. In this paper, we introduce a unified data selection
framework called QuaDMix, which automatically optimizes the data distribution
for LLM pretraining while balancing both quality and diversity. Specifically,
we first propose multiple criteria to measure data quality and employ domain
classification to distinguish data points, thereby measuring overall diversity.
QuaDMix then employs a unified parameterized data sampling function that
determines the sampling probability of each data point based on these quality
and diversity related labels. To accelerate the search for the optimal
parameters involved in the QuaDMix framework, we conduct simulated experiments
on smaller models and use LightGBM for parameters searching, inspired by the
RegMix method. Our experiments across diverse models and datasets demonstrate
that QuaDMix achieves an average performance improvement of 7.2% across
multiple benchmarks. These results outperform the independent strategies for
quality and diversity, highlighting the necessity and ability to balance data
quality and diversity.