ICLR 2025

Published as a conference paper at ICLR 2025
COMPUTE-CONSTRAINED DATA SELECTION
Junjie Oscar Yin
Johns Hopkins University
jyin27@jhu.edu
Alexander M. Rush
Cornell University
arush@cornell.edu
ABSTRACT
Data selection can reduce the amount of training data needed to finetune LLMs;
however, the efficacy of data selection scales directly with its compute. Motivated
by the practical challenge of compute-constrained finetuning, we consider the set-
ting in which both the cost of selecting data and training are budgeted for. We first
formalize the problem of data selection with a cost-aware utility function, and
model the data selection problem as trading off initial-selection cost for training
gain. We run a comprehensive sweep of experiments across multiple tasks, vary-
ing compute budget by scaling finetuning tokens, model sizes, and data selection
compute. Interestingly we find that many powerful data selection methods are al-
most never compute-optimal, and that cheaper data selection alternatives dominate
both from a theoretical and empirical perspective. For compute-optimal training,
we find that perplexity and gradient data selection require training-to-selection
model size ratios of 5x and 10x, respectively.
1
INTRODUCTION
The growth of large language models (LLMs) has motivated research into their resource profiles.
The compute cost of training LLMs is substantial, and, in many cases, the total compute budget is
predetermined: the number of accelerators and their usage hours are allocated in advanced. Thus, it
is critical to determine the optimal allocation of resources under a budget constraint. Past work on
compute-optimal LLMs (Hoffmann et al., 2022) studies if one could attain a better perplexity for a
given pretraining compute budget by balancing architecture and training decisions.
Similar resource questions exist during post-training finetuning of LLMs. We assume a common
setting where a single-base LLM needs to be finetuned for a downstream task. Numerous...