Compute-Constrained Data Selection

## Abstract

Abstract Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the setting in which both the cost of selecting data and training are budgeted for.
We first formalize the problem of data selection with a cost-aware utility function,
and model the data selection problem as trading off initial-selection cost for training gain.
We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute. These experiments show the validity of this model in real-world experiments. Interestingly we find that many powerful data selection methods are almost never compute-optimal, and that cheaper data selection alternatives dominate both from a theoretical and empirical perspective.

## 1 Introduction

The growth of large language models (LLMs) has motivated research into their resource profiles. The compute cost of training LLMs is substantial, and, in practice, the total compute budget is predetermined: the number of accelerators and their usage hours are allocated in advanced. Thus, it is critical to determine the optimal allocation of resources under a budget constraint.
Past work on compute-optimal LLMs (Hoffmann et al., ) studies if one could attain a better perplexity for a given pretraining compute budget by balancing architecture and training decisions.

Similar resource questions exist during post-training finetuning of LLMs. We assume a common setting where a single-base LLM needs to be finetuned for a downstream task. Numerous works have sought to induce certain abilities by training from large instruction tuning datasets (Sanh et al., ; Wei et al., ; Mishra et al., ). There are several different resource constraints which might make this challenging. For example work on parameter efficient fine-tuning, targets improving the memory-usage of this stage (Hu et al., ). In this work we focus instead on compute-constrained finetuning.

A promising approach to reducing compute requirements for finetuning is data selection. Data selection is a foundational approach in machine learning where the objective is to create an minimal dataset from a collection of data (John, ). Given the large computational cost of each gradient step, reducing dataset size is an appealing way to reduce this resource usage. Moreover, as larger and more diverse instruction tuning collections become available, likely only a subset of the data provides the most value for any given task. Recent work has shown that careful data selection can vastly increase the effectiveness of finetuning per step (Chen et al., ; Zhou et al., ).

Yet, even if data selection is effective, it does not a priori imply that it is compute-optimal. Given the base-level compute effectiveness of gradient descent on LLM models, data selection methods need to improve upon standard training in proportion to their added cost.
In other words, a compute-optimal method should both improve training and be cheap to compute. In this work, we study this setting of compute-constrained data selection, and argue that this is a critical factor for practical adoption that is being under considered in method development.

Concretely, we aim to quantify the trade off between model size, number of tokens, and data selection in LLM finetuning, such that practitioners can make well-informed decisions when choosing how to best allocate compute. We first formalize this problem as a compute-constrained combinatorial optimization problem and then discuss our compute-aware modification. We develop an categorization of different approaches for this task and model their compute scaling. This is used to frame a broad series of empirical training runs under varying compute constraints, scaling both model sizes and training length. We train over 600 models, ranging from 7 to 70 billion parameters, across 6 data selection methods and 3 downstream tasks, recording final task performances for each.

The results from this study argue that complex data selection methods are almost never Pareto-optimal in the compute-constrained setting and that simple statistical methods such as sparse retrieval should be preferred. In particular, powerful data selection methods that use model perplexity or gradient information tend to be FLOP inefficient both from a theoretical and empirical perspective. That is not to say these methods are not performance effective in improving the effectiveness of finetuning. For example they should be used in settings with repeated training with different tasks on the same underlying models. We additionally fit parameteric models to quantify the effectiveness of various approaches in a compute-aware manner. We hope that this framework and setting can motivate further research into cheaper data selection methods that can produce better models with less compute. Codebase and datasets to reproduce our results are available at [https://github.com/oseyosey/CCDS](https://github.com/oseyosey/CCDS).

## 2 Related Work

Compute Scaling for Model Size and Transfer Learning. Kaplan et al. () established the use of compute-scaling laws for language models and showed a power-law relationship between model size and loss over varying orders of magnitude. More recent works expand on the original formulations by considering learning rate schedule matching, multiple-epoch training, and hyperparameters (Hoffmann et al., ; Muennighoff et al., ; Bi et al., ).

Hernandez et al. () study scaling in the post-training setting but only model the relationship between pretraining and finetuning data loss. Similar work  (Lin et al., ; Isik et al., ; Zhang et al., ) studies the scaling in post-training by modeling the relation between pretraining data size, model size, finetuning method and downstream test loss. These models do not consider the use of data selection. Recent work considers the relationship between data selection and scaling in the vision domain (Goyal et al., ), but this work does not consider the compute needed for data selection in their scaling analysis.

Data Selection for Language Models. Data selection takes the full training data as input and chooses a subset to the train (Albalak et al., ). It can be viewed as a coreset selection problem (Mirzasoleiman et al., ; Killamsetty et al., ), which aims to select a subset from the given training dataset such that the model achieves performance similar to the full dataset.

There are many different approaches to LLM data selection. The simplest are non-model specific approaches such as manual scoring functions (Chen et al., ), surface level features (Robertson et al., ), and n-gram features (Xie et al., ). On the other hand, more effective methods use LLMs to assign utility. One class uses model forward inference information such as utility scores from generations, model embeddings, and perplexity (Wettig et al., ; Marion et al., ; Ivison et al., ). Another class uses model gradients to define influence function style selections (Killamsetty et al., ; Han, ; Xia et al., ). See Section 4 for further description.

Task-Specific Finetuning from General-Purpose Instruction Datasets.
While data selection can be applied in any finetuning setting, it is most impactful as a method to train a targeted model
from a general-purpose dataset. In the case of LLMs, this setting is commonly training a task-specific model from an “instruction-tuning” dataset.
Several general-purpose instruction tuning datasets exist for LLM post-training (Sanh et al., ; Wei et al., ; Mishra et al., ; Chung et al., ). Instruction-tuned models can handle a variety of possible inputs for downstream use cases as either classification or generative model (Wang et al., ; ).

Instruction-tuned models are the most effective model for many downstream tasks; however they require training on a very large and expensive set of data.
To get around this issue, models have demonstrated strong results using small subsets of instruction tuning data (Chen et al., ; Lu et al., ; Zhou et al., ).
As the size of dataset increases, automated measures of quality selection has become a growing focus, particularly when many targeted models are needed.
Therefore while instruction-tuning is not the direct focus of this work, it provide a real-world applications of compute-constrained data selection.

## 3 Background

The goal of data selection is to choose a subset of data points from a large dataset to optimize model performance on a target task. In a learning task, we are given a large training set, $\mathcal{D}$, a target test dataset, $\mathcal{T}$, and a validation set, $\mathcal{V}$. Our goal is to find the optimal subset $\mathcal{S}\subseteq\mathcal{X}$ such that the model $\theta=T(\mathcal{S})$ trained on $\mathcal{S}$ maximizes the performance on $\mathcal{T}$ under a given data constraint:

$$ $\displaystyle\mathcal{S}^{*}=\underset{\mathcal{S}\subseteq\mathcal{D}}{\arg \max}\quad P(\mathcal{T};T(\mathcal{S}))$ (1) $\displaystyle\text{subject to}\quad|\mathcal{S}|\leq K,$ $$

where $P$ denotes the performance of the model on the test set and $K$ is the max cardinality.

This problem is challenging to solve in the general case, particularly without access to the test set $\mathcal{T}$.
Approaches to the problem therefore commonly make two implicit assumptions:
(1) the performance function, $P(\mathcal{T};T(\mathcal{S}))$, is monotonic and submodular, non-increasing marginal utility, in the dataset chosen (Kirchhoff & Bilmes, ), and (2) the validation set $\mathcal{V}$ correlates with the test set $\mathcal{T}$.
Under these assumptions, we can argue for a greedy data selection approach (Kirchhoff & Bilmes, ). This allows decomposing the total objective, $P(\mathcal{T};T({\mathcal{S}}))$, by considering the contribution of individual training points to the performance on the validation set $P(\mathcal{V};T(\{x\}))$ for each $x\in\mathcal{S}$.

To estimate the marginal contribution of $x\in\mathcal{S}$, most data selection methods use a utility function $v(x;\mathcal{V})$—as a proxy to $P(\mathcal{V};T(\{x\}))$—to give the utility of each data point $x$ based on its relevance (Albalak et al., ).
By ranking the data points $\mathcal{D}$ based on $v$ and selecting those that minimize the total utility within the data budget $K$, greedy data selection aims to obtain a high-performing subset $\mathcal{S}^{*}$.
To summarize, we consider data selection methods that target Equation 1 with a two-step greedy algorithm: score all points and then select points up to the budget $K$.

## 4 Compute-Constrained Data Selection

While the framework presented in Section 3 provides a general method for data selection, we argue that it is insufficient for the practical challenge of finetuning LLMs.
The issue is that LLM finetuning is often bottlenecked by a computational budget and not a data budget.
There are two major computational bottlenecks in this process: (1) the cost of training the model on this data ($C_{T}$), and (2) the cost of computing the utility function on this data ($C_{v}$). The true cost of $C_{v}$ can reduce significantly the amount of training points we can select for given computational budget .

Assuming we at minimal require the computation of a utility function over the dataset, we can define the compute-constrained data selection objective as,

$$ $\displaystyle\mathcal{S}^{*}=\underset{\mathcal{S}\subseteq\mathcal{D}}{\arg \max}\quad P(\mathcal{T};T(\mathcal{S}))$ (2) $\displaystyle\text{subject to}\quad C_{T(\mathcal{S})}+\sum_{x\in\mathcal{D}}C _{v(x)}\leq K.$ $$

Here $K$ is now the compute, e.g. maximum number of FLOPs, allocated for data selection and training, and we assume calculation of $v$ is a fixed-cost independent of optimization.

### 4.1 Compute Cost of Data Selection Utilities

**Table 1: Utility Functions and Computational Costs for Data Selection Methods.**
| Method | Utility Function | Computational Cost | $C_{\text{forward}}(x)$ |
| --- | --- | --- | --- |
| Lexicon-Based | $\frac{1}{|\mathcal{V}|}\sum_{x^{\prime}}\text{BM25}(x,x^{\prime})$ | $c_{\text{BM25}}(|x|+|\mathcal{V}||x|)$ | $\approx$ 0 |
| Embedding-Based | $\frac{1}{|\mathcal{V}|}\sum_{x^{\prime}}\cos(\text{Emb}(x),\text{Emb}(x^{<br>\prime}))$ | $C_{\text{embed}}(x)+C_{\text{embed}}(\mathcal{V})$ | $\approx\epsilon$ |
| Perplexity-Based | $\text{PPL}_{\theta_{\mathcal{V}}}(x)$ | $C_{\text{forward}}(x)$ | $\approx$ 1 |
| Gradient-Based | $\eta_{t}\langle\nabla_{\theta}\ell(x;\theta^{t}),\nabla_{\theta}\ell(\mathcal{<br>V};\theta^{t})\rangle$ | $3\times C_{\text{forward}}(x)+C_{\text{grad}}(\mathcal{V})$ | $\approx$ 3 |

To make these costs more tangible, we consider four classes of data selection in this work, that represent three different levels of compute. This section summarizes their main properties, i.e. their core utility functions and computational costs. Table 1 describes the important computational costs from this section.

#### Lexicon-Based.

Lexicon data selection methods utilize statistical properties of text to evaluate the relevance of data points without relying on deep learning models. One of the most effective lexicon-based methods is BM25 (Robertson et al., ; Silva & Barbosa, ), which scores data points based on the frequency of terms. The utility function $v_{\text{BM25}}(x;\mathcal{V})$ assigns a relevance score to each data point by averaging the BM25 scores with the validation, $v_{\text{BM25}}(x)=\frac{1}{|\mathcal{V}|}\sum_{x^{\prime}\in\mathcal{V}}\text
{BM25}(x,x^{\prime})$. Since the algorithm can be run efficiently with a single-core cpu, the data selection FLOPs are almost 0.

#### Embedding-Based.

These methods utilize embedding models to select data points that are most similar to the target data (Rubin et al., ). The utility function $v_{\text{retrieval}}(x;\mathcal{V})$ assigns a score to each data point $x$ based its cosine similarity with validation data in $\mathcal{V}$; that is, $v_{\text{retrieval}}(x)=\frac{1}{|\mathcal{V}|}\sum_{x^{\prime}\in\mathcal{V}}
\cos\left(\text{Emb}(x),\text{Emb}(x^{\prime})\right)$. Assuming we are using a very small model (on the order of BERT-size) and the embedding requires only one-time transformation of data points into dense vectors, the data selection FLOPs are quite small. Liu et al. () demonstrated significant performance gains when selecting in-context examples. We apply a similar method, Embed, to the fine-tuning setting using a small T5-based dense embedding model (Ni et al., ).

#### Perplexity-Based.

Perplexity-based data selection utilizes language models to evaluate the utility of data points based on model loss (Antonello et al., ). The utility function $v_{\text{ppl}}(x;\mathcal{V})$ assigns a score to each data point $x$ by computing the perplexity (PPL) of $x$ under a language model $\theta_{\mathcal{V}}$ finetuned on $\mathcal{V}$; that is, $v_{\text{ppl}}(x)=\text{PPL}_{\theta_{\mathcal{V}}}(x)$. Top-PPL and Mid-PPL have both shown improved performance and training efficiency (Ankner et al., ; Marion et al., ), where Top-PPL ranks data points with the highest perplexity scores, and Mid-PPL does the same for points in the middle of the score distribution.

#### Gradient-Based.

These methods evaluate the utility of data points based on their influence on the model’s loss with respect to the target data (Pruthi et al., ). The utility function $v_{\text{grad}}(x;\mathcal{V})$ quantifies this influence by computing the inner product between the gradient of the loss on $x$ and the gradient of the loss on $\mathcal{V}$, scaled by the learning rate $\eta_{t}$; that is, $v_{\text{grad}}(x)=\eta_{t}\left\langle\nabla_{\theta}\ell(x;\theta^{t}),
\nabla_{\theta}\ell(\mathcal{V};\theta^{t})\right\rangle$, where $\nabla_{\theta}\ell(\mathcal{V};\theta^{t})=\frac{1}{|\mathcal{V}|}\sum_{x^{
\prime}\in\mathcal{V}}\nabla_{\theta}\ell(x^{\prime};\theta^{t})$, and $\theta^{t}$ are the model parameters at time step $t$. The computational cost of computing $v_{\text{grad}}(x)$ for each $x$ is approximated by:

$$ $C_{v_{\text{grad}}}(x)\approx C_{\text{backward}}(x)+C_{\text{grad}}(\mathcal{ V})\approx 3\times C_{\text{forward}}(x)+C_{\text{grad}}(\mathcal{V}),$ $$

where $C_{\text{forward}}(x)$ is the cost of a forward pass for $x^{(i)}$ on the model $\mathcal{M}$, and $C_{\text{grad}}(\mathcal{V})$ is the cost of computing $\nabla_{\theta}\ell(\mathcal{V};\theta^{t})$, computed once for all $x$. Computing gradients involves both forward and backward passes, totaling approximately three times the cost of a forward pass. Low-rank sgradiEnt Similarity Search (LESS) shows superior performance gain over cheaper methods and random selection (Xia et al., ).

Different implementation of these general categories leads to different cost. A detailed cost analysis of our selected data selection methods can be found in the Appendix B.

## 5 Modeling the Compute-Performance Relationship

Figure: Figure 1: Simulation of Performance under Constraints. $P(k)=\bar{P}\times\left(1-\exp\left(-\lambda\frac{C(k)}{C(|\mathcal{D}|)} \right)\right)$ The behavior of different data selection methods using our performance model. (Left-Small Budget) The Lexicon method may consistently outperform more advanced data selection methods if their initial cost is too high. Under our assumptions gradient can never be optimal as its cost exceeds 1 epoch of training. (Middle-Medium Budget) The perplexity method can become optimal once the total cost exceeds a given amount. (Right-Large Budget) The gradient methods can be optimal if training is more expensive than the fixed-cost, for example if using a much larger base model then data selection model. The simulation shows that the compute-optimal data selection method changes as a function of the compute budget and the performance rate associated with each method.
Refer to caption: x1.png

To analyze the trade-off between the compute of data selection methods and the expected gain in model performance, we define a simplified parametric form for expected peformance. Let $c$ be the fixed-cost of training on a single data point and $k=|\mathcal{S}|$ be the number of data points. Define $C(k)$ as the total cost of training and selection,

$$ $C(k)=c\times k+\sum_{x}C_{v(x)}.$ $$

The assumption in data selection is that more compute intensive methods are able to achieve higher performance with less samples,
but that the value of information gained from selecting additional data points diminishes as more points are explored.
We will assume that data points are exchangeable, and that all methods eventually reach the same upper-bound performance after 1 epoch, $\bar{P}=P(\mathcal{T};T(\mathcal{D}))$.

Our parametric model for the expected performance after training on $k$ data points should capture diminishing returns, dependence on computational cost of the method, and convergence toward an upper bound. We model this function as:

$$ $P(k)=(\bar{P}-P_{0})\times\left(1-\exp\left(-\lambda\frac{C(k)}{C(|\mathcal{D} |)}\right)\right)+P_{0}$ (3) $$

where $P_{0}$ is the zero-shot performance, $\bar{P}$ is the upper bound and $\lambda$ is the value the method extracts from additional compute in the utility function.

#### Simulation

Figure 1 presents a simulation of this performance function across different methods under varying compute constraints. We set $c$ to {4.5E+10, 4.5E+11, 9E+12}, $k$ to 100M, and utility compute to {4.5E+10, 4.5E+11, 9E+12}. The parameter $\lambda$ is set to {5, 10, 40, 80} for the Random, Lexicon, Perplexity, and Gradient methods, respectively. These settings demonstrate how different data selection methods may become compute-optimal under varying compute budgets. Note that for Medium and Large Budgets, we use a Small-size model for PPL and Gradient.

Using methodology similar to Hoffmann et al. (), the parameter $\lambda$ can be fit on the empirical measurements, which yields data-driven estimates. See Appendix C for more details on derivations and the fitting procedure.

## 6 Experimental Setup

As shown in Table 2, experiments vary the number of finetuning tokens for 5 data-selection methods and a fixed family of models, ranging from 7B to 70B parameters. The finetuning data budget is fixed as a percentage of the total finetuning tokens: {2.5, 5, 10, 25, 50, 100}%, across 3 target tasks. For each finetuning budget, we conduct multiple training runs with increasing compute, which is either allocated toward larger pre-trained model sizes or more sophisticated data selection methods.
We analyze each FLOP count to identify which runs achieve the highest performance on target benchmark. We then fit a power law to obtain a finetuned Pareto frontier for each model sizes.

**Table 2: Experimental Setup Overview.**
| Training Data | Data Selection Method | Model Size | Target Task |
| --- | --- | --- | --- |
| 2.5% | Random | Llama2 7B | MMLU |
| 5% | BM25 | Llama3 8B | BBH |
| 10% | Embed | Llama2 13B | IFEval |
| 25% | PPL | Llama2 70B |  |
| 50% | LESS |  |  |
| 100% |  |  |  |

#### Datasets

We follow Wang et al. () and curate a representative sample of instruction-tuned datasets as listed in Table 8. This includes: (1) datasets generated by researchers from existing NLP datasets, such as COT (Wei et al., ) and Flan V2 (Longpre et al., ) ; (2) datasets written by humans from scratch specifically for instruction tuning, including Dolly (Conover et al., ) and Open Assistant 1 (Köpf et al., ).

For evaluating the model we run on three challenging but different downstream tasks. These include: the Massive Multitask Language Understanding (MMLU, Hendrycks et al. ()) dataset measures models’ factual knowledge, comprising questions across 57 subjects, spanning difficulty levels from elementary to professional; Big-Bench-Hard (BBH, Suzgun et al. ()) curates 23 complex reasoning tasks from Big-Bench Srivastava et al. (). It is used to evaluate models’ general reasoning capabilities; and Instruction Following Evaluation (IFEval, Zhou et al. ()), which evaluates models’ instruction following abilities. For MMLU, we report 5-shot accuracy; for BBH, we report 3-shot exact match score; and for IFEval, we report 0-shot accuracy.

#### Pretrained Models

For all experiments, we train transformer language models with Llama architecture and tokenizer (Touvron et al., ; Dubey et al., ). Models range from 7B parameters to 70B parameters, and are trained for close to 100 million total fine-tuning tokens. Our experiments closely follow prior work on training and evaluating instruction-tuned models (Wang et al., ; Ivison et al., ).

We primarily focus on the Llama-2 model suite, containing Llama-2-7B, Llama-2-13B, and Llama-2-70B. Because the latest Llama-3 model suit only contains two model sizes (8B and 70B), we believe the three Llama-2 models are better suited in modelling the scaling behaviour. Nevertheless, We show that our results generalizes well to Llama-3 by experimenting with the smaller base model Llama-3-8B.

Figure: Figure 2: Performance for Different Data Selection Methods. We show all of our different runs for a given model size, where each scatter point is the final target task performance of a single run. (A, B, C) show MMLU results across three model sizes, while (D, E, F) present BBH results across three model sizes. For each run, we determine the optimal finetuning strategy—a combination of data selection method and number of finetuning tokens—that achieves the highest performance under a particular FLOPs budget. We fit a pareto front in dashed line based on these optimal strategies. At small and medium compute budgets (A, B, D, E), cheaper data selection methods like BM25 and EMBED outperform PPL and LESS, which rely on model information. At larger compute budgets (C, F), however, PPL and LESS become compute-optimal after using 5% of the fine-tuning tokens.
Refer to caption: x2.png

## 7 Results

#### Empirical Results.

Figure 2 shows the full results with 5 data selection methods across 3 pretrained model size and 2 of the target task. For these experiments, the unique training data budget is fixed at roughly {2.5%, 5%, 10%, 25%, 50%, 100%} of tokens. For each data budget, we finetune a set of models with increasing amount of compute that is allocated to either more parameters or more expensive data selection methods.

Note that for PPL and Gradient, a 7B model of the same class is always used for data selection; whereas for Embed a small encoder model is used. For MMLU, PPL is implemented as Mid-PPL, where points in the middle of perplexity score distribution is being ranked first. For BBH, PPL is implemented as Top-PPL, where points in the top of perplexity score distribution is ranked first.

The main 7B results in Figure 2 (A, D) and Figure 5 (b), show that cheap lexicon-based methods (BM25) and embedding-based methods (Embed) significantly outperform perplexity-based (PPL) and gradient-based methods (LESS). While PPL and LESS achieve better performance at the same data budget compared to these methods (see Figure 5), they are not compute optimal under the same compute budget due to the high FLOPs required for data selection. The marginal benefit one can get from using a more sophisticated data selection methods does not outweigh its cost in selecting these data. Additional results can be found in Appendix F.

As models scale to 13B Figure 2 (B, E),
we still find that expensive data selection methods underperform, even though, the relative cost of doing data selection for these methods diminishes as the training model size increases. As shown in Figure 2 (B, E), at 13B model sizes, cheaper methods are still preferred. We do see that after 5% finetuning tokens, PPL is more competitive, outperforming Embed in MMLU, and almost matching the pareto front at 25% finetuning tokens.

At the 70B model size, shown in Figure 2 (C, F), PPL and LESS outperform both BM25 and Embed finetuning tokens for the first time. This suggests that at very large compute budgets, more sophisticated and costly methods can gain a greater advantage compared to lexical and embedding methods. As model sizes continue to scale and LLM sizes shrink, these methods may become more efficient.

Results on Llama3 8B are nearly identical to the 7B we see for Llama2. This verify that the approach is not model specific. We include full results in Figure 7.

Figure: Figure 3: Parametric Fit of Performance with Compute-Constrained Data Selection. We fit a parametric model of the performance in Equation 3 and display that as curves to pair with the empirical results as scatter points. (A, B, C) show MMLU results and their parametric fit across three model sizes, while (D, E, F) present BBH results and their parametric fit across three model sizes. For each fixed model sizes, we show the finetuned pareto frontier in dashed gray line, which is a line in the log-linear space. Note that the x-axis is log-scale.
Refer to caption: x3.png

Fit of Compute-Performace Relationship. In Section 5 we propose a parameteric model for the relationship of data selection compute to model performance, Equation 3. This formula assumed that data selection diminish in utility and that performance is a direct function of compute.
To verify these assumption, we fit the parametic form to the the empirical curves by fitting the $\lambda$ parameter and allowing tolerance in $P_{0},\bar{P}$, per method and dataset. We model all final performances from our experiments as a parametric function of the compute budget $P(k)$.

Figure 3 shows fittings for each curve. Since the shape of these curves is dataset and model dependent, we are unable to forward predict specific performances from additional data selection compute. However the close fit of the curves indicates that the underlying model — significant performance gain at the cost of exponentially more compute investment — holds up in practical settings.

## 8 Analysis

Figure: Figure 4: Multiple Task-Specific Model Break-Even Analysis . Costs to perform gradient-based method (LESS) are spread over all the target tasks. Performance under compute-constraints reach the finetuned Pareto frontier at 10 tasks, surpassing it at 20 tasks.
Refer to caption: x4.png

#### Comparing Training versus Total Compute Budgeting.

While our primary interest is in the full compute constrained setting, we note that different results hold if targeting only a small training budget as in Equation 1. When the training-budget is fixed, we observe in  Figure 5 (a) that the gradient-based method (LESS) consistently outperforms other data selection methods, with the perplexity-based method (PPL) being the closest competitor.

Figure: (a)
Refer to caption: x5.png

#### Multiple Task-Specific Models.

Another interesting setting is when the goal is to train
multiple models from the same large training set, where each targets a different task. In this setting, the
gradient information calculation can be amortized between target tasks. However, even in this setting the cost of gradient calculation is still quite high. As shown in Figure 4, with Llama2 7B as the data selection model, LESS needs 20 tasks to surpass the finetuned pareto frontier. We perform similar analysis for other model sizes in Appendix G.

#### Data Similarity.

Figure 6 presents the Jaccard similarity of data selected by different data selection methods. We observe that BM25 and Embed tend to select highly similar data, which explains their comparable performance. In contrast, the data selected by LESS bears the least resemblance to the other methods. We perform similar analysis for different target tasks in Appendix H.

Figure: Figure 6: Data Similarity Between Data Selection Methods for MMLU.
Refer to caption: x7.png

## 9 Conclusion

Data selection is a valuable tool for fine-tuning language models, but its primary benefit — saving compute — must be balanced against the compute costs required to identify an optimal dataset. In practice, popular methods are surprisingly compute intensive, yielding training data reduction at the cost of worse compute-constrained performance. We hope that these results motivate further research into more compute-efficient data selection methods. As demonstrated by our 70B finetuning experiments, sophisticated data selection methods can leverage smaller models for data selection, allowing larger models from the same family to be trained more efficiently in the LLM setting.

## 10 Acknowledgments

We would like to thank Woojeong Kim and Celine Lee for their insightful feedback on the paper’s writing. We are grateful to Mengzhou Xia for assistance with setting up the codebase and valuable discussions on data selection. Additionally, we thank Junxiong Wang, Jing Nathan Yan, and Wenting Zhao for their support in conducting the early experiments. This work was supported by NSF IIS-1901030 and NSF CAREER 2037519.

## Appendix A FLOPs Computation

To compute the training FLOPs of a transformer model, we assume the backward pass has twice the FLOPs of the forward pass and follow the FLOP counts of various components of a transformer model for a single forward pass as in Kaplan et al. () as in Table 3. We account for all training FLOPs in our analysis, including those from the embedding matrices. For large models, the contribution of FLOPs and parameters from embedding matrices is minimal. We apply a factor of 2 to represent the multiply-accumulate cost (MAC).

**Table 3: Parameters and FLOPs per token for different operations.**
| Operation | Parameters | FLOPs per Token |
| --- | --- | --- |
| Embed | $n_{\text{vocab}}d_{\text{model}}$ | $2n_{\text{vocab}}d_{\text{model}}$ |
| Attention: QKV | $n_{\text{layer}}d_{\text{model}}3d_{\text{attn}}$ | $2n_{\text{layer}}d_{\text{model}}3d_{\text{attn}}$ |
| Attention: Mask (Q-K Dot) | — | $2n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$ |
| Attention: Project | $n_{\text{layer}}d_{\text{attn}}d_{\text{model}}$ | $2n_{\text{layer}}d_{\text{attn}}d_{\text{model}}$ |
| Feedforward | $n_{\text{layer}}3d_{\text{model}}d_{\text{ff}}$ | $2n_{\text{layer}}3d_{\text{model}}d_{\text{ff}}$ |
| De-embed (Head) | $d_{\text{model}}n_{\text{vocab}}$ | $2d_{\text{model}}n_{\text{vocab}}$ |
| Total (Non-Embedding) | $N^{\prime}=2d_{\text{model}}n_{\text{layer}}(2d_{\text{attn}}+1.5d_{\text{ff}}<br>)+d_{\text{model}}n_{\text{vocab}}$ | $C_{\text{forward}}=2N^{\prime}+2n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$ |
| Total | $N=2d_{\text{model}}n_{\text{layer}}(2d_{\text{attn}}+1.5d_{\text{ff}})+d_{<br>\text{model}}n_{\text{vocab}}+n_{\text{vocab}}d_{\text{model}}$ | $C_{\text{forward}}=2N+2n_{\text{layer}}n_{\text{ctx}}d_{\text{attn}}$ |

For better granularity, we show a comparision between our calculation and one that uses the common approximation $C_{train}=6ND$, where $C_{train}$ denotes FLOPs, $D$ denotes total number of training tokens, and $N$ is the number of parameters in Table 4. In practically, the differences in FLOP calculation is small and thus pose no difference our analysis.

**Table 4: FLOP comparison. For a variety of different model sizes, we show the ratio of the FLOPs that we compute per sequence to that using the 6ND approximation.**
| Model | $N$ | $n_{\text{layer}}$ | $n_{\text{ctx}}$ | $d_{\text{model}}$ | $d_{\text{ff}}$ | $d_{\text{attn}}$ | FLOPs per token | FLOP Ratio (Ours/6ND) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 2 7B | 7B | 32 | 4096 | 4096 | 11008 | $128\times 32$ | 4.69e+10 | 1.12 |
| Llama 2 13B | 13B | 40 | 4096 | 5120 | 13824 | $128\times 40$ | 8.82e+10 | 1.13 |
| Llama 2 70B | 70B | 80 | 4096 | 8192 | 28672 | $128\times 80$ | 5.03e+11 | 1.20 |

We use this in our calculation of data selection costs for methods that utilizes LLMs (i.e. perplexity-based and gradient-based).

## Appendix B Data-Selection FLOPs

In this section, we detailed our estimation of the costs to perform different data selection methods.

**Table 5: Data Selection Cost Summary for Each Data Selection Method**
| Data Selection Method | FLOPs Cost |
| --- | --- |
| BM25 | $1\times 10^{8}$ |
| Embed | $4.4\times 10^{16}$ |
| PPL | $1.53\times 10^{18}$ |
| LESS | $8.27\times 10^{18}$ |

### B.1 BM25

We assume that each data point incurs a computational cost of 1 FLOP for BM25, denoted as $c_{\text{BM25}}=1$. The total computation cost is approximated as proportional to the size of the dataset, $|\mathcal{D}|$. Therefore, the data selection cost for BM25 is estimated to be $1\times 10^{8}$ FLOPs.

### B.2 Embed

For the embedding-based method, we approximate the computation cost using the formula $C_{\text{forward}}=2ND$, where $N$ represents the number of model parameters and $D$ is the dataset size. Given that the embedding model used has $N=220M$ parameters (Ni et al., ), the data selection cost for Embed is estimated to be $4.4\times 10^{16}$ FLOPs.

### B.3 PPL

Perplexity-based methods require passing every data point through the language model. Given that the cost per token for a 7B model is $c=4.69\times 10^{10}$, as shown in Table 4, the data selection cost for PPL is equivalent to one forward pass. Therefore, using the Llama2 model for data selection, we approximate the perplexity computation cost to be $1.53\times 10^{18}$ FLOPs.

### B.4 LESS

LESS involves a two-step process: a 4-epoch warm-up training on 5% of the dataset $\mathcal{D}$ followed by gradient feature computation over the entire training and validation datasets. First, we calculate the FLOPs required for the 4-epoch warm-up training. Then, using the relationship between the time required for warm-up training and gradient feature computation, as outlined in Table 6 from (Xia et al., ), we estimate the FLOPs needed for gradient feature computation. Combining these, we approximate the total cost of LESS using Llama2 as the data selection model to be $8.27\times 10^{18}$ FLOPs.

**Table 6: Asymptotic complexity, wall-clock runtime (measured as single A100 GPU hours), and storage cost associated with each step in LESS Xia et al. ().**
|  | Warmup LoRA Training | Gradient Features Computation | Data Selection |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Compute | Complexity | Actual | Complexity | Actual | Complexity | Actual |
| Compute | $\mathcal{O}(|\mathcal{D}_{\text{warmup}}|\cdot N)$ | 6 Hours | $\mathcal{O}(|\mathcal{D}|\cdot N)$ | 48 Hours | $\mathcal{O}(|\mathcal{D}|\cdot|\mathcal{D}_{\text{val}}|\cdot d)$ | $\leq 1$ Min |
| Storage | - | - | $\mathcal{O}(|\mathcal{D}|\cdot N\cdot d)$ | 17.7 GB | - | - |

## Appendix C Fitting Parametric Function

In this appendix, we describe the process for fitting a parametric model that captures the relationship between the number of data points $k$ and performance $P(k)$, as a function of computational cost. The model captures diminishing returns, dependence on computational cost, and convergence toward an upper bound.

We model the expected performance $P(k)$ after training on $k$ data points as follows:

$$ $P(k)=(\bar{P}-P_{0})\times\left(1-\exp\left(-\lambda\frac{C(k)}{C(|\mathcal{D} |)}\right)\right)+P_{0}$ $$

where:

- •
$P_{0}$ is the zero-shot performance (i.e., performance without additional training).
- •
$\bar{P}$ is the upper bound on performance (i.e., the maximum achievable performance).
- •
$\lambda$ is a parameter controlling how efficiently the method extracts value from additional compute.
- •
$C(k)$ is the computational cost of selecting and training on $k$ data points.
- •
$C(|\mathcal{D}|)$ is the total computational cost of training on the entire dataset.

The goal is to fit the parameters $P_{0}$, $\bar{P}$, and $\lambda$ to observed data. We fit the model by minimizing the difference between the predicted performance $P(k)$ and the observed performance $P_{\text{obs}}(k)$. This is formulated as the following optimization problem:

$$ $\min_{P_{0},\bar{P},\lambda}\sum_{i=1}^{N}\left(P(k_{i};P_{0},\bar{P},\lambda) -P_{\text{obs},i}\right)^{2}$ $$

where:

- •
$N$ is the number of data points.
- •
$P_{\text{obs},i}$ is the observed performance at the $i$-th data point.
- •
$k_{i}$ is the number of data points used for training in the $i$-th observation.
- •
$P(k_{i};P_{0},\bar{P},\lambda)$ is the predicted performance using the parametric model.

To ensure meaningful parameter estimates, we impose the following constraints:

- •
$P_{0}\geq 0$, as performance cannot be negative.
- •
$P_{0}\leq\bar{P}$, ensuring that performance does not exceed the upper bound $\bar{P}$.
- •
$\lambda\geq 0$, as $\lambda$ represents the rate at which performance improves with compute.

The parameter $\bar{P}$ is set slightly above the maximum observed performance:

$$ $\bar{P}=\max_{i}P_{\text{obs},i}+\epsilon$ $$

where $\epsilon=0.05$ is a small buffer to ensure convergence to the upper bound.

We set the initial guesses for the parameters as follows:

- •
$P_{0}^{(0)}=P_{\text{obs},1}$, the observed performance at zero-shot (i.e., without training).
- •
$\bar{P}^{(0)}=\max_{i}P_{\text{obs},i}$.
- •
$\lambda^{(0)}=1.0$, a reasonable initial guess for the compute extraction efficiency.

We use the Levenberg-Marquardt algorithm to minimize the objective function. This method is effective for solving non-linear least squares problems and efficiently handles the non-linear nature of our parametric model.

The optimization problem is solved as follows:

$$ $(P_{0}^{\ast},\bar{P}^{\ast},\lambda^{\ast})=\arg\min_{P_{0},\bar{P},\lambda} \sum_{i=1}^{N}\left(P(k_{i};P_{0},\bar{P},\lambda)-P_{\text{obs},i}\right)^{2}$ $$

The optimization yields the following fitted parameters:

$$ $P_{0}^{\ast}=\text{[fitted value]},\quad\bar{P}^{\ast}=\text{[fitted value]}, \quad\lambda^{\ast}=\text{[fitted value]}$ $$

These fitted parameters provide a close match between the observed data and the model and helps us understand for better decision-making in resource allocation.

## Appendix D Experimental Details

### D.1 Finetuning Settings

All experiments were conducted with parameter-efficient finetuning method LoRA Hu et al. (). For the LoRA adapter, we specified a rank of 128, an $\alpha$ value of 512, and a dropout rate of 0.1 and applied it across all attention matrices. Adding the LoRA adapter introduce minimal FLOPs overhead during training—having no impact on our FLOPS analysis—and mainly reduce memory requirements for more accessible training.

We follow standard practices in LLM finetuning Wang et al. (); Ivison et al. () and use the AdamW optimizer with beta-parameters $(\beta_{1},\beta_{2})=(0.9,0.99)$.

The learning rate is set to 2e-5 for the 7B/8B/13B models and 1e-5 for the 70B models. For data budget {2.5%,5%}, we double the learning rate to ensure convergence in loss. For all experiments, we use a warmup ratio of 0.03, BFloat16 precision, and an effective batch size of 128. For 70B model training, we used QLoRA to reduce the memory requirements and speedup the training.

For smaller data budget (2.5%-5%) experiments, we perform five trials across five random seeds. For larger data budget (10%-100%) experiments, we perform three trials across three random seeds. We report mean target task performance in our analysis. Optimization seeds are controlled through the entirety of the experiments.

### D.2 Pretrained Models

**Table 7: Base models that we finetuned in this work.**
| Base LMs | # Params | # Tokens |
| --- | --- | --- |
| Llama-2 | 6.7B | 2.0T |
| 13.0B | 2.0T |  |
| 65.2B | 2.0T |  |
| Llama-3 | 8B | 15.6T |

Table 7 lists the pre-trained models we finetuned in this work. We expect our findings to generalize to these models and future, stronger open base models.

### D.3 Training Datasets

For training, we use the same four processed datasets as in Wang et al. (); Ivison et al. (), all of which are either human-authored or annotated. Details are provided in Table 8. The FLAN V2 and COT datasets are derived from existing NLP datasets, while DOLLY and OPEN ASSISTANT 1 contain open-ended generation examples with human-written responses. These datasets vary in format, sequence length, and tasks. Following Wang et al. (), we standardize their format using the ‘Tulu’ structure.

**Table 8: Details of training dataset from Wang et al. (). Len. is short for token length.**
| Dataset | # Instance | Sourced from | # Rounds | Prompt Len. | Completion Len. |
| --- | --- | --- | --- | --- | --- |
| Flan V2 | 100,000 | NLP datasets and human-written instructions | 1 | 355.7 | 31.2 |
| CoT | 100,000 | NLP datasets and human-written CoTs | 1 | 266 | 53.2 |
| Dolly | 15,011 | Human-written from scratch | 1 | 118.1 | 91.3 |
| Open Assistant 1 | 55,668 | Human-written from scratch | 1.6 | 34.8 | 212.5 |
| Total Number of Tokens: 95.7 Million |  |  |  |  |  |

## Appendix E Results: LLAMA3

We plot additional results on target task MMLU using Llama3 8B model in Figure 7. Similar to Llama2, Llama3 8B results show that cheaper lexicon-based (BM25) and embedding-based (Embed) methods significantly outperform perplexity-based (PPL) and gradient-based (LESS) method. The marginal gains from using more sophisticated methods do not justify their selection costs.

Figure: Figure 7: Performance and Parametric Fit of Performance With Compute-Constrained Data Selection. (Left) We show all of our different runs for a given model size, where each scatter point is the final target task performance of a single run. (Right) We fit a parametric model of the performance in Equation 3 and display that as curves to pair with the empirical results as scatter points.
Refer to caption: x8.png

Figure: Figure 8: Performance and Parametric Fit of Performance With Compute-Constrained Data Selection on IFEval. (Left) We show all of our different runs for a given model size, where each scatter point is the final target task performance of a single run. (Right) We fit a parametric model of the performance in Equation 3 and display that as curves to pair with the empirical results as scatter points. We fit a pareto front in dashed line based on these optimal strategies. At medium compute budgets, cheaper data selection methods outperform PPL and LESS, which rely on model information.
Refer to caption: x9.png

## Appendix F Additional Results: IFEval

We plot additional results on target task IFEval with Llama3 13B model in Figure 8. As models scale to 13B, expensive data selection methods still underperform, despite their relative cost diminishing with larger models. Cheaper methods remain preferred.

## Appendix G Additional Analysis: Multiple Task-Specific Models.

Figure: Figure 9: Multiple Task-Specific Model Break-Even Analysis for 13B MMLU. Performance under compute-constraints reach the finetuned Pareto frontier at 5 tasks, surpassing it at 10 tasks.
Refer to caption: x10.png

Figure: Figure 10: Multiple Task-Specific Model Break-Even Analysis for 7B BBH. Performance under compute-constraints reach the finetuned Pareto frontier at 30 tasks.
Refer to caption: x11.png

Figure: Figure 11: Multiple Task-Specific Model Break-Even Analysis for 13B BBH. Performance under compute-constraints reach the finetuned Pareto frontier at 5 tasks, surpassing it at 10 tasks.
Refer to caption: x10.png

Figure: Figure 12: Multiple Task-Specific Model Break-Even Analysis for 7B IFEval. Performance under compute-constraints reach the finetuned Pareto frontier at 2 tasks, surpassing it at 5, 10 tasks.
Refer to caption: x12.png

Figure: Figure 13: Multiple Task-Specific Model Break-Even Analysis for 13B IFEval. Performance under compute-constraints reach the finetuned Pareto frontier at 2 tasks, surpassing it at 5, 10 tasks.
Refer to caption: x13.png

We perform break-even analysis by varying the number of target tasks for each method. Figure 9 presents the analysis for the 13B model on MMLU, while Figure 10 focuses on the 7B model on BBH and Figure 11 examines the 13B model on BBH. Additionally, Figure 12 and Figure 13 provide results for the 7B and 13B models on IFEval, respectively.

## Appendix H Additional Analysis: Data Similarity.

Figure 14 and Figure 15 show the Jaccard similarity of data selected by different data selection methods for the BBH and IFEval target tasks, respectively. Across various percentages of selected data, Embedding and BM25 exhibit the highest similarity. In contrast, LESS shares the least similarity with the other data selection methods.

Figure: Figure 14: Data Similarity Between Data Selection Methods for BBH.
Refer to caption: x14.png

Figure: Figure 15: Data Similarity Between Data Selection Methods for IFEval.

## Appendix I Limitations

#### Repeating Finetuning Data

In this work, we focus on finetuning the entire dataset for only one epoch. With a larger compute budget, it is possible to repeat fractions or entire datasets multiple times. Multi-epoch settings, as explored in (Xia et al., ), could potentially provide further training speedup by repeating data selectively.

#### Sensitivity to Hyperparameters

The effectiveness of finetuning can be highly sensitive to hyperparameters such as learning rate, dropout, or optimizer choice. There may be a specific learning rate that leads to quicker convergence. In this work, we fixed most hyperparameters to commonly used values for fine-tuning LLMs, leaving further exploration of hyperparameter tuning for future work.

#### Other Data Selection Methods

There are additional data selection methods not covered in this work that warrant investigation. While we focused on methods in terms of their compute efficiency, other approaches, such as classifier-based methods, could offer insights and deserve further exploration.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: The value of the strategies that depend on similarity of samples to validation samples worry me in that they seem very dependent on the size of the validation set, and that if the validation set is too small one might overfit.  But perhaps it doesn't matter too much since you are always selecting some large number of training samples anyways, and so even if the validation set is as small as Paris (to use a 2D example), you still correctly pick the subset of training samples in Europe and dump the ones in the Americas, and that wouldn't have changed much if the validation set was all of France instead of just Paris. Be great to see some discussion & experiments about this, even if they are tiny, in this paper. 

See also the weaknesses section.
soundness: 4
strengths: I found this is an important experimental contribution for practitioners and academics alike, and is likely to be heavily cited in the future. While there will inevitably be some discussion of whether they compared to all the right and best methods, I think that's in the details: they compared good and sufficiently recent example methods from high level strategies and showed significant enough differences that seem endemic to these different strategies.
summary: Surveys and experimentally compares different data selection methods for LLM fine-tuning, and reasonably and quantitatively concludes that only rather cheap methods that choose train samples based on some cheap similarity to the validation samples are likely to be worthwhile, but depends (of course) on how much training computation you are going to run.
weaknesses: The weaknesses I detail below should all be corrected, but they are all minor, none of them individually or in total would be a good reason to reject the paper. 

SECTION 3 PROBLEMS:

At the beginning of Section 3: “Our goal is to find the optimal subset S ⊆ X” pretty sure you mean subset S ⊆ D there?

I think you are implying that the train set is not necessarily IID with the validation set, but that the validation set is IID with the test set. All I see you say is that the validation set is “correlated” with the test set, which is a really  weak and vague thing to say, but if that’s all you want to say, okay, but I will be curious if in your experiments you actually make the Val and Test sets IID. 

You need to define that $T$ represents a training procedure, you just use it without defining it now. 

“By ranking the data points….” Given a large initial train set D, having to rank the datapoints at cost O(D log D) is not free, hope you guys are taking that into account. Of course, you might argue just touching all D samples is O(D), but that is less relevant if, say, we have an infinite generator of data D (e.g. a real-time reader of the datastream formerly known as Twitter) and an independent (non-ranking) decider of whether each incoming $x$ is worth training on, that is, we shouldn’t have to assume we need to sort at cost O(D log D).


I’m uncomfortable as a reader that in (2) you are still defining your objective in terms of the test set. I agree that’s the ultimate goal, but if you actually implemented (2) it assumes knowledge of the test set. By the time you get to (2), I expected you to have switched to the validation set in the stated objective, which is different than the final metric, which should of course than be on the test set. 


SECTION 4 FEEDBACK: 
You can cut some of the intro to Sec 4, but please add-in that Lexicon-based and Embedding-based are both strategies that try to select train samples that are similar to the validation samples, whereas Perplexity and Gradient solutions are optimizing for the effect on the model loss.

SECTION 5 FEEDBACK:
Why do you assume training on all x is equal? Is that really true (honest question)? My guess is yes due to the very beaurocratic nature of how these models are run, but that’s not always true of machine-learned models, for example, a classic decision tree is much faster to evaluate for some inputs than others (if it has leaves of varying depths). 

In computing C(k), you sum over C_v(x), which I assume is for x \in D? Please be explicit there about which x you are summing over.  And I’m surprised that that cost does depend on $x$ Does C_v(x) really vary so much? Could that not just be \|D\| (= size of D) times some cost per training sample?  


RANDOM: Really appreciate you comparing to just a random sample as a baseline. 

MINOR BUT IMPORTANT QUIBBLES: 
Authors state too unequivocally: “in practice, the total compute budget is
predetermined: the number of accelerators and their usage hours are allocated in advanced”.   That certainly is NOT true in many large companies that are actively training and leading with LLMs. So please hedge and preface that sentence with “In many cases,”. 


This sentence didn’t make sense to me:
“For example work on parameter efficient fine-tuning, targets improving the
memory-usage of this stage (Hu et al., 2021).”

TYPO “create an minimal” -> “a minimal”

Since you are citing John 1975, consider  also citing the famous and foundational Hart 1968 paper on Condensed Nearest Neighbors, the canonical and classic approach for selecting data for training (summarized e.g. in wikipedia: https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm). However, nearest neighbors is such a different machine learning paradigm that if you don’t feel that adds value to this work, you can ignore this suggestion, but given the statement in your paper is “Data
selection is a foundational approach in machine learning where the objective is to create an minimal
dataset from a collection of data” the Hart 1968 paper is exactly the classic paper to have done just that.

TYPO “Data selection takes the full training data as input and
chooses a subset to the train” -> “to train”

This sentence doesn’t quite parse, please re-write “Instruction-tuned models can handle a variety of possible inputs for downstream use cases as
either classification or generative model”

This sentence needs some polishing of singular vs plural:
“Therefore
while instruction-tuning is not the direct focus of this work, it provide a real-world applications of
compute-constrained data selection.”


“Assuming we at minimal” -> “at minimum”

Equation (2) can be written all on one line, winning you a bit of precious space. 

In 4.1 you refer to Section 4.1, which is a bit weird and really you can just delete that whole sentence. 

Do figure out how to bold the last column title C_forward in Table 1. It can be done (probably \mathbf{}). 

TYPO: Fit of Compute-Performace Relationship  -> Performance

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Could the author discuss a real-world scenario to demonstrate how the proposed methods could be applied to guide practitioners?
2. Are the studied methods sensitive to the choice of model architecture?
3. How do these methods scale with hardware improvements?
soundness: 3
strengths: This paper addresses compute-efficient fine-tuning, which is an important task in training LLM. Extensive simulations are conducted to provide empirical evidence and support the framework.
summary: This paper studies a framework considering the practical challenges of training and fine-tuning large language models (LLMs) under computational constraints. It has established a trade-off between achieving better performance using larger data and lowering computational costs by selecting smaller subsets of data. A key takeaway is that simpler data selection methods, such as lexicon-based and embedding-based approaches, often provide a more efficient solution compared with more complex, compute-intensive methods like perplexity-based and gradient-based strategies.
weaknesses: 1. Although the author claims some simple methods such as Lexicon outperform the complex ones such as Perplexity and Gradient, as shown in Figure 1, the complex ones perform quite well especially under medium and large budget situations. It would be more important to study the tipping point, where the performance gains plateau became flat. This is the place where further increases in computing resources yield diminishing returns. 
2. It is not surprising to see the tradeoff between performance and data size. The conclusions in this paper are largely empirical and may not generalize well to other situations. The practical limit of parametric fit is limited, as it mainly fits observed data points without clear guidance on how to *extrapolate* results to new scenarios. For example, can the results from smaller models (e.g., 13B) be used to predict outcomes for larger models (e.g., 70B)? Can the parameters estimated from smaller models be reliably transferred for larger model? If practitioners need to run experiments on 70B models to obtain these insights and fit the parametric model, the results may not be useful.

---

confidence: 3
contribution: 1
initial_rating: 5
presentation: 2
questions: See the weaknesses part.
soundness: 2
strengths: 1. This paper considers an interesting problem, data selection under computational constraints, and has interesting observations that the initial cost cannot be neglected when considering the computational budget.
summary: This paper considers selecting data for finetuning LLMs under a computational budget. The computational cost is divided into two parts: 1) when using the validation set to evaluate the performance, the validation set will incur an initial cost; 2) training on each sample will cost a fixed amount of computation. The authors propose an exponential distribution model to fit the model performance v.s. the training costs for four types of data selection methods: lexicon-based, embedding-based, perplexity-based, and gradient-based. The paper consists of numerical experiments over several models and several tasks.
weaknesses: 1. (Major) Lack of novelty: although this paper proposes a framework for analyzing the computational cost of each data selection method, it does not provide any new techniques based on this framework. Furthermore, the key observation is not very surprising: the computational cost contains an initial cost when evaluating the validation set, thus the perplexity-based or the gradient-based is clearly not optimal under a limited compute budget.
2. (Major) Lack of soundness: a) the parametric model is selected to be an exponential distribution and the model is fitted to minimize the squared error, but the choice is never justified by any theoretical analysis or numerical results. The fitted curve is also not very convincing (e.g. Figure 3 and Figure 7). b) The Pareto frontier is never formally defined in this paper nor sufficiently discussed. It's very hard for me to believe that the fitted Pareto curve is indeed Pareto optimal as the points in Figure 8 and Figure 10 exceed the Pareto frontier by a large margin. Also, the fitted exponential curves surpass the fitted Pareto curve considerably in Figure 3, implying that the two curves even contradict each other.
3. (Moderate) Lack of insights: this paper is rather an experiment report than a well-motivated paper. The motivation for studying such a computation-constrained data selection problem is not fully supported. The authors just launch a bunch of models, adopt several tasks, and collect all the results without providing sufficient analyses.
4. (Moderate) The style file seems to not follow ICLR 2025 templates: the lines are not numbered.
5. (Minor) Typo(s):
  Figure 1 "using a much larger base model then data selection model": "then" should be "than".

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - Regarding the notion of utility in Section 3: Is utility here something that is to be minimized, i.e. alternatives with lower utility are preferred over alternatives with higher utility? In the remainder of the paper (expected) model performance is considered for which clearly higher values are preferred.

- I am not sure whether I understood the greedy data selection introduced in Sections 3 and 4. I am under the impression that all data points are scored individually and afterwards they are ranked according to their scores and selected until budget K is exhausted. Isn't it neccesary to do this in an interleaved manner, in order to capture effects like redundancy and the submodularity of utility? Consider the extreme case in which the most informative data point x is repeated K in the dataset, then we would end up with a selection that contains the same datapoint K times.   

- In Figure 2, the plot represents performance of Mid-PPL, while the plot in Figure 3 represents performance of Top-PPL. What is the reason for this discreapency?

- In Figure 2, what exactly is the dashed line? Shouldn't the Pareto front contain all solutions, that are dominating on the dimensions of compute (low) and accuracy (high)? The line is straight, is it something like a linear regression applied to the solutions on the Pareto front?
soundness: 2
strengths: - The study is well motivated and its results are of practical importance for finetuning large language models.
- Empirical findings correspond well with theoretical framework
summary: The authors present a study on compute constrained data selection for training large language models (LLMs).
Unlike preceding works, they do not constrain the size of the training set, but the compute, which is the sum of computational expenditure for data selection as well as LLM training. 
They compare various existing data selection methods under this setting and come to the conclusion that many powerful data selection methods are almost never compute-optimal due to their computational cost, making cheaper data selection the favorable choice.
weaknesses: - The title of the paper is rather broad, while the study is rather specific. "Compute-Constrained Data Selection" does not indicate which type of data is selected for which type of task.

Minor remarks:
- p.3 line 3: \mathcal{S} \subseteq \mathcal{D}, as \mathcal{X} is not introduced
- p. 6 bottom: methods -> method

Meta-review: The paper studies the cost of data selection by formalising the trade-off between data selection cost and training gain. The authors validate their insights experimentally. They show for instance that one has to pay attention to the computational cost of sophisticated model selection techniques and that simpler (and cheaper) methods might be preferable in practice. Authors revised the paper and appendices based on the feedback of the reviewers, addressing their comments and providing additional experimental evidence.

Decision: Accept (Poster)