Compute-Constrained Data Selection
Authors: Junjie Yin
Alexander M Rush
Venue: ICLR 2025 Poster (2025)
Keywords: Data Selection
Scaling Laws
Compute-constrained
Compute-optimal Training.
Primary area: foundation or frontier models, including LLMs
TL;DR: Post-training data selection in a scaling laws framework.

Abstract: Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the setting in which both the cost of selecting data and training are budgeted for. We first formalize the problem of data selection with a cost-aware utility function, 
and model the data selection problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute. Interestingly we find that many powerful data selection methods are almost never compute-optimal, and that cheaper data selection alternatives dominate both from a theoretical and empirical perspective. For compute-optimal training, we find that perplexity and gradient data selection require training-to-selection model size ratios of 5x and 10x, respectively.

Published as a conference paper at ICLR 2025
DATA SELECTION
Junjie Oscar Yin Johns Hopkins University dyin270 jhu edu
Alexander M. Rush Comell University
arush@cornell.eet
ABSTRACT
Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the set- ting in which both the cost of selecting data and training are budgeted for. We first formalize the problem of data selection with a cost-aware utility function, and model the data selection problem as trading off initial-selection cost for training gain. We run comprehensive sweep of experiments across multiple tasks, vary- ing compute budget by scaling finetuning tokens, model sizes, and data selection compute. Interestingly we find that many powerful data selection methods are al- most never compute-optimal, and that cheaper data selection alternatives dominate both from theoretical and empirical perspective. For compute-optinge training, we find that perplexity and gradient data selection require soaning-t--selcctonn model size ratios of 5x and 10x, respectively.
INTRODUCTION
The growth of large language models (LLMs) has motivated research into their resource profiles. The compute cost of training LLMs is substantial, and, in many cases, the total compute budget is predetermined: the number of accelerators and their usage hours are allocated in advanced. Thus, is critical to determine the optimal allocation of resources under budget constraint. Past work on compute-optimal LLMs (Hoffmann al. 2022) studies one could attain better perplexity for a given pretraining compute budget by balancing architecture and training decisions.
Similar resource questions exist during post-training finetuning of LLMs. We assume a common setting where single-base LLM needs to be finetuned for downstream task. Numerous works have sought to induce certain abilities by training from large instruction tuning datasets (Sanh et al. 2021 Wei et al., 2021 Mishra et al. 2021). There are several different resource constraints which might make this challenging. For example, parameter-efficien fine-tuning methods like LoRA (Hu al. 2021) aim to reduce memory usage during finetuning by updating only small subset of the model's parameters. In this work we focus instead on compute-consttraieed finetuning.
A promising approach reducing compute requirements for finetuning is data selection. Data selection is foundational approach in machine learning where the objective is to create a minimal dataset from collection of data (Hart] 1968 John 1975). Given the large computational cost of each gradient step, reducing dataset size is an appealing way to reduce this resource usage. Moreover, as larger and more diverse instruction tuning collections become available, likely only subset of the data provides the most value for any given task. Recent work has shown that careful data selection can vastly increase the effectiveness of finetuning per step (Chen et al. 2023 Zhou et al. 2024).
Yet, even if data selection is effective, it does not priori imply that it is compute-optimal. Given the base-level compute effectiveness of gradient descent on L.M models, data selection methods need improve upon standard training in proportion to their added cost. In other words, compute- optimal method should both improve training and be cheap to compute. In this work, we study this setting of compute-coe constraied data selection, and argue that this is critical factor for practical adoption that is being under considered in method development.
Concretely, we aim quantify the trade off between model size, number of tokens, and data selec- tion in LLM finetuning, such that practitioners can make well-informed decisions when choosing how best allocate compute. We first formalize this problem as compute-constrained combi-

---

Published as a conference paper at ICLR 2025
natorial optimization problem and then discuss compute-awan modification. We develop an categorization of different approaches for this task and model their compute scaling. This is used to frame a broad series of empirical training runs under varying compute constraints, scaling both model sizes and training length. We train over 600 models, ranging from 7 to 70 billion parameters, across 6 data selection methods and 3 downstream tasks, recording final task performances for each.
The results from this study argue that complex data selection methods are almost never Pareto- optimal in the compute-constrained setting and that simple statistical methods such as sparse re- trieval should be preferred. In particular, powerful data selection methods that use model perplexity or gradient information tend be FLOP inefficient both from a theoretical and empirical perspec- tive. That is not to say these methods are ineffective; for example, they should be used in settings with repeated training with different tasks on the same underlying models. We parameteric mod- manner. Using these
els to quantify the effectiveness of various approaches in a compute-awar fits, we extrapolate and empirically validate that, for comptue-optimal finetuning, perplexity and gradient data selection require caining-to-selection model size ratios of 5x and 10x, respectively.
We hope that this framework and setting can motivate further research into cheaper data selection methods that can produce better models with less compute. Codebase and datasets to reproduce our results are available at
RELATED WORK
Compute Scaling for Model Size and Transfer Learning. Kaplan et al. (2020) established the use of compute-scaling laws for language models and showed a power-law relationship between model size and loss over varying orders of magnitude. More recent works expand on the original formulations by considering learning rate schedule matching, multiple-epoch training, and hyperpa-
rameters (Hoffmann et al. 2022 Muennighoft et al., 2024: Bi et al. 2024).
Hernandez et al. (202) study scaling in the post-training setting but only model the relationship between pretraining and finetuning data loss. Similar work (Lin et al., 2024 Isik et al,, 2024] Zhang et al. 2024) studies the scaling in post-training by modeling the relation between pretraining data size, model size, finetuning method and downstream test loss. These models do not consider the use of data selection. Recent work considers the relationship between data selection and scaling in the vision domain (Goyal et al. 2024), but this work does not consider the compute needed for data selection in their scaling analysis.
Data Selection for Language Models. Data selection takes the full training data as input and chooses subset to train (Albalak et al., 2024 It can be viewed as a coreset selection problem (Mirzasoleiman et al. 2020; Killamsetty et al. al. 2001b), which aims to select a subset from the given training dataset such that the model achieves performance similar to the full dataset.
There are many different approaches to LLM data selection. The simplest are non-model specific approaches such as manual scoring functions (Chen et al., 20244,, surface level features (Robertson et al. 2009), and n-gram features (Xie et al. 2023). On the other hand, more effective methods use LLMs to assign utility. One class uses model forward inference information such as utility scores from generations, model embeddings, and perplexity (Wettig et al.) 2024 Marion et al. 2023 Ivison
ct al. 2022). Another class uses model gradients to define influence function style selections (Kil-
lamsetty et al. 202 Han
2023
Xia et al.) 2024). See ReefSection for further description.
Task-Specific Finetuning from General- Instruction Datasets. While data selection can be applied in any finetuning setting, it is most impactful as method to train targeted model from a general-purpose dataset. In the case of LLMss, this setting is commonly training task- specific
model from an "instruction-tuning" dataset (Sanh et al. 1 Wei et al. 2021 Mishra et al.) 202
Instruction-tune models are effective for many downstream tasks; however they require training on very large and expensive set of data. To get around this issue, models have demonstrated strong results using small subsets of instruction tuning data (Chen et al. 2023 Lu et al. 2023 Zhou et al. 2024). As datasets grow, automated measures of quality selection has become a growing focus, particularly when many targeted models are needed. Therefore while instruction-tuniig is not the direct focus of this work, it provide a real-world applications of compute-constrained data selection.
2

---

Published as a conference paper at ICLR 2025
3 BACKGROUND
The goal of data selection is to choose a subset of data points from a large dataset to optimize model performance on target task. learning task, we are given large training set, a target test dataset, T, and a validation set, V. Our goal is to find the optimal subset S S D such that the model (S) trained on S maximizes the performance on under a given data constraint:
S arg max P(T:T(S)) SCD subject to
| S K,
where denotes the performance of the model on the test set and K is the max cardinality.
This problem is challenging solve in the general case, particularly without access to the test set T. Approaches to the problem therefore commonly make two implicit assumptions: (1) the performance function, P(T:T(S)) is monotonic and submodular, non-increasing marginal utility, in the dataset chosen (Kirchhoff & Bilmes] 2014), and (2) the validation set V is IID with the test T. Under these assumptions, we can argue for greedy data selection approach (Kirchhoff & Bilmes 2014). This allows decomposing the total objective, P(T:T(S)), by considering the contribution of individual training points to the performance on the validation set P(V; ({())) for each x E S.
To estimate the marginal contribution of x S, most data selection methods use utility func- tion v(æ; V/) proxy to P(V; ()2;;((=))))- give the utility of each data point based its relevance (Albalak et al. 2024). By ranking the data points based on e and selecting those that maximize the total utility within the data budget K, greedy data selection aims obtain high- performing subset S To summarize, we consider data selection methods that target Equation (1) with two-step greedy algorithm: score all points and then select points up the budget K.
COMPUTE-CONSTRAING DATA SELECTION
While the framework presented in Section provides general method for data selection, we argue that it is insufficient for the practical challenge of finetuning LLMs. The issue is that LLM fine- tuning often bottlenecked by a computational budget and not data budget. There are two major computational bottlenecks in this process: (1) the cost of training the model this data (Cr), and (2) the cost of computing the utility function on this data (C.).. The true cost of C2 can reduce significantly the amount of training points we select for given computational budget.
Assuming we at minimum require the computation of a utility function over the dataset, we can define the compute-constoe conttreeco data selection objective
= arg max P(V:T(S)) SCD
subject to Cr(s) + Σ Colo K.
RED
Here K is now the compute, e.g. maximum number of FLOPs, allocated for data selection and training, and we assume calculation of a fixed-cost independent of optimization.
4.1 COMPUTE CoST OF DATA SELECTION UTILITIES
To make these costs more tangible, we consider four classes of data selection this work, that represent three different levels of compute. This section summarizes their main properties, i.e. their core utility functions and computational costs.
Lexicon-Basse Lexicon data selection methods utilize statistical properties text to evaluate the relevance of data points without relying deep learning models. One of the most effective lexicon- based methods BM25 (Robertson et al, 2009 Silva & Barbosa 2024), which scores data points based the frequency of terms. The utility function (a; assigns relevance score each
3

---

where Crorward (x) is the cost of a forward pass for (++) on the model M, and Cgraa (V) is the cost of computing Vo((V) 0'), computed once for all æ. Computing gradients involves both forward and backward passes, totaling approximately three times the cost of forward pass. Low-rank Published as a conference paper at ICLR 2025
Method
Utility Function
Computational Cost
Cforward
Lexicon-Baie Based
BM25(x, a')
CBM25 + [x]))
2 0
Embeddingg-asset
MV Σ cos(Emb(r)) Bmb(x'))
Cembed (2) + Cembed (2)
Perplexity-Ban -Base
PPLay(2) (x)
Ciarward (x)
R
Gradient-Bar -Based
(Vol(z; 0*), Vol(V;0))
3 x Crorward (z) + Cgrad (V)
R
Table 1: Utility Functions and Computational Costs for Data Selection Methods.
data point by averaging the BM25 scores with the validation, UBM25(1) = [V] Ex'er BM25 x"). Since the algorithm can be run with a single-core cpu, the data selection FLOPs are almost 0.
These methods utilize embedding models to select data points that are most similar to the target data (Rubin et al. 2021). The utility function Uretrieall (a; V) assigns a score to each data point based its cosine similarity with validation data in V; that is, Vretrieval [VT Ev'ey cos (Emb(x), Emb(x")). Assuming we are using a very small model (on the order of BERT-size) and the embedding requires only one-time transformation of data points into dense vectors, the data selection FLOPs are quite small. [iu al. (2021) demonstrated significant per- formance gains when selecting in-context examples. We apply a similar method, Embed, to the fine-tuning setting using small T5-based dense embedding model (Ni et al., 2021).
Embedding-Basse
Perplexity-Based. Perplexity-based data selection utilizes language models to evaluate the utility of data points based on model loss (Antonello al., 2020). The utility function "ppi(2; V) assigns score to each data point by computing the perplexity (PPL) of x under language model By finetuned on V: that is, (x) = PPLo, Top-PPL and Mid-PPL have both shown improved performance and training efficiency (Ankner et al., 2024; Marion et al. 2023), where Top-PPL ranks data points with the highest perplexity scores, and Mid-PPL does the same for points in the middle of the score distribution.
Gradient-Basee
These methods evaluate the utility of data points based on their influence on the model's 's loss with respect the target data (Pruthi al. 2020). The utility function Ugrad (z; V) quan- tifies this influence by computing the inner product between the gradient of the loss on x and the gra- dient of the loss on V, scaled by the learning rate mı; that is, (z) = Te { (Vol(a; 8*), Vol(V; 8*)), where V 8*) computational cost of computing Ugrna (1)) for each x is approximated by:
1 Evev Vel(x'; 0²), and H are the model parameters at time step The
[VT
Orgne (x)) Chackward (x:) + Cgrad (V) R 3 Cforwana(r) + Cgma(V)),
sgradiEnt Similarity Search (LESS) shows superior performance gain over cheaper methods and random selection (Xia et al. 2024).
While lexicon and embedding- based methods aim to select training samples similar to validation samples, perplexity and gradient-based methods focus optimizing their effect model loss. Different implementation of these general categories leads to different cost. A detailed cost analysis of our selected data selection methods can be found in the Appendix
MODELING THE COMPUTE-PERFORMANE
RELATIONSHIP
To analyze the trade-off between the compute of data selection methods and the expected gain in model performance, we define a simplified parametric form for expected peformance. Let e be the fixed-cost of training on a single data point and i |S| be the number of data points. Define C(k) as the total cost of training and selection,
C(k) = CX K ++ Σ Co(e)
x
4

---

Published as a conference paper at ICLR 2025
Small 4.5E+10
Medium Budget 4.5E+11
Budget. 9
80
Performenonn
Perfomaazo
60
20
10
10
ALT
10
10
10"
Comgute (PLO7S)
Compute (PLOPS)
Compute (FLOPS)
Data Selection Methods
Random
Lexicon
Perplexity
Gradient
Figure I: Simulation of Performance under Constraints. P(k) = Px (1
(1-exp exp C(P C(k
The behavior of different data selection methods using our performance model. (Left-Small Budger) The Lexicon method may consistently outperform more advanced data selection methods if their initial cost is too high. Under our assumptions gradient can never be optimal as its cost exceeds epoch of training. (Middle-Medium Budget) The perplexity method can become optimal once the total cost exceeds a given amount. (Right-Large Budget) The gradient methods can be optimal training is more expensive than the fixed-cost, for example if using much larger base model than data selection model. The simulation shows that the compute-optima data selection method changes function of the compute budget and the performance rate associated with each method.
The assumption in data selection is that more compute intensive methods are able to achieve higher performance with less samples, but that the value of information gained from selecting additional data points diminishes more points are explored. We will assume that data points exchange- able, and that all methods eventually reach the same upper-bound performance after epoch, P P(T;
Our parametric model for the expected performance after training on data points should capture diminishing returns, dependence computational cost of the method, and convergence toward an upper bound. We model this function as:
P(k) (P (P- Po) x (1- (1-exp exp
C(k) (->
+ Po
(3)
where Po the zero-shot performance, P is the upper bound and À is the value the method extracts from additional compute in the utility function.
Simulation Figure presents a simulation of this performance function across different methods under varying compute constraints. We set c to {4.5E+10, 4.5E+11, 9E+12), 1 & to 100M, and utility compute to (4,5E+10, 4.5E+11, 9E+12). The parameter À is set to {5, 10, 40, 80) for the Random, Lexicon, Perplexity, and Gradient methods, respectively. These settings demonstrate how different data selection methods may become compute-optimal under varying compute budgets. Note that for Medium and Large Budgets, we use Small-size model for PPL and Gradient.
Using methodology similar to Hoffmann et al. 2022 the parameter À can be fit on the empirical measurements, which yields data-driven estimates. See Appendix C.I for more details on derivations and the fitting procedure.
6 EXPERIMENTAL SETUP
experiments vary the number of finetuning tokens for 5 data-selection methods and a fixed family of models, ranging from 7B to 70B parameters. The finetuning data budget is fixed as percentage of the total finetuning tokens: 52.5, 10, 25, 50, 100} %, across target tasks. For each finetuning budget, we conduct multiple training runs with increasing compute, which is either allocated toward larger pre-trained model sizes or sophisticated data selection methods.
As shown in Table 2
5

---

Published as a conference paper at ICLR 2025
Training Data
Data Selection Method
Model Size
Target Task
2.5%
Random
LLAMA2 7B
MMLU
5%
BM25
LLAMA3 8B
BBH
10%
Embed
LLAMA2 13B
IFEval
25%
PPL
LLAMA2 70B
50%
LESS
100%
Table 2: Experimental Setup Overview.
We analyze each FLOP count to identify which runs achieve the highest performance on target benchmark. We then fit power law to obtain finetuned Pareto frontier for each model sizes.
Datasets We follow Wang et al. 2023) and curate representative sample of instruction-tun cunnte datasets as listed in Table This includes: (1) datasets generated by researchers from existing NLP datasets, such COT Wei al. 2022) and Flan V2 (Longpre et al. (023) datasets written by humans from scratch specifically for instruction tuning, including Dolly (Conover et al. 2023 and
Open Assistant (Köpf et al. 2024).
For evaluating the model we run on three challenging but different downstream tasks. These in- clude: the Massive Multitask Language Understanding (MMLU, [Hendrycks al. (2020)) dataset measures models' factual knowledge, comprising questions across 57 subjects, spanning difficulty levels from elementary to professional; Big-Bench-Hard (BBH, (Suzgun
(2022) curates 23
complex reasoning tasks from Big-Ber Big-Bench Srivastava et al. (2022). It is used to evaluate models' general reasoning capabilities; and Instruction Following Evaluation (IFEval, [Zhou al.] (2023), which evaluates models' instruction following abilities. For MMLU, report 5-shot accuracy; for BBH, we report 3-shot exact match score; and for IFEval, we report O-shot accuracy.
Pretrained Models For all experiments. chitecture and tokenizer (Touvron et al., 2023 Dubey et al., 2024). Models range from 7B pa- rameters to 70B parameters, and are trained for close 100 million total fine-tuning tokens. Our experiments closely follow prior work on training and evaluating instruction-tuned models (Wang) Ivison al. 2023).
train transformer language models with LLAMA
al. 2023
We primarily focus on the LLAMA-2 model suite, containing LLAMA-2-7B, LLAMA-2-13B. and LLAMA-2-705 Because the latest LLAMA-3 model suit only contains two model sizes (8B and 70B), we believe the three LLAMA-2 models are better suited in modelling the scaling behaviour. Nevertheless, We show that our results generalizes well to LLAMA-3 by experimenting with smaller base model LLAMA-3-8B.
RESULTS
Empirical Results. Figure shows the full results with data selection methods across pre- trained model size and 2 of the target task. For these experiments, the unique training data budget is fixed at roughly (2.5%, 5%, 10%, 25% 50%, 100%) } of tokens. For each data budget, we finetune set of models with increasing amount of compute that is allocated to either more parameters or more expensive data selection methods.
Note that for PPL and Gradient, 7B model of the same model family is always used for data selection; whereas for Embed small encoder model used. For MMLU, PPL is implemented as Mid-PPL. For BBH, PPL is implemented as Top-PPL.
The fine-tuned efficient Pareto frontier comprises all runs that are Pareto-optimal with respect to compute (X-axis) and performance (y-axis). These runs represent the most efficient choices, provid- ing the best possible performance for a given compute budget under specific data selection methods and training token lengths. Furthermore, model the efficient computational frontier using a power law, specifically of the form P(C) = a (og(C)) + b, where and are parameters fitted to the data. The fitted function is depicted dashed gray line in the figures.

---

Published as a conference paper at ICLR 2025
(A) 7B parameters
(8) 138 parameters
(C) 70B parameters
47.5
47.0
55.0
Rtaraaed Farats Froo:
The Parel Frefes
MbJU Accurscy
51.5
75
31.32
46.0
51.5
2.22
630-
21.02
86.5
00.72
520
88.53
10ft
13tt Compute (FLOPs)
Compute (FLDPs)
Compute (FLOPs)
(D) 7B parameters
(F) 13B paramelars
(F) 70B parameters
From
71.3
coaed Penata Print
20.5-
211t Aoc
41.2
435
70.2
46
40.5
2-
09.2
10l6 Compute (FLOP))
Compute (FLOP4)
Computo (FLOP4)
Selection Methods
Random
BM25
Embed
PPL
LESS
Figure 2: Performance for Different Data Selection Methods. We show all of our different runs for a given model size, where each scatter point is the final target task performance of single run. (A, B, C) show MMLU results across three model sizes, while (D, F) present BBH results across three model sizes. For each run, we determine the optimal finetuning strategy combination of data selection method and number of finetuning tokens-that achieves the highest performance un- der a particular FLOPs budget. We fit a pareto front in dashed line based on these optimal strategies, which is a line in the linear-log space, At small and medium compute budgets (A, B, D, E), cheaper data selection methods like BM25 and EMBED outperform PPL and LESS, which rely model information. At larger compute budgets (C, F), however, PPL and LESS become compute-optimal after using 5% of the fine-tuning tokens.
The main 7B results in Figure 2 (A, D) and Figure 5 (b), show that cheap lexicon-based methods (BM25) and embedding- based methods (Embed) significantly outperform perplexity-based (PPL) and gradient-based methods (LESS). While PPL and LESS achieve better performance at the same data budget compared to these methods (see Figure 5), they are not compute optimal under the same compute budget due to the high FLOPs required for data selection. The marginal benefit one can get from using a more sophisticated data selection methods does not outweigh its cost in selecting these data. Additional results can be found in Appendix
As models scale to 13B Figure (B. E), we still find that expensive data selection methods under- perform, even though, the relative cost of doing data selection for these methods diminishes as the training model size increases. As shown in Figure (B, E), at 13B model sizes, cheaper methods are still preferred. We do see that after 5% finetuning tokens, PPL is more competitive, outperforming Embed in MMLU, and almost matching the pareto front at 25% finetuning tokens.
At the 70B model size, shown in Figure (C, F), PPL and LESS outperform both BM25 and Embed finetuning tokens for the first time. This suggests that at very large compute budgets, more sophisti- cated and costly methods can gain greater advantage compared to lexical and embedding methods. As model sizes continue to scale and LLM sizes shrink, these methods may become more efficient.
Results on LLAMA3 8B are nearly identical to the 7B we see for LLAMA2. This verify that the approach is not model specific. We include full results in Appenddix E
7

---

Published as a conference paper at ICLR 2025
(A) 7B parameters
138 parameters
(C) 708
85.5
70.0
55.9
68.8
47.0
54.5
56.0-
69.4
66.5
63.5
88.0
69.0
52.5
02.0
68.6
68.0
16f9 Computa
Compute
Compute (FLO7))
(D) 7B meter
138 parameters
(F) 70B parameters
52
4.5
50-
70.5
48
70.0
46
40.0
44.
69.0
10
1819
Compute (FLOP)
Compute (FL07%)
Compute
Models Trained
Parametric Fil
Random
BM25
Embed
PPL
LESS
Figure 3: Parametric Fit of Performance with a parametric model of the performance in Equation (3) and display that curves pair with the empirical results as scatter points. (A. B, C) show MMLU results and their parametric fit across three model sizes, while (D, F) present BBH results and their parametric fit across three model sizes.
Contte--ndr Data Selection. We fit
Fit of Compute-Performane Relationship. In Section 5 we propose parameteric model for the relationship of data selection compute to model performance, [Equation (3) This formula assumed that data selection diminish in utility and that performance is direct function of compute. To verify these assumption, we fit the parametic form to the the empirical curves by fitting the parameter and allowing tolerance in Po. per method and dataset. We model all final performances from experiments as a parametric function of the compute budget P(k).
Extrapolation from Parametrie Fits. Figure 3 shows the fitted curves for each method. Since the shape these curves dataset and model dependent, we cannot predict exact performance for different levels of data selection compute. However, the close fit of the parametric models obtained from smaller models allows us to estimate the compute-optimal ratio between the training model size and the selection model size. For perplexity- suggests that the method becomes compute-optimal when the training model is 5x larger than the data selection model-around 35B parameters. For gradient-based data selection, our extrapolation indicates that the training model needs to be approximately 10x larger than the data selection model be compute-optimal-are around 70B parameters. Appendix presents our results and details the extrapolation.
data selection, our extrapolation
ANALYSIS
Comparing Training versus Total Compute Budgeting. While our primary interest is in the full compute constrained setting. we note that different results hold targeting only a small training budget in EEquatinn (1) When the training- is fixed, observe in Figure 5 (a) that the gradient-basse based method (LESS) consistently outperforms other data selection methods, with the perplexity-based method (PPL.) being the closest competitor.
8

---

Published as a conference paper at ICLR 2025
Number of Tasks =
Number of Thiks 5
Number at Thsks = 10
Number of Thaks 20
0.5
6356
0158
43.0
06:55
46.5
16.33
10400
0.5
60.00
40
12F
1D1 Compate (FLOPs)
Compute IFLOPS)
Compuse (FLOPs)
Conpute IFLOPN
Fiae taned Pareto Pront
LESS
Figure 4: Multiple Task-Specific Model Break-Even Analysis Costs to perform gradient-base method (LESS) are spread over all the target tasks. Performance under compute-constrains reach the finetuned Pareto frontier 10 tasks, surpassing it at 20 tasks.
LLAMAZ
ILAMA2
Fixed Train Budget
32
32
MMMUU Ascurasy
122rr Morumayy
20
lered Preer
30.
46.5
26
46.0
24
40.5
22-
22
45.0
20
20
Training Compute (FLOPs)
10
10
Compute
Comp (PLOP))
Methods
Random
Endod
LESS
Models Truined Purometite Fit
Isterpllation
BN2S Embed
PPL LESS
8325
7PS
Randon
(b)
Figure 5: (a) Fixed Training Budget. Considering only training budget, sophisticated methods consistently outperforms cheaper methods. (b) Performance and Parametric Fit on IFEval. At small compute budget, sophisticated methods are not compute optimal.
2.5% Selected
Selected
50% Selected
100
42%
14%
03%
12%
53%
57%
52%
PRL
1.3
211
A1
20%
11.4
15.3
47%
28
43%
100
-2%
Lasa
BACO
IP
LEGS
BMSS
BaRa
Figure 6: Data Similarity Between Data Selection Methods for MMLU.
Multiple Task-Specific Models. Another interesting setting is when the goal is to train multiple models from the same large training set, where each targets a different task. this setting, the gradient information calculation can be amortized between target tasks. However, even in this setting the cost of gradient calculation is still quite high. As shown in Figure 4 with LLAMA2 7B as the data selection model, LESS needs 20 tasks to surpass the finetuned pareto frontier. We perform similar analysis for other model sizes in AApendix H!
Data Similarity. Figure presents the Jaceard similarity of data selected by different data selec- tion methods. We observe that BM25 and Embed tend to select highly similar data, which explains their comparable performance. In contrast, the data selected by LESS bears the least resemblance to the other methods. We perform similar analysis for different target tasks in Appendix

---

Published as a conference paper at ICLR 2025
9 CONCLUSION
Data selection is valuable tool for fine-tuning language models, but its primary benefit - saving compute must be balanced against the compute costs required to identify optimal dataset. In practice, popular methods are surprisingly compute intensive, yielding training data reduction at the cost of worse search into more compute-efficient data selection methods. As demonstrated by our 70B finetuning experiments, sophisticated data selection methods can leverage smaller models for data selection, allowing larger models from the same family to be trained more efficiently in the LLM setting.
constraind performance. We hope that these results motivate further re-
10 ACKNOWLEDGMENTS
We would like thank Woojeong Kim and Celine Lee for their insightful feedback the paper's writing. We are grateful to Mengzhou Xia for assistance with setting up the codebase and valuable discussions on data selection. Additionally, we thank Junxiong Wang, Jing Nathan Yan, and Wenting Zhao for their support in conducting the early experiments. This work was supported by NSF IIS- 1901030 and NSF CAREER 2037519.
REFERENCES
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey data selection for language models. arXiv preprint arXiv:2402. 16827, 2024.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew I. Leavitt, and Man- sheej Paul. Perplexed by perplexity: Perplexity-basee data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024.
Richard Antonello, Nicole Beckage, Javier Turek, and Alexander Huth. Selecting informative con- texts improves language model finetuning. arXiv preprint arXiv:2005.00/73 2020.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0295: 022.0.0.9944 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.08700 2023.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 11m. Company Blog of Databricks, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-dal data curation cannot be compute agnostic. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22702-22711 2024.
Xiaochuang Han. In-context alignment: Chat with vanilla language models before fine-tuning. arXiv preprint arXiv:2308. 04270023008..755,, 2023.
Peter Hart. The condensed nearest neighbor rule (corresp.). JEEE transactions on information theory, 14(3):515-516, 1968.
10

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. S0KV::2000003300. 2020.
arXiv preprint
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprini arXiv:2102.01293 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models, arXiv preprint arXiv:2203. 15556, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint (rxXi::2402.01177,, 2024.
Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. arXiv preprint arXiv:2212.00116, 2022.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing cli- mate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702 2023.
RC St John. D-optimality for regression designs: a review. Technometrics, 17(1):15-23, 1975.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8110-8118. 2021b.
Katrin Kirchhoff and Jeff Bilmes. Submodularity for data selection in machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 131--11,, 2014.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations demrrratiiddmm large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Haowei Lin, Baizhou Huang. Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model fine-tune via rectified scaling law. arXiv preprint 2rXiv:2402.0233 2024.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2 1.6804. 2021.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648. PMLR, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference Learning Representations, 2023.
11

---

Published as a conference paper at ICLR 2025
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint arXiv:2309.04550 04504, 2023.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalizationn via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alcksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Was Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899 2021.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint urXiv:211.00800 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.000
2021.
Levy Silva and Luciano Barbosa. Improving dense retrieval models with Ilm augmented data for dataset search. Knowledge-Bard Based Systems, 294: 11740, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Aonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks 2022.
and whether chain-of-thought can solve them. arXiv preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXivv2307.0922 2023.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764-74786 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thougge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arX14:2402.09739, 2024.
12

---

Published as a conference paper at ICLR 2025
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang, Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuming: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, arXiv preprint arXiv:2311.07911, 2023.
13

---

Published as a conference paper at ICLR 2025
A FLOPS COMPUTATION
To compute the training FL.OPs of transformer model, we assume the hackward pass has twice the FLOPs the forward pass and follow the FLOP counts of various components of a transformer model for single forward pass in Kaplan et al. (2020) as in Table 3 We account for all training FLOPs in our analysis, including those from the embedding matrices. For large models, the contri- bution of FLOPs and parameters from embedding matrices is minimal. We apply factor of 2 to represent the accumulate cost (MAC).
Operation
Parameters
FLOPs per Token
Embed
Deacab muodel
Attention: QKV
21 Smodg
Attention: Mask (Q-K Dot)
Maayer dut
Attention: Project
Mayy
Feedforward
3dodadd
21 3tlaee3thmoodll
De-embed (IHead)
dmodel
2daoda/Woocab
N' 2o (2dgan +
Total (Non-Embedding)
Cooward 2N Maa
1.5dg) + docadelPoscab
N model Player (2damn
Total
Corward
2N + 2Whayer
1.5dg) Mmnoel?2 modee
Table 3: Parameters and FLOPs per token for different operations.
For better granularity, we show comparision between our calculation and one that uses the common approximation Ctrain 6.V where Ctrain denotes FLOPs, D denotes total number of training tokens, and N is the number of parameters in Table 4 In practically, the differences in FLOP calculation is small and thus pose no difference analysis.
Model
N
Playar
11ss
duuidl
dg
dam
FL.PSs per token
FLOPP Ratio (Ours/6ND)
LLAMA 2 7B
7B
32
4096
4096
11008
128 x 32
4.69c+10
1.12
LLAMA 13B
13B
40
4096
5120
13824
128
40
8.82c+10
LLAMA 70B
7UB
80
4096
8192
28672
128
80
5.03e+11
1.20
Table FLOP comparison. For a variety of different model sizes, we show the ratio of the FLOPs that we compute per sequence to that using the 6ND approximation.
We use this in our calculation of data selection costs for methods that utilizes LLMS (i.e. perplexity- based and gradient-based)
B DATA-SELECTION FLOPS
In this section, we detailed our estimation of the costs to perform different data selection methods.
Data Selection Method
FLOPS Cost
BM25
10a 4.4 x 10
Embed
PPL
1.53 x 10
LESS
8.27 x 10
Table 5: Data Selection Cost Summary for Each Data Selection Method
14

---

Published as a conference paper at ICLR 2025
Warmup LoRA Training
Gradient Features Computation
Data Selection
Compute
Complexity
Actual
Complexity
Actual
Complexity
Actual
Compute Storage
(Danmull N)
6 Hours
((()) NN O[[P] JIDI-N-44 N.
48 Hours
O(D
Daal d)
1 Min
17.7 GB
Table 6: Asymptotic complexity, wall-clock runtime (measured single A100 GPU hours), and storage cost associated with each step in LESS Xia et al. (2024).
B.1 BM25
We assume that each data point incurs computational cost of FLOP for BM25, denoted as CBM25 = 1. The total computation cost is approximated as proportional to the size of the dataset, [D]. Therefore, the data selection cost for BM25 is estimated to be 10a FLOPs.
B.2 EMBED
For the embedding-bbsed method, approximate the computation using the formula Corwardd = D, where N represents the number model parameters and D the dataset size. Given that the embedding model used has N 220M parameters (Ni et 202I). the data selection cost for Embed is estimated to be 4.4 x 10 FLOPs.
B.3 PPL
Perplexity- methods require passing every data point through the language model. Given that the cost per token for 7B model is c 4.69 1010 shown in Table 4 the data selection cost for PPL is equivalent to one forward pass. Therefore, using the LLAMA2 model for data selection, we approximate the perplexity computation cost be 1.53 10 FLOPs.
B.4 LESS
LESS involves a two-step process: 4-epoch warm-up training on 5% of the dataset D followed by gradient feature computation over the entire training and validation datasets. First, we calculate the FLOPs required for the 4-epoch warm-up training. Then, using the relationship between the time required for warm-up training and gradient feature computation, as outlined in Table 6 from (Xia al. 2024), we estimate the FLOPSs needed for gradient feature computation. Combining these, we approximate the total cost of LESS using LLAMM2 as the data selection model to be 8.27 x 10 FLOPPs
C PARAMETRIC FUNCTION
C.I FITTING PARAMETRIC FUNCTION
In this appendix, we describe the process for fitting a parametric model that captures the relationship between the number of data points and performance P(k), as function of computational cost. The model captures diminishing returns, dependence on computational cost, and convergence toward an upper bound.
We model the expected performance P(k) after training on data points as follows:
P(k) (P- Po) x (1 (1- exp -> C(k) (x0()) + Po
where:
Po is the zero-shot performance (i.e., performance without additional training).
P is the upper bound performance (i.e., the maximum achievable performance).
15

---

Published as a conference paper at ICLR 2025
À is parameter controlling how efficiently the method extracts value from additional com- pute.
((k) is the computational cost of selecting and training on data points.
C(DI) the total computational cost of training on the entire dataset.
The goal is to fit the parameters Po. P. and À to observed data. We fit the model by minimizing the difference between the predicted performance P(k:) and the observed performance Pot (k). This is formulated as the following optimization problem:
min Pa.
(P(kg Po, - Pota,s)
where:
N is the number of data points.
i is the observed performance at the i-th data point.
ky is the number of data points used for training the i-th observation.
P(k; Po, P, is the predicted performance using the parametric model.
To ensure meaningful parameter estimates, we impose the following constraints:
Po 2 0, performance cannot be negative.
* Po < P, ensuring that performance does not exceed the upper bound P.
A 2 0, as X represents the rate at which performance improves with compute.
The parameter P is set slightly above the maximum observed performance:
P max Pars,i +
where e 0.05 a small buffer to ensure convergence to upper bound.
We set the initial guesses for the parameters as follows:
p( 1, the observed performance at zero-shot (i.e., without training).
p(0) max; Pobs.i-
(0) 1.0, a reasonable initial guess for the compute extraction efficiency.
We use the Levenberg-Marquardt algorithm to minimize the objective function. This method is effective for solving non-linear least squares problems and efficiently handles the non non-linear nature of our parametric model.
The optimization problem is solved as follows:
N Σ
(P arg min
(o, - Pots,))
The optimization yields the following fitted parameters:
P = [fitted value], p [fitted value],
X [fitted value]
These fitted parameters provide a close match between the observed data and the model and helps understand for better decision-making in resource allocation.
16

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL DETAILS
D.1 FINETUNING SETTINGS
All experiments were conducted with parameter-effiient finetuning method LoRA et al. (2021) For the LoRA adapter, we specified rank of 128, value of 512, and dropout rate of 0.1 and applied across all attention matrices. Adding the LoRA adapter introduce minimal FLOPs overhead during caining-having no impact our FLOPS analysis mainly reduce memory requirements for more accessible training.
We follow standard practices in LLM finetuning Wang al. (2023 Ivison et al. (2023) and use the AdamW optimizer with veta-parameters (81, (2) (0.9, 0.99).
The learning rate is set to 2e-5 for the 7B/SB/13B models and le-5 for the 70B models. For data budget {2.5%,5%% }. we double the learning rate to ensure convergence loss. For all experiments, warmup ratio of 0.03, BFloat16 precision, and effective batch size of 128. For 70B model training, we used QLoRA to reduce the memory requirements and speedup the training.
For smaller data budget (2.5%-5%) experiments, perform five trials across five random seeds. For larger data budget 100%-10%)) experiments, we perform three trials across three random seeds. We report mean target task performance in our analysis. Optimization seeds are controlled through the entirety of the experiments,
D.2 PRETRAINED MODELS
Table 7
lists the pre-trained models we finetuned in this work. We expect our findings to generalize to these models and future, stronger open base models.
D.3 TRAINING DATASETS
For training, all of which are either human-authored or annotated. Details are provided Table 8 The FLAN V2 and COT datasets are derived from existing NLP datasets, while DOLLY and OPEN ASSISTANT contain open-ended generation examples with human- written responses. These datasets vary in format, sequence length, and tasks. Following Wang al. (2023). standardize their format using the 'Tulu' structure.
the same four processed datasets in Wang al. (2023); Ivison et al. (2023).
D.4 EVALUATION DATASETS
We evaluate our method on three benchmark datasets: MMLU (Hendrycks et al. 2020), BBH (Suz- gun al. 2022), and IFEval (Zhou al. 2023). Table contains more details about each tasks. Each subtask comes with few-shot examples or sample responses, which are used as validation set V for data selection and few-shot in-context learning demonstration in evaluation.
An important aspect of the data selection approach is the size of the validation set used for computing utility scores. While small validation set might cause overfitting insufficient representation of the target task, our experiments indicate that even modestly sized validation sets (e.g., containing more than 50 examples) Since we are selecting large subsets from the training data, the precision required from the validation is not overly stringent. These methods can effectively capture relevant training samples without
sufficient for methods like BM25 and Embed perform effectively.
Base LMs
# Params
Tokens
6.7B
2.0T
LLAMA-2
13.0B
2.0T
65.2B
2.0T
LLAMA-3
8B
15.6T
Table 7: Base models that finetuned in this work.
17

---

Published as a conference paper at ICLR 2025
Dataset
Instance
Sourced from
Rounds
Prompt Len.
Completion Len.
FLAN V2
100,000
NLP dutasets and numan-wriltees instructions
355.7
31.2
NLP datascts
CoT'
266
53.2
DOLLY
15.011
Human written from scratch
118.1
91.3
OPEN ASSISTANT
55,068
Human writeen from scratch
1.6
34.8
212.5
Table 8: Details of training dataset from Wang et al. (2033)
Len. is short for token length.
Dataset
# Shot
# Tasss
V
7
Answer Type
MMLU
5
57
285
18,721
Letter options
BBH
3
23
69
920
COT and answer
IFEval
50
541
Open Generation
Table 9: Statistics of evaluation datasets. The selection of evaluation tasks cover different kinds of
answer types.
LLAMA3-BB
LLAMA3-8B
65.4
50000001
100%
100%
10%6
25%
10%
2.%
65.2
65.2
25%
59%
MMLU Accuracy
65.0
65.0
2.5%
10T5
0%
2,5%
64.8
64.8
5%
64.6
255%
2.5%
955
64.4
64.4
2.5%
64.2
64.2-
1017
10 Compute (FLOPs)
1019
1017
1018 Compute (FLOPs)
101
Models Trained Parametric Fit
Interpolation
BM25
PPL
Random
Embed
LESS
Figure 7: Performance and Parametric Fit of Performance With Compute-Constrained Data Selection. (Left) We show of our different runs for given model size, where each scatter point is the final target task performance of single run. (Right) We fit parametric model of the performance in Equation (3) and display that as curves to pair with the empirical results as seatter points.
significant risk of overfitting, suggesting that, practice, reasonably sized validation sets suffice for similarity-based data selection LLM finetuning.
E
RESULTS: LLAMA3
We plot additional results on target task MMLU using LLAMA3 8B model in Figure Similar to LLAMA2, LLAMA3 8B results show that cheaper lexicon-based (BM25) and embedding based (Em- bed) methods significantly outperform serplexity-basee (PPL) and gradient- baseed (LESS) method. The marginal gains from using more sophisticated methods do not justify their selection costs.
18

---

Published as a conference paper at ICLR 2025
LLAMA2-13B
LLAMA2-13B
100%
40.0
5054
40.0
50%
37.5
37.5-
35.0
35.0-
IFBval Accuracy
10%
102
32.5
32.5
Fine tuned Fareto Pront
Fine tuned Pareto Front
30.0
30.0
29%
10k
27.5
27.5
25.0
25.0
22.5
22.5
2g%
2.5%
20.0
20.0
10
1019
10
101H
Compute (FLOPs)
Compute (FLOPs)
Models Trained
Interpolation
BM25
PPL
Paramotric Fit
Random
Embed
LESS
Figure 8: Performance and Parametric Fit of Performance With Compute-Constraanee Data Selection on IFEval. (Left) We show all of our different runs for given model size, where each scatter point the final target task performance of single run. (Right) We fit a parametric model of the performance in Equation (3) and display that curves to pair with the empirical results as scatter points. fit a pareto front dashed line based on these optimal strategies. At medium compute budgets, cheaper data selection methods outperform PPL and LESS, which rely on model information.
F ADDITIONAL RESULTS: IFEVAL
We plot additional results target task IFEval with LLAMA3 13B model in Figure As models scale to 13B, expensive data selection methods still underperform, despite their relative cost dimin- ishing with larger models. Cheaper methods remain preferred.
G EXTRAPOLATING FROM THE PARAMETRIC FUNCTION
Figure annd Figure 10, we extrapolate the parametric fits obtained from our 7B and 13B model runs by increasing the ratio between the training model size and the data selection model size, thereby effectively reducing the proportional cost of gradient- data selection. Graphically, this is seen when the parametric fit crosses the existing compute-optimal frontier-a tipping point at which the method becomes compute-optimal.
We found that the training-to- to--e-ttt----eioo model size ratio should be approximately 5 for perplexity- based data selection, and 10 for gradient- data selection. This suggests that using more expen- sive data selection methods becomes advantageous when the training model size largely exceeds the selection model size by the compute-optimal ratio. Under these conditions, meaningful efficiency gains can be achieved compared to cheaper data selection methods.
H ADDITIONAL ANALYSIS: MULTIPLE
MODELS.
We perform break-even analysis by varying the number of target tasks for each method. Figure 11 presents the analysis for the 13B model on MMLU, while Figure 12 focuses the 7B model BBH Figure examines the 13B model on BBH. Additionalll..Figuu 14j Figure provide results for the 7B and 13B models IFEval, respectively.
19

---

Published as a conference paper at ICLR 2025
7B Train Model 7B Select. Model
Trale/frkeet Model Size Ratio
Trainchelect Model Sin Radio 5
Trala/eelet Modat Size Ratio 10
Traingbolert Vodel Size Ratio 20
67.59
4755
47.0
47.30
947.00
47.0
47.00
47.00
66.99
66:22
46:0
46.00
15:00
66:00
4500
6000
6150
10m
10
4554
19
10
(PLOP)
Cocouate
13B Train Model / 7B Select Model
Vruity Selloct Midel Sizo Ratle a
Traindhetect Madel Sinn Radio 2.5
Train/eeect Model Size Ratio 5
VeittSelent Medal Sim Ratie 10
00.5
35.3
515
12.0
35.0
253
550
$4.5
54.5
54.5
545
14.0
50.
16.3
32.5
35.3
533
Coopate (FLON))
Ouspeno (FLO%)
Compute IFLOPO
Congate (FLO))
Methods
FPL.
1ESS
Compute Oycinal Pareto Proct Tipging Poirr
PEL (Parometri FID
LESS (Paramecic FLO
Figure 9: Extrapolating Compute-Optimal Model Size for MMLU. By extrapolating from the parametric fits obtained from the 7B and 13B model results for MMLU, we can find the compute- optimal ratio between the training model size and the selector model size required for the perplexity- based and gradient-based method. For perplexity-based data selection, our extrapolation suggests that training model should be larger (5x) than the selector podell-rroume 35B model parameters. For gradient-based data selection to be compute-optimal, our extrapolation suggest that the training model should be significantly larger (10x) than the selector model spalell-spcciilllll..... around 70B param- eters in this case.
7B Train Model / 7B Select Model
Trafe//ielect Madel Sias Ratio
Tralactiedect Model Kize Ratio
Tradtteedect Model Size Ratio 10
Trada Model Siae Ratio
422
41.8
41.5
41.5
11.3
41.5
41.
11.
41.
35
42.
43.
39th
122
Compota (TLohe)
Compote 2000S
Compate (TLDN))
Compate (TUONN)
13B Train Model / 7B Select Model
Trafn/Select Madel Size Batia
TratntSelect Medel Size Rati : 2,5
Traln/Select Model Soe Batia 55
Train/Select Madel Rize Batle 10
2 S
45
Congute (RUDRS)
Comprn (BLDN))
Congune (PLOes)
Domposo (PLOPS)
Methods
2PL
LESS
Compute Oprimal Piacto Front Tiging hunr
2PL (Paramstric FiD
LESS
(Pamanetri: F10
Figure 10: Extrapolating Compute-Optimal Model Size for BBH. We perform a similar extrap- olation for BBH and find results consistent with those in Figure 9 Specifically, for the perplexity- based method to be computc-optimal, the training model should be 5x larger than the selection model, and for the gradient-based method, the training model should be approximately 10x larger than the selection model.
20

---

Published as a conference paper at ICLR 2025
Number of Tacks
Number of Tasks
Number of Tasks
Number of Tasks 10
555.
$25
$55
M001
$15
12 Compine (TLOIN))
Compres (FTL00))
Compute (TTOPP)
Compute (PLO0))
tuned Pareto Front
LESS
Figure 11: Multiple Task-Specific Model Break-Even Analysis for 13B MMLU. Performance under compute-constrantt reach the finetuned Pareto frontier at 5 tasks, surpassing it at 10 tasks.
Number of Tasks
Number of Tasks =
Number of Tasks 10
Number of Tasks 20
R102
414
41.0
6122
412
402
412
61:2
01
134-
51.4
15
101
16+
1d+
Complte IFLOPS)
Compote (PLOP))
Compute (LLPP))
Corppate (2LO00
Parato Front
LESS
Figure 12: Multiple Task-Specifie Model Break-Even Analysis for 7B BBH. Performance under conpute-constraints reach the finetuned Pareto frontier at 10 tasks.
Number Tooks
Number Tanks
Number of Tasls
Number of Tanks
53
Compute ('LIIN)
Compute (PLOIN)
Compute IELOPI
Compute (TLDDi
Fine-tuned Parato Front
Figure 13: Multiple Task-Specific Model Break-Even Analysis for 13B BBH. Performance under compute-constraints reach the finetuned Pareto frontier at 5 tasks, surpassing it at 10 tasks.
Number
13
27
27
10
10
1p
187
15
13
10
13"
Compite (NUOIN)
Compute (FLON))
Compute (TUDD0)
Compute (7L0//4
Fize tuzed Pareto Front
LISS
Figure 14: Multiple Task-Specific Model Break-Even Analysis for 7B IFEval. Performance under compute-constraints reach the finetuned Parcto frontier at 5 tasks, surpassing it at 10 tasks.
ADDITIONAL ANALYSIS: DATA SIMILARITY
Figure 16 and Figure 17 show the Jaccard similarity of data selected by different data selection meth- ods for the BBH and IFEval target tasks, respectively. Across various percentages of selected data, Embedding and BM25 exhibit the highest similarity. In contrast, LESS shares the least similarity with the other data selection methods.
21

---

Published as a conference paper at ICLR 2025
Number of Thaks =1
Number of Thiks
Number of Tasks
Number af Taskes
46
42
25
25
20
20
13mh
Compute (PLOIN))
Compute (FLOIN)
UTLDIO
Compute (PLO//))
Fize tuzod Pareto Front
LBSS
Figure 15: Multiple -packSSpecffii Model Break-Even Analysis for 13B IFEval. Performance
under compute-constraite reach the finetuned Pareto frontier 2 tasks, surpassing it at 5, 10 tasks.
2.5% Selected Data
Selected
50% Selected
676
2016
79%
25%
07%
2016
20%
53%
52%
-2%
L005
28
106
29.5
40%
BM22
Lass
PPL
Labs
MEC
1E05
Figure 16: Data Similarity Between Data Selection Methods for BBH.
2.5% Selected Data
10% Selected Data
50% Selected Data
1002s
10025
70%
128
5.3
25%
I5.0
20%
73.4
535
28
PoL
Ph
Lob8
0.8
18
1.3
E268
26.3
25.4
33.6
32%
mn
Las8
Blgs
Emmo
APL.
LEGo
BMC3
Loos
Figure 17: Data Similarity Between Data Selection Methods for IFEval.
J LIMITATIONS
Repeating Finetuning Data In this work, focus finetuning the entire dataset for only one epoch. With a larger compute budget, is possible to repeat fractions or entire datasets multiple times. Multi-epoch settings, as explored in (Xia et al. 2024) could potentially provide further training speedup by repeating data selectively.
Sensitivity to Hyperparameters The effectiveness of finetuning can be highly sensitive to hyper- parameters such as learning rate, dropout, or optimizer choice. There may be specific learning rate that leads to quicker convergence. In this work, we fixed most hyperparameters to commonly used values for fine-tuning I.LMs, leaving further exploration of hyperparameter tuning for future work.
Other Data Selection Methods There are additional data selection methods not covered in this work that warrant investigation. While focused methods in terms of their compute efficiency, other approaches, such as classifier-based methods, could offer insights and deserve further explo- ration.
22


Reviews:
[{"review_id": "Dm65QZWfdq", "reviewer": "Reviewer_9BiK", "summary": "Surveys and experimentally compares different data selection methods for LLM fine-tuning, and reasonably and quantitatively concludes that only rather cheap methods that choose train samples based on some cheap similarity to the validation samples are likely to be worthwhile, but depends (of course) on how much training computation you are going to run.", "questions": "The value of the strategies that depend on similarity of samples to validation samples worry me in that they seem very dependent on the size of the validation set, and that if the validation set is too small one might overfit.  But perhaps it doesn't matter too much since you are always selecting some large number of training samples anyways, and so even if the validation set is as small as Paris (to use a 2D example), you still correctly pick the subset of training samples in Europe and dump the ones in the Americas, and that wouldn't have changed much if the validation set was all of France instead of just Paris. Be great to see some discussion & experiments about this, even if they are tiny, in this paper. \n\nSee also the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I found this is an important experimental contribution for practitioners and academics alike, and is likely to be heavily cited in the future. While there will inevitably be some discussion of whether they compared to all the right and best methods, I think that's in the details: they compared good and sufficiently recent example methods from high level strategies and showed significant enough differences that seem endemic to these different strategies.", "weaknesses": "The weaknesses I detail below should all be corrected, but they are all minor, none of them individually or in total would be a good reason to reject the paper. \n\nSECTION 3 PROBLEMS:\n\nAt the beginning of Section 3: “Our goal is to find the optimal subset S ⊆ X” pretty sure you mean subset S ⊆ D there?\n\nI think you are implying that the train set is not necessarily IID with the validation set, but that the validation set is IID with the test set. All I see you say is that the validation set is “correlated” with the test set, which is a really  weak and vague thing to say, but if that’s all you want to say, okay, but I will be curious if in your experiments you actually make the Val and Test sets IID. \n\nYou need to define that $T$ represents a training procedure, you just use it without defining it now. \n\n“By ranking the data points….” Given a large initial train set D, having to rank the datapoints at cost O(D log D) is not free, hope you guys are taking that into account. Of course, you might argue just touching all D samples is O(D), but that is less relevant if, say, we have an infinite generator of data D (e.g. a real-time reader of the datastream formerly known as Twitter) and an independent (non-ranking) decider of whether each incoming $x$ is worth training on, that is, we shouldn’t have to assume we need to sort at cost O(D log D).\n\n\nI’m uncomfortable as a reader that in (2) you are still defining your objective in terms of the test set. I agree that’s the ultimate goal, but if you actually implemented (2) it assumes knowledge of the test set. By the time you get to (2), I expected you to have switched to the validation set in the stated objective, which is different than the final metric, which should of course than be on the test set. \n\n\nSECTION 4 FEEDBACK: \nYou can cut some of the intro to Sec 4, but please add-in that Lexicon-based and Embedding-based are both strategies that try to select train samples that are similar to the validation samples, whereas Perplexity and Gradient solutions are optimizing for the effect on the model loss.\n\nSECTION 5 FEEDBACK:\nWhy do you assume training on all x is equal? Is that really true (honest question)? My guess is yes due to the very beaurocratic nature of how these models are run, but that’s not always true of machine-learned models, for example, a classic decision tree is much faster to evaluate for some inputs than others (if it has leaves of varying depths). \n\nIn computing C(k), you sum over C_v(x), which I assume is for x \\in D? Please be explicit there about which x you are summing over.  And I’m surprised that that cost does depend on $x$ Does C_v(x) really vary so much? Could that not just be \\|D\\| (= size of D) times some cost per training sample?  \n\n\nRANDOM: Really appreciate you comparing to just a random sample as a baseline. \n\nMINOR BUT IMPORTANT QUIBBLES: \nAuthors state too unequivocally: “in practice, the total compute budget is\npredetermined: the number of accelerators and their usage hours are allocated in advanced”.   That certainly is NOT true in many large companies that are actively training and leading with LLMs. So please hedge and preface that sentence with “In many cases,”. \n\n\nThis sentence didn’t make sense to me:\n“For example work on parameter efficient fine-tuning, targets improving the\nmemory-usage of this stage (Hu et al., 2021).”\n\nTYPO “create an minimal” -> “a minimal”\n\nSince you are citing John 1975, consider  also citing the famous and foundational Hart 1968 paper on Condensed Nearest Neighbors, the canonical and classic approach for selecting data for training (summarized e.g. in wikipedia: https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm). However, nearest neighbors is such a different machine learning paradigm that if you don’t feel that adds value to this work, you can ignore this suggestion, but given the statement in your paper is “Data\nselection is a foundational approach in machine learning where the objective is to create an minimal\ndataset from a collection of data” the Hart 1968 paper is exactly the classic paper to have done just that.\n\nTYPO “Data selection takes the full training data as input and\nchooses a subset to the train” -> “to train”\n\nThis sentence doesn’t quite parse, please re-write “Instruction-tuned models can handle a variety of possible inputs for downstream use cases as\neither classification or generative model”\n\nThis sentence needs some polishing of singular vs plural:\n“Therefore\nwhile instruction-tuning is not the direct focus of this work, it provide a real-world applications of\ncompute-constrained data selection.”\n\n\n“Assuming we at minimal” -> “at minimum”\n\nEquation (2) can be written all on one line, winning you a bit of precious space. \n\nIn 4.1 you refer to Section 4.1, which is a bit weird and really you can just delete that whole sentence. \n\nDo figure out how to bold the last column title C_forward in Table 1. It can be done (probably \\mathbf{}). \n\nTYPO: Fit of Compute-Performace Relationship  -> Performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VfBCFIBP88", "reviewer": "Reviewer_Z24H", "summary": "This paper studies a framework considering the practical challenges of training and fine-tuning large language models (LLMs) under computational constraints. It has established a trade-off between achieving better performance using larger data and lowering computational costs by selecting smaller subsets of data. A key takeaway is that simpler data selection methods, such as lexicon-based and embedding-based approaches, often provide a more efficient solution compared with more complex, compute-intensive methods like perplexity-based and gradient-based strategies.", "questions": "1. Could the author discuss a real-world scenario to demonstrate how the proposed methods could be applied to guide practitioners?\n2. Are the studied methods sensitive to the choice of model architecture?\n3. How do these methods scale with hardware improvements?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses compute-efficient fine-tuning, which is an important task in training LLM. Extensive simulations are conducted to provide empirical evidence and support the framework.", "weaknesses": "1. Although the author claims some simple methods such as Lexicon outperform the complex ones such as Perplexity and Gradient, as shown in Figure 1, the complex ones perform quite well especially under medium and large budget situations. It would be more important to study the tipping point, where the performance gains plateau became flat. This is the place where further increases in computing resources yield diminishing returns. \n2. It is not surprising to see the tradeoff between performance and data size. The conclusions in this paper are largely empirical and may not generalize well to other situations. The practical limit of parametric fit is limited, as it mainly fits observed data points without clear guidance on how to *extrapolate* results to new scenarios. For example, can the results from smaller models (e.g., 13B) be used to predict outcomes for larger models (e.g., 70B)? Can the parameters estimated from smaller models be reliably transferred for larger model? If practitioners need to run experiments on 70B models to obtain these insights and fit the parametric model, the results may not be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KbSXEZcC7a", "reviewer": "Reviewer_mMXd", "summary": "This paper considers selecting data for finetuning LLMs under a computational budget. The computational cost is divided into two parts: 1) when using the validation set to evaluate the performance, the validation set will incur an initial cost; 2) training on each sample will cost a fixed amount of computation. The authors propose an exponential distribution model to fit the model performance v.s. the training costs for four types of data selection methods: lexicon-based, embedding-based, perplexity-based, and gradient-based. The paper consists of numerical experiments over several models and several tasks.", "questions": "See the weaknesses part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. This paper considers an interesting problem, data selection under computational constraints, and has interesting observations that the initial cost cannot be neglected when considering the computational budget.", "weaknesses": "1. (Major) Lack of novelty: although this paper proposes a framework for analyzing the computational cost of each data selection method, it does not provide any new techniques based on this framework. Furthermore, the key observation is not very surprising: the computational cost contains an initial cost when evaluating the validation set, thus the perplexity-based or the gradient-based is clearly not optimal under a limited compute budget.\n2. (Major) Lack of soundness: a) the parametric model is selected to be an exponential distribution and the model is fitted to minimize the squared error, but the choice is never justified by any theoretical analysis or numerical results. The fitted curve is also not very convincing (e.g. Figure 3 and Figure 7). b) The Pareto frontier is never formally defined in this paper nor sufficiently discussed. It's very hard for me to believe that the fitted Pareto curve is indeed Pareto optimal as the points in Figure 8 and Figure 10 exceed the Pareto frontier by a large margin. Also, the fitted exponential curves surpass the fitted Pareto curve considerably in Figure 3, implying that the two curves even contradict each other.\n3. (Moderate) Lack of insights: this paper is rather an experiment report than a well-motivated paper. The motivation for studying such a computation-constrained data selection problem is not fully supported. The authors just launch a bunch of models, adopt several tasks, and collect all the results without providing sufficient analyses.\n4. (Moderate) The style file seems to not follow ICLR 2025 templates: the lines are not numbered.\n5. (Minor) Typo(s):\n  Figure 1 \"using a much larger base model then data selection model\": \"then\" should be \"than\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXJanwChJu", "reviewer": "Reviewer_B9fL", "summary": "The authors present a study on compute constrained data selection for training large language models (LLMs).\nUnlike preceding works, they do not constrain the size of the training set, but the compute, which is the sum of computational expenditure for data selection as well as LLM training. \nThey compare various existing data selection methods under this setting and come to the conclusion that many powerful data selection methods are almost never compute-optimal due to their computational cost, making cheaper data selection the favorable choice.", "questions": "- Regarding the notion of utility in Section 3: Is utility here something that is to be minimized, i.e. alternatives with lower utility are preferred over alternatives with higher utility? In the remainder of the paper (expected) model performance is considered for which clearly higher values are preferred.\n\n- I am not sure whether I understood the greedy data selection introduced in Sections 3 and 4. I am under the impression that all data points are scored individually and afterwards they are ranked according to their scores and selected until budget K is exhausted. Isn't it neccesary to do this in an interleaved manner, in order to capture effects like redundancy and the submodularity of utility? Consider the extreme case in which the most informative data point x is repeated K in the dataset, then we would end up with a selection that contains the same datapoint K times.   \n\n- In Figure 2, the plot represents performance of Mid-PPL, while the plot in Figure 3 represents performance of Top-PPL. What is the reason for this discreapency?\n\n- In Figure 2, what exactly is the dashed line? Shouldn't the Pareto front contain all solutions, that are dominating on the dimensions of compute (low) and accuracy (high)? The line is straight, is it something like a linear regression applied to the solutions on the Pareto front?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The study is well motivated and its results are of practical importance for finetuning large language models.\n- Empirical findings correspond well with theoretical framework", "weaknesses": "- The title of the paper is rather broad, while the study is rather specific. \"Compute-Constrained Data Selection\" does not indicate which type of data is selected for which type of task.\n\nMinor remarks:\n- p.3 line 3: \\mathcal{S} \\subseteq \\mathcal{D}, as \\mathcal{X} is not introduced\n- p. 6 bottom: methods -> method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)