SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
Authors: Yu Yang
Siddhartha Mishra
Jeffrey N Chiang
Baharan Mirzasoleiman
Venue: NeurIPS 2024 poster (2024)
Keywords: data selection
Primary area: optimization_for_deep_networks

Abstract: Despite the effectiveness of data selection for pretraining and instruction fine-tuning
large language models (LLMs), improving data efficiency in supervised fine-tuning
(SFT) for specialized domains poses significant challenges due to the complexity
of fine-tuning data. To bridge this gap, we introduce an effective and scalable
data selection method for SFT, SmallToLarge (S2L), which trains a small
model, clusters loss trajectories of the examples, and samples from these clusters to
guide data selection for larger models. We prove that during fine-tuning, samples
within the same loss trajectory cluster exhibit similar gradients. Then, we show
that S2L subsets have a bounded gradient error w.r.t. the full data, hence guarantee
convergence to the neighborhood of the optimal solution. We demonstrate through
extensive experiments that S2L significantly improves data efficiency in SFT for
mathematical problem-solving, reducing the training data requirement to just $11$%
of the original MathInstruct dataset to match full dataset performance while
outperforming state-of-the-art data selection algorithms by an average of $4.7$%
across $6$ in- and out-domain evaluation datasets. Remarkably, selecting only 50K
data for SFT, S2L achieves a $32.7$% accuracy on the challenging MATH
benchmark, improving Phi-2 by $16.6$%. In clinical text summarization on the
MIMIC-III dataset, S2L again outperforms training on the full dataset using
only $50$% of the data. Notably, S2L can perform scalable data selection using a
reference model $100\times$ smaller than the target model, proportionally reducing the
computational cost.

SMALLTOLARGE (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Loss Trajectories of Small Models
Yu Yang!
Siddhartha Mishra
Jeffrey Chiang² Baharan Mirzasoleiman'
*Department Computer Science, *Department of Computational Medicine University of California, Los Angeles (UCLA)
Abstract
Despite the effectiveness of data selection for pretraining and instruction fine-tuning large language models (LLMs), improving data efficiency in supervised fine-tuning (SFT) for specialized domains poses significant challenges due to the complexity of fine-tuning data. To bridge this gap, introduce effective and scalable data selection method for SFT, SMALLTOLARGE (S2L), which trains a small model, clusters loss trajectories of the examples, and samples from these clusters guide data selection for larger models. We prove that during fine-tuning, samples within the same loss trajectory cluster exhibit similar gradients. Then, show that S2L subsets have a bounded gradient error w.r.t. the full data, hence guarantee convergence to the neighborhood of the optimal solution. We demonstrate through extensive experiments that S2L significantly improves data efficiency SFT for mathematical problem-solving,, reducing the training data requirement just 1 11% of the original MathInstruct dataset [64] to match full dataset performance while outperforming state-of-the-art data selection algorithms by average of 4.7% across 6 in- and out-domain evaluation datasets. Remarkably, selecting only 50K data for SFT. S2L achieves a 32.7% accuracy on the challenging MATH [9] benchmark, improving Phi-2 (28) by 16.6%. In clinical text summarization on the MIMIC-II dataset [21]. S2L again outperforms training on the full dataset using only 50% of the data. Notably, S2L can perform scalable data selection using a reference model 100x smaller than the target model, proportionally reducing the computational cost.
Introduction
In recent years, large language models (LLMs) have revolutionized artificial intelligence by demon- strating an unprecedented ability understand and generate human language [7]. Among all the contributing factors, the quality and selection of data is becoming increasingly recognized for its importance training LLMs effectively. Recent research indicates that LLMs benefit more from training for additional epochs on carefully curated data rather than on larger, uncurated ones during pretraining [88 and instruction fine-tuning (67], making data selection one of the most promising means to unlock the next level of L.MMs language capability. However, while generalist models obtained through pre-training or instruction fine-tuning excel general language tasks, they may not deliver optimal outcomes in specialized domain, such as mathematics [1 [3] 30 64), code
42 32 medicine [43 44. 91, or finance 55 9.. These domains are not only critical for real-world
applications but also hold substantial economic and societal impacts.
"Code available at http://gghhh...ooooogggll...C.Ah
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Train ina time thours)
30
t-SNE 2
D-SNN
-0
Pytsla-7088
Pythis 4108
Pythis A0
Pythis 8.98
U-SNE
E-SNE
Model
(a) Hidden states the Pile on pretrained Pythia-410M
(b) Hidden states of MathInstruct on pretrained Pythia-410M
(c) Increase in training time as the size of the model scales up
Existing data selection methods depend heavily the feature representations from reference model, which makes their effectiveness vulnerable to the quality of training on the target domain (34]. For supervised fine-tuning (SFT), while pretrained models can effectively separate topics (shown in different colors) in natural language (Figure Tay. they struggle with fine-tuning data that deviates from the pretraining distribution (Figure lb). Additionally, the cost of training a reference model escalates with model size (Figure Ic making existing data selection methods for large models prohibitively expensive.
Figure 1:
To maximize performance in specialized domains, models fine-tuned on domain data offer superior capabilities over generalist models [20]. Yet, maximizing the data efficiency in supervised fine-tuning (SFT) for specialized domains remains challenging and under-explored problem. Firstly, heuristic approaches that are effective the instruction fine-tuning stage, like manual curation [67] or using advanced models such as GPT-4 for dataset evaluation (8) are less reliable due the need for specialized knowledge and become costly with large volumes of uncurated fine-tuning data. Beyond these heuristic methods, other approaches rely generating representations for each training example using a reference model, often utilizing metrics like perplexity (34), confidence [46] 53 or hidden states 48 4 as features. However, these techniques also fall short in SFT for specialized
domains for two reasons: (1) the significant shift between pretraining and SFT data can render these metrics less informative (Figure [b), and (2) the computation and memory demands associated with generating representations for each training example become prohibitive, as these specialized domains often require larger models, some with up to 540 billion parameters [0 43 leading to substantial scalability challenges (Figure
To tackle the challenges of data efficiency in SFT for specialized domains, we present SMALLTOLARGE (S2L), an effective and scalable data selection algorithm. S2L operates by first gathering training loss trajectories for each training example using small model. These trajecto- ries are then clustered, and similar number of examples are selected from these clusters uniformly at random. This process grounded in our thcoretical findings that examples within the same cluster exhibit similar gradients during training, thereby affecting the model similarly. Consequently, subsets sampled from these clusters have bounded gradient error w.r.t. the full data, allowing for training a comparable model with only a subset of data. Furthermore, we provide a convergence rate analysis for training on these subsets, establishing robust theoretical foundation for S2L's 's effectiveness and efficiency.
To validate S2L.'s effectiveness, we applied it the challenging tasks of SFT for (1) mathematical problem-solving and (2) clinical text summarization. Our experiments on MathInstruct (44] shows that S2L can significantly reduce the required training data size to just 11% of the original dataset size while still matching the performance levels of the full dataset, outperforming current state-of-the-e one-shot and online data selection algorithms by an average of 4.7% across in- and out-domain evaluation datasets. Remarkably, on the MATH benchmark [19]. S2L attained a 32.7% accuracy with just 50K data points, improving the best open-sourced model under billion parameters, Phi-2, by 16.6%. For clinical text summarization tasks on the MIMIC-III [1] dataset, S2L outperforms training on the full dataset, using only half of the data. Unlike existing methods that require training and getting features from large models, S2L achieves superior data efficiency using a model with as few as 70 million parameters, which is 100x smaller than the largest target model we train with billion parameters.

---

2 Related Work
Foundations of Data Selection. Data selection has been well studied for small models and classifica- tion tasks. There are one-shot algorithms that select data based on rankings of the proposed training statistics, for example, the L.2-norms of error and gradient vectors (EL.N and GraNd) [3), confidence and its variability across epochs [46, and the number of times each example is learned but then forgot at the subsequent training step 49 Besides these heuristic indicators, there are embedding-
pruning algorithms [45 and online selection algorithms with theoretical performance guarantees
60) and robustness [59] 62 [6]. Coleman et al. proposed to use the intermediate feature representation of small proxy model to select data for image classification. Most recently, data selection has shown great potential in more substantial training speedup when implemented near-storage hardware [1]. and data selection beyond supervised learning of image and multimodal learning
for efficiency [35||]||| 23
40
data, e.g., for self-supervised learning
33 also emerged.
Data Efficient Training of Large Language Models. For the pre-training of LLMs, Marion et al. studied data quality indicators including Perplexity, Error L2-Norm (EL2N) (39, and memorization ranking [5) and found training on examples with middle Perplexity rankings outperforms training on examples selected based on the other two metrics, and sometimes even outperforms training on the entire dataset. Tirumala al. uses pre-trained model embeddings to select data for LLM pre-training. The proposed algorithm, D4, first applies an embedding- based data de-duplication algorithm [I] and then discards data points that are the closest to the K-Means cluster centroids in the embedding space 45] to ensure diversity. On fine-tuning LLMs, existing work on data efficiency primarily focused on manually curating high-quality instructions [67], using strong closed-source models (e.g., GPT- 2 or ChatGPT) to rate the quality of each training example [8] 27 8 Bhatt et al. implemented an experimental design framework evaluate the existing data selection methods for instruction fine-tuning of LLMs and found selecting facility locations based on hidden representations (i.e., embeddings) the most effective. As the only data selection algorithm for specialized domains, SCIP 61 focuses on pruning low-quality code data for training code L.MMs. Since it relies on breaking the code syntax understand the characteristics of low-quality code in the embedding (i.e, hidden states) space, adapting SCIP to domains other than Python code data is non-trivial.
Adapting Large Language Models for Specialized Domains. The rapid development of large language models (LLMs) gives rise to new of-te-of-thhaarr models in specialized domains. For mathematical reasoning, Galactica 47). MINERVA [26] and Llemma [3] continue to train an LLM on large-scale math-related web data to improve model's 's general scientific reasoning; WizardMath 31 and TinyGSM 30 fine-tune LLMs using supervised data. Similarly for medical LLMs, Cheng et al. continued training pre-trained LLMs on medical text, and 43 fine-tuned PaLM with instruction prompt tuning on medical domain data.
3 Problem Formulation
LLM Fine-tuning Objective. Consider transformer-based language model, parameterized by 0, and denoted as pg. This model, when provided with a sequence of prompt tokens x = ( ZM)) generates a sequence of response tokens y (y1...., (u)). The conditional probability of generating y given x is then formulated as
x)( II pe (vllvixs11
Note that Y1.0 is an empty sequence. To adapt the pre-trained LLM for specialized domain of distribution D, supervised fine-tuning (SFT) is usually employed with domain-specific training D containing pairs of prompt x and annotated response y. The
dataset Durin = y) fine-tuning objective is thus to minimize the following negative log likelihood loss, expressed as:
min L(O, Drrain) =
1
Σ (xm)):EDmis
[logro(ys|x:]]
Data Selection Objective. In a general setting for data selection, we consider target language model pe with parameters 0. Given fixed data budget B, which constrains the number of data points
3

---

that can be used for training, our objective is to select a subset S S Durin to train the target model, such that obtains a superior generalization performance. In practice, the subset S is selected based on a reference model r parameterized by Φ, which generates representations, confidence scores, or other metrics for each data point (x; y); E Drrain denoted by (x, which will be utilized by a data selection algorithm to produce S.
In existing data selection algorithms, Φ is commonly either weights of the pre-trained target model or a target model that has been fully trained on the dataset Duain However, as evidenced by Figure [ representations generated by the pretrained model may not always be good enough for data selection in specialized domains, and fine-tuning the target model significantly increases the computational cost of data selection.
4 Methodology
Training a large target model to obtain feature representations for each example in Diain can be computationally intensive. However, recent finding demonstrates that the training dynamics of most examples are consistent across differently sized models of the same family, and this phenomena even generalizes across different model families [58]. Our proposed method, SMALLTOLARGE (S2L), leverages loss trajectories of training examples collected during fine-tuning small reference model on the full or a subset of training data.
Loss Trajectory. Let (b(1) be the parameters of small LM during training on Diain at times tq. q E reference model c.proyy
S2L records the loss trajectory for each data point at times tg during training the C.toxy (x((t)) where
C. (T))) Caroxy (o (1), (x,,
Poge Pace (x))),;,
and T is the length of the loss trajectory. Note that (6(t) is trained for fixed number of iterations from (tt-1)
Assume the parameter vector (( represents the parameters of the target model at the time t. Define I.tory - ( ...... L.moxy (c)() and L training loss trajectory of the example i on the small proxy model and the large target model, respectively. Let H; E IRd be the Hessian matrix for each example and assume that the loss function for each example during fine-tuning can be modeled by second-order Taylor approximation with bounded curvature (c < | S C), a reasonable assumption in fine-tuning settings. The following lemma shows that examples with similar loss trajectories on the proxy model have similar gradients throughout the training of the target model.
(A(t))) as the ((((t)) -.. L.tagee
Theorem 4.1. !f examples and j have similar loss trajectories on the proxy model, i.e., [ L.WNY|
S c, and their loss trajectories on the proxy and target model is similar, i.e., I L [urgerr]
< 6 for p {i,j}, then and j huve similar gradients throughout training the target model:
2c' 2CD2
V Corren (0)
V.
(orre (0)1
A.
d
where e' = 26 and ||0|| < D for all
The proof of Theorem 4.1 can be found in Appendix A.1 Theorem 4.J shows that examples with similar loss trajectories have similar gradients during the training, thereby influencing the model in a similar manner.
Data selection from Loss Trajectory Clusters. Once the loss trajectories are recorded on the proxy model, we apply a clustering algorithm to group examples based on the similarity of their loss trajectories. This results in set of clusters (C1, C2...., Ck), ). where each cluster C, contains examples with similar loss and gradient trajectory throughout the training:
C (x, Durai i arg min d(L.j, Lc,)), jE(K]
(5)
where LG, is the centroid of the loss trajectories cluster C. and d(---) is a distance metric, such as Euclidean distance, used for clustering. For datasets that contain different sources of data, we cluster each source separately.

---

17.5
15.0
2.75
Small model
12.5
2.50
Large mods!
10.0
7.5
5.0
mrjectory Index 1B
TT Index (3
Trjectory Index
Tejeetory indet 01)
Tapectory index (1)
(a)
(b)
(c)
(a) In the same cluster. (b) In different clusters.
Figure 3: Examples in the same clusters of train- ing trajectories small model (Pythia-70M) also have similar training trajectories on a large model (Pythia-2.88)) even if the trends may not be the same on both models.
Figure 2: Examples in the same clusters have very similar loss trajectories (Figure while the loss trajectories of examples in different clusters are very different (Figure
As shown Figure 2 clustering algo- rithms can effectively find groups of ex- amples with similar training dynamics. In Figure 3 we empirically show that can identify groups of examples with sim- ilar training dynamics a larger model by clustering the training trajectories of
Algorithm 1 Data Selection Based on Training Trajecto- ries (S2L)
Require: Training dataset Dusin with corresponding training trajectories, fixed data budget B, number of clusters K.
Ensure: Subset S Durain |S| S B.
1: Initialize S as an empty set.
Drrain smaller proxy model. With the clusters formed, the data selection strat- egy selects equal number of examples random from all clusters, as detailed in Algorithm|] I] In doing so, it effectively pri- oritizes selecting examples from smaller clusters. This is particularly important for datasets containing multiple imbalanced sources. this setting, training and test distributions often differ, and balanced se- lection from clusters ensures superior test performance on all groups in the test data.
2: Train a small proxy model and cluster examples in (each data source of) Dinin based on their loss trajectories and sort them by size to get ( CI.,, Ck}.
3: for each cluster Cg in C do
4: Calculate R4. the number of exam- ples
randomly sample from Ck. i.e., Rs (B - |SS))((K - 1).
5:
if Crl S Rg then
6: S + (...
7:
else
8:
5 Ss.Ss, where S C Cs is selected uniformly at random from C₁ and |Sp] R₃
The following theorem shows that, under the assumptions of Theorem 4.1 training with Incremental Gradient (IG) methods on the subset selected by S2L converges close neighborhood of the optimal solution found by training the target model on the full dataset. IG methods such as Stochastic Gradient Descent (SGD) update parameters iteratively based on the gradient of the loss of individual examples, multiplied by stepsize a. Formally,
9: end if
10: end for
11:
Return S
g++1 =
a
(6)
Corollary 4.2. Under the assumptions of Theorem 4.1 applying IG with stepsize a to subsets found by S2L, converges to the neighborhood of the optimal solution, follows:
||0++1 12 < (1 acc)++1 1 1-2 1 2 + a (F (2)2g²
where c S || B k K is the total size of the subset, Jmax is the largest gradient norm of individual examples during training, Tuin min, |Cyl, (Cyl,Tmax = max; |C2), R = min Bgmax
E/c} and do | 0* 1| is the initial distance to the optimal solution 0*, and & is given by:
K K[[min + (Fmax Tmia) m
(8)
The proof can be found in Appendix A.2
Experiments
In this section, we present the comprehensive experiments conducted to evaluate the efficacy of the proposed data selection method, SMALLTTOLARGE (S2I.), across two challenging domains (mathematical reasoning and clinical text summarization).
5

---

5.1 Baselines
We systematically compare S2L against a comprehensive set of open-soureed data selection methods.
These methods categorized based on the type of representation they use and selected as the most representative or hest-performing methods as identified in prior work. These include: (1) Random Sampling: selecting examples with the (2) Least Confidence 4 (3)Middle Perplexity
[34; (4) High Learnability, determined by the loss decrease before and after full fine-tuning [68];
and (5) Facility Locations selection based on hidden states [4). Additionally, we incorporate one online selection techniques: (6) Confidence Curriculum proposed by Varshney et al. which selects examples with decreasing confidence during the training. Given that the optimal reference model may vary for each one-shot selection method, we ensure a fair comparison by adopting the approach used in (34, which runs each method with both the fully fine-tuned target model and an early fine-tuning checkpoint as the reference model. We report the best results from these setups.
5.2 Specialized Domain 1: Mathematical Reasoning
Training Settings. We focus on fine-tuning using the Math dataset 64 with 262,040 training examples for its comprehensive coverage of diverse mathematical fields and its capability in training models achieve state-of-the-arr performance on the standard evaluation benchmarks. We employ the open-source model suites Pythia [6]. Phi-2 (28] Llama-2 (50) as our base models to validate our S2L algorithm and directly compare its performance against the state-of-the-art.
Evaluation Datasets. We follow the framework established in [64] for comprehensive assessment using several well-regarded datasets, including in-domain and out-of-domain datasets. For the in-domain datasets, we consider GSM8K . of-domain datasets, we consider SVAMP 38 Mathematics [33. SimulEq 25 These datasets
MATH [19], and NumGLUE [36]. For the out-
collectively span diverse range of mathematical subjects, such as algebra, probability, number theory, calculus, and geometry. Additionally, some questions in these datasets require the application of commonsense, reading comprehension, and multi-step reasoning. All questions are open-formed.
Evaluation Metric. We use the standard evaluation metric for open-formed questions, exact match, which measures the model's accuracy by comparing its generated answers against the correct solutions. For an answer to be considered correct, it must match the reference solution precisely.
More details about the settings and baseline implementations can be found in Appendix B
5.2.1 Setting 1: Less Data for Better Models
In the first setting, we standardize the number of training steps correspond to 3 epochs on the full dataset, aligning with [64]. This allows us maintain a consistent training schedule across different methods and data budgets, ensuring fair and accurate comparisons of the quality of data.
Time (hours)
Train Ret Model
2
Select Data
SCALING THE DATA: SOTA algorithms fail with small data budgets while S2L stands out across data scales. In Figure 4 we compare S2L against the baselines from Section[5.I] on Pythia- 410M across varying data sizes. The training trajectories used by S2L are based on P0th a model approximately 6x smaller than Pythia-410M. With the same number of training steps the full training, S2L exceeds the full dataset's 's performance using only 30K examples, only 11% % of the full dataset. It leads the runner-up baselines by an average of 4.7%, 4.6% and 2.4% with data budget 30K, 50K and 100K across all six evaluation datasets. While state-of-the-ar data selection algorithms like Facility Locations 4 and High Learnability [68] have decent performance with a large enough data budget (i.e.. 100K), all SOTA algorithms except S2L cannot even outperform the random sampling baseline when the allowed data size is small (i.e., 30K). Unlike the existing algorithms, S2L consistently outperforms all baselines and even full training across all data sizes. Note that compared to the runner-up algorithm in this setting, Facility Locations, the cost of S2I. is much lower in both training the reference model and data selection stages (Figure 5) and therefore more scalable to both larger target models or larger data sizes.
Facility Location S2L (Ours)
Figure 5: Wall-clock time re- quired to train the reference model and select 100K data from MathInstruct for training PYthia-410M.

---

Relative Accuracy to Full Data
1.29
Random
Full
1.12
1.13
Least Confi confidence
0.98
-+-
Middle Perplexity
High Learnailiity
0.84
Facility Location
0.65
Confidence Curricuuum
S2L (Ours)
0.48
Pretrained
0.42
0.32
0.28
0.16
38%
76%
38%
76%
Data Size
Data Size
(a) In-domain Avg
(b) Avg
Figure 4: Data Scaling: Accuracies (f) on in-domain and out-of-domain datasets using PYthia-410M. Data size refers the total number of unique training examples used. All experiments this table share the same total training steps and learning rate schedule (see Section 5.2) See breakdowns in Figure 14
Table I: Less Data, Same Compute: Zero-shot accuracies (%, 1) when S2L and the baselines select 50K data to train with the same number of iterations the full-data training. Results surpassing full training are highlighted in bold. Figure 4 follows the same setting but uses the Pythia 410M model.
FINE-TUNING
IN-DOMAIN
MODEL
DATA
GSMSK
MATH
NUMGLUE
AVG
SVAMP
MATHEMATICS
SIMULEQ
AVG
(PRRTRAINED)
34.9
31.1
PHI-2 (2.70)
RANDOM
67.9
30.1
60.7
52.9
77.1
51.2
37.5
54.1
LEARNABILITY
MIDDLE PERPLEXITY
66.4
29.5
54.1
50.0
74.8
50.4
39.8
52.5
LEAST
61.7
24.7
67.0
51.1
76.5
43.3
52.5
54.3
FACILITY LOCATIONS
66.2
31.3
62.4
53.3
74.4
58.4
34.6
54.5
S2L(OURS)
69.1
32.6
65.7
55.8
79.6
56.4
40.1
57.3
FULL-262K
SCALING THE MODEL: Data selected by S2L transfer to larger models in different model suites. We also test whether this subset, chosen using POth can effectively train larger models beyond 410M and models outside the Pythia suite. As shown in Table [ our experiments with Phi-2 reveal that fine-tuning on only 50K $21-selected data again outperforms full dataset training on the most challenging MATH [19] benchmark improving the pretrained Phi-2 by 16.6% and is more data efficient than training on the full MathInstruct dataset to get the same performance.
5.2.2 Setting 2: Less Data for Faster Training
The second setting consider is when fixing the num- ber of times each example can be seen over the entire course of training, directly translating smaller datasets into reduced training and storage costs. This is particularly beneficial for large models that would require extensive training times without data selection. By experimenting with models of larger sizes than the previous setting, we observe in Table 2 that S2L can achieve comparable per- formance full-data training when using only 50% data and thereby reducing both the data storage space and the training time by half.
30
Clusters
Number of Clusters
Loss Trajectories (Ours)
Fully-finetuned Embeddings
20
0.4
0.6
Fraction
Common Topic
5.2.3 Why is S2L So Effective?
Figure Distribution of the coverage of top-1 topic in each cluster. Taller bars on the right end the plot indicate clusters with higher concentration of a single topic and therefore suggest a grouping of similar examples.
Examples in Clusters Encode the Same Knowl- edge/Skill. In Appendix C we compare actual training examples in MathInstruct that get clustered together due to their similar training trajectories on the small Pythia- 70M model. We observe that examples in the same cluster are of the same type and related the same knowledge/skill, e.g., open-formed algebra questions (Figure 15). examples requiring extracting useful information from
long text and writing programs

---

Table 2: Less Data, Same Epochs: Zero-shot accuracies (%, 1) when S2L trains 50% data for the same number of epochs as the full-data training. S2L can achieve comparable performance to full-data training while reducing both the data storage space and the training time by half.
TARGET MODEL
FINE-TUNING DATA
IN-DOMAIN MATH
OUT-DOMAIN MATHEMATICS
GSMBK
NUMGLUE
AYG
SVAMP
SIMULEO
(PRETRAINED)
74.5
26.5
52.1
$1.1
83.7
44.3
34,8
52.7
PHI-3-MINI (3.8B)
S2L-50 (OURS)
76.3
42.5
76.4
65.1
83,8
62.1
51.6
65.4
FULL
76.4
42.9
75.3
64.9
84,6
60.2
51.9
65.2
(PRETR.INED)
3.1
4.2
16.5
7.9
14.1
8.3
2.3
8.1
LLAMA-2-7B
S21. % (OURS)
53.3
28.9
6.500
49.1
65.1
45.2
31.9
48.2
FULL-262K [64]
52.2
30.4
47.7
65.3
50.2
(Figure I6), and multiple choice questions that require multi-step reasoning (Figure 17), etc. There- fore, by sampling from different clusters, we make sure the selected examples cover the knowledge required for all topics and skills required for all types of questions.
Loss Trajectories can Capture the Similarity Between Data Points As Much As Embeddings of a Fully Fine-tuned Model. We conducted a quantitative analysis to assess how effectively S2L identifies similar examples using loss trajectories from a small model. Assuming math problems under the same topic require similar knowledge and share question formats, we used unknown topic labels during S2L's 's data selection to check if each cluster predominantly contains single topic. By calcu- lating the fraction of the most common topic in each cluster and plotting this in Figure (with K=100, excluding clusters of size one), we compared the loss trajectory clusters from S2L (in blue) against those from the embeddings of a fully fine-tuned Phi-2 model (in (range)-considee for similarity. Results show that most clusters formed by S2L using the Pythia-70M model are based on a single topic and capture topic similarities more effectively than those from the Phi-2 model's em- beddings. This analysis not only confirms the homogeneity within S2L clusters but also highlights the computational efficiency of using loss trajectories on small models to identify representative examples.
(with
the ground truth
5.3 Specialized Domain 2: Clinical Text Summarization
S2L can improve data efficiency not only for fine-tuning data not only in mathematics but also in other specialized domains, This subsection explores its application to clinical text summarization within radiology reports. This task involves processing the detailed analysis and results listed in the findings section of a radiology report and distilling them into a concise impression section. Such summaries are crucial for providing clinicians with quick and actionable insights from radiological studies.
Dataset & Setup. We use the MIMIC-III dataset [2I], a comprehensive collection of radiology reports and findings authored by attending physicians in routine clinical care. We the same preprocessing procedures as [4 15) to extract the findings and impression sections and remove invalid reports. Given that access to MIMIC-II requires specific credentials, example of a radiology report generated by GPT-4 (2) for illustrative purposes in Table 3 We employ the Pythia-IB model and keep the training setting consistent with the mathematical reasoning task.
provide synthetic
Evaluation. Our evaluation generated clinical summaries on the MIMIC-III dataset'ss test split employs three key metrics as recommended in [52 51): (1) BLEU (37, which measures word sequence overlap between the generated and reference texts; (2) ROUGE-L [29]. assessing the longest common word sequence; and (3) BERTScore 655. evaluating semantic similarity using BERT's contextual embeddings. These metrics together offer a comprehensive evaluation, ensuring summaries are not only precise in language but also meaningful and coherent in the context of clinical information. We compare S2L to random selection, a surprisingly strong baseline as evidenced in Section (5.2, to check the validity of the data selection problem on this dataset and then compare it to training on the full dataset to assess its effectiveness.
We compare using 30K examples selected by random vs. selected through S2L. Even with only half of the data, the model trained with S2L selected data achieves similar BLEU and significantly higher ROUGE-L and BERTSCore compared to the model trained on the entire 61.5K data. Mcanwhile, training on randomly selected 30K examples performs worse than training on the full dataset on all 3 metrics. These results together demonstrate S2L's effectiveness.
Results.
5.4 Ablation Studies
8

---

2.4
We conduct ablation studies on MathInstruct and Pythia-410M further understand the best practices for using S2L
80.0
2.0
1.8
7.0
79.0
1.6
S2L is robust w.r.f. the length of the trajecto- ries but can benefit more from longer trajec- tories. Figure 8 compares models trained with data selected by S2L based on training trajecto- ries of different lengths. The shorter trajectories are derived from a uniform sample of the longer trajectories. From the small slopes of the lines, we can conclude that S2L is relatively robust to the length of the training trajectories. Neverthe- less, we can also observe slight increase in the performance on some of the datasets when longer trajectories are used, so having longer trajectories is still preferred.
6.5
Randen
14
S2t (Ours)
None
Randem
sal (Ours)
Nome
Random
52t (Ours)
Home
(a) BLEU
(b) ROUGE-L
BeRTS
Figure 7: Performance (1) of models trained on (1) randomly selected 30K examples, (2) S2L se- lected 30K examples, and (3) full 61K examples (none) evaluated with different metrics. The minimum value on the y-axis is the performance of the model before fine-tuning. S2L improves the data efficiency for the clinical text summariza- tion task by outperforming training the full dataset with only less than half of the data.
S2L can utilize training trajectories collected at any stage of training but preferably denser ones. With the length of the trajectories fixed to 4, we can observe in Figure that denser trajectories recorded at any training stage (early, middle, or late) are more helpful for S2L than trajectories recorded sparsely throughout the training.
S2L does not require the full training data to train the proxy and can scale efficiently larger datasets. To further demonstrate the scala- bility of the proposed S2L method, we conducted experiments by training the proxy on smaller sample of the data (100K examples) for the same number of epochs (3 epochs) and saving the loss for all examples. The results, shown in Figure TO confirm that S2I. remains effective when the proxy model is trained on a smaller subset of training data and therefore is scalable to larger datasets without a proportional increase in computational costs.
0.5
Sange Simulio
0.25
End
RATQLE
0.20
Middle
Accuraca
Avg Accurscy
0.15
0.2
0.30
0.05
0.00
Training Trajectory Longth
Figure 8: S2L is robust to the length of training trajectories.
Figure 9: S2L prefers dense trajectories over sparse ones.
S2L is robust across different clustering pa- rameter values for K. We conducted detailed experiments varying the clustering parameter K. as shown in Figure II The results demonstrate that S2L. maintains high performance across different values of K. highlighting the robustness of our method to different clustering parameter choices. We chose K=100 for our experiments as provided the best average accuracy across the evaluation datasets for the math reasoning task.
S2L remains effective and efficient compared to using full data when trained for the same number of epochs. Figure [2 illustrates the relative accuracy to full data across different epochs, comparing S2L-selected data and full data with the same number of epochs. Both in-domain and overall average accuracy are shown. S2L demonstrates superior performance with fewer data and fewer training iterations.
S2L supports a range of small models as effective proxies. To understand whether different small models could serve as effective proxies, we used GPT-2 (124M) and Pythia-160M, proxy models for data selection to train Pythia-410M. The results, illustrated in Figure [3] show that both proxy models perform comparably in guiding the data selection, demonstrating the versatility and effectiveness of using different small models for S2L.
Conclusion and Limitations

---

Prosy Training
100K Examples
Accuracy
100
Accuraly
Data
150
coekk
Humolue
Stite
Madhrearii
Sinusla
Avg
medemain
MATH
Avg
Spition
Mathemenics
Simulig
Avg
trstomain
Figure 1I: Per-dataset and average accuracy across different values of the clustering param- eter K. S2L is relatively robust to the choice of K.
Figure 10: Per-dataset and average accuracy com- paring proxy training on 100K examples and full data. S2L remains effective.
Rel. Acc.
to Rull Data
S2L (Durs)
Rel. Acc. Rull Data
--- S2L (Ours)
Data
Data
1.12
Full Same Iterations
Full Data,
Same Iterations
1.03
0.98
Epoch
Epoch
(a) In-domain Average Accuracy
(b) Overall Average Accuracy
Figure 12: Relative accuracy to full data across different epochs, comparing S2L-selected data and
full data. S2L achieves superior performance with fewer data and fewer training iterations.
In this work, introduced SMALLTOLARGE (S2L), scalable data selection method improve the data efficiency of supervised fine-tuning (SFT) for large language models (LLMs) in specialized domains. By clustering data points based on their training dynamics smaller models and balanced sampling from all clusters, $2I. significantly reduces the required training data size without compro- mising performance compared to using the entire training dataset. Our comprehensive experiments across the mathematical problem-solving and clinical text summarization domains demonstrate the elfectiveness of S2L.
Our study does come with its limitations. S2L has been only tested within two domains, mathemat- ics and medicine, and on models up to billion parameters, constrained by computational sources. Additionally, our experiments employed fixed training schedule across all methods with- out further optimization or hyperparameter tuning for each method, including S2L. This unified ap- proach, while it ensures a fair comparison, may not fully capture the potential performance improve- ment that could be achieved with more tailored training strategies. We encourage further research to extend the application of S2L across broader spectrum of domains and investigate the impact of hyperparameter tuning on its effectiveness.
Proxy Model PytNi-11GDM
Acoracy
GPT-2(124M)
GRMaK
HumBLuE
Instomals Avg
Staap
Matheratic
Simulin
Avd
Figure 13: Per-dataset and average accuracy comparison between using different proxy mod- els (Pythia- 60M and GPT-2 (124M)) for data selection. Using both proxy models show com- parable performance, demonstrating the effec- tiveness of different small models as reference models for S2L.
160M
Acknowledgments
This research was partially supported by the National Science Foundation CAREER Award 2146492, National Science Foundation 2421782 and Simons Foundation, Cisco Systems, Optum AI, and UCLA Hellman Fellowship.
10

---

References
[1] Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[2] OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman. Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavar- ian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapinn
Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Camphell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings. Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti. Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim' on Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang. Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider. Nitish Shirish Keskar, Tabarak Khan, Logan Kil- patrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mly Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O' Keeff, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambatissa Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song. Na- talie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer' on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin WU, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2023. URL https://app..eematiicccoooo https://api www.sseeaannnnnineeeee wwwpttiicccccoooooor/reooo ortp/CorrpssI::25555222....
[3] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.106: 2023.
11

---

[4] Gantavya Bhatt. Yifang Chen, Arny M Das, Jifan Zhang, Sang T Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon Du, Kevin Jamieson, al. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint arXiv:2401.0666 2024.
[5] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin An- thony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint urXiv:2304.1 158, 2023.
[6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430 PMLR, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, 2020. URL
mew.eedddinnnnnnneeeeeeeeeeeeeee nwwrips. wtt.cc/pppppo woc/paper_fille files wtt.:ffllesppape//eeeee pwwpppppppop/apaappppp Paper. pdf
[8] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data, In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreeieezz net//oormm
[9] Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.0955 2023.
[10] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
[11) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
[12] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL
[13] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathemat- ics by guiding human intuition with ai. Nature, 600(7887):70-77) 2021.
[14] Jean-Benoit Delbrouck, Maya Varma, Pierre Chambon, and Curtis Langlotz. Overview of the RadSum23 shared task on multi-modas and multi multi-anatomica radiology report summarization. In Dina Demner-fushman, Sophia Ananiadou, and Kevin Cohen, editors, The 22nd Workshop on Riomedical Natural Language Processing and BioNL. Shared Tasks, pages 478-482, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/w120233.ionm bionlp-.. org 2023. bionlp-1.45
45. URL http:tttt///aaattooo////
[15] Dina Demner-fushman, Sophia Ananiadou, and Kevin Cohen, editors. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Toronto, Canada, July 2023. Association for Computational Iaaiiiioonnlinigini Linguistics. URL /tt.ttts//aaccaaaooloo.... orr.h//22ooyor//2233
bionlp-1.0
12

---

[16] Yihe Deng. Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httph::t/pppeeeeeee wwwperrvieewwwetffoooooee www.:/eme/foummiit 9QEVJ9qm46
[17] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
[18] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023.
[19] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL http:ttttt////trrte
[20] Joel Jang. Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the beneñits of training expert language models over instruction tuning. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 14702-14729. PMLR, 23-29 Jul 2023. URL https: pttp:////ooceeeee..e/ooo www.mlrrppree.. press/ #202/jang23a.hte html
[21] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[22] Siddharth Joshi and Baharan Mirzasoleiman. Data-efficient contrastive self-supervised learn- ing: Most beneficial examples for supervised learning contribute the least. In International conference on machine learning, pages 15356-15370. PMLR, 2023.
[23] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464 5474. PMLR, 2021.
[24] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8110-81 8118, 2021.
[25] Rik Joncel-Kedziorska Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, pages 1152-1151 San Diego, California, June 2016. Association for Computational Linguisties. doi: 10. 08653/1//N1-1111 11N16111366 1136. URL. htt.tts://aalaaatoollooe
org/N16. 1136
[26] Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[27] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: 1hi-1.5 technical report. arXiv preprint vrXiv:2309.05460 2023.
[28] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi- technical report. arXiv preprint 20X.v2230900463, 2023.
[29] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: http:/aaaaaaatcy......co 27rg//04-1011
13

---

[30] Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprini arXiv: 2tt.:3332...222 22.0241, 2023.
[31] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced enotitttrutt.. arXiv preprint arXiv:2308. 09583, 2023.
[32] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.0855 0856688. 2023.
[33] Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. Sieve: Multimodal dataset pruning using image-captioning models. In Conference on Computer Vision and Pattern Recognition, 2024. URL /tttps://ppenrevier
[34] Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint MxXiv:2309005644 2023.
[35] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6950-6960, PMLR, 13-18 Jul 2020. URI htttps::/ppocceeinngg
mlr. wwwps/eviisseeee http:///111/////ml/lleemmmmaaaaa
html
[36] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. NumGLUE: A suite of fundamental yet challenging mathe- matical reasoning tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3505-3523, Dublin, Ireland, May 2022. Associ- ation for Computational Linguistics. doi: 10. No.166//1/1022..com acl- long.266. URL https
//aclanthology org 2022 acl- 246
[37] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 111-118, 2002.
[38] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: 10. 20..1853/v1l 16533vv122211. naacI-main 168. URL
org. or0/2211.
2021 nwwcl maaec.lmmii.... 168
[39] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596-20607, 2021.
[40] Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 77848-17869, PMLR, 17-23 Jul 2022. URL https: https:// pttp://i/oocceee...//oo mlr. press/ h162:pooladzaad::1
html
[41] Neha Prakriya, Yu Yang, Baharan Mirzasoleiman, Cho-Jui Hsieh, and Jason Cong. Nessa: Near-storage data selection for accelerated machine learning training. In Proceedings of the 15th ACM Workshop on Hot Topics in Storage and File Systems, HotStorage '23, page 8-15, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702242. doi: 10. 10.1145/3599691 1.30599999996634444 URL https: //wii///oiiioii org 10. 1145 /0155//599991111 3603404
14

---

[42] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2.08. 12950, 2023.
[43] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):122-18,, 2023.
[44] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305. 096617 2023.
[45] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19533-19566 2022.
[46] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Lin- guistics. doi: 11885/61//22000000
(EMNLP), pages
main.746. URL
org/ 2020
emmlp-
main. mumlp-main... 746
[47] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:221 (tt::2211.09000 0,0085, 2022.
[48] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM pretraining via document de-duplication and diversification. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
https: http://pwererreseeeeeeerrrrrrrae
[49] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon, An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL. https //openreviev. Sepearrereee..et/foomr--------
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09220 2023.
[51] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. arXiv preprint arXiv:2307. 1433244....344,, 14344, 2023.
[52] Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, William Collins, Neera Ahuja, al. Clinical text summarization: adapting large language models can outperform human experts. arXiv preprint arXiv:2309.07433 2023.
[53] Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Let the model decide its curriculum for multitask learning. In Colin Cherry, Angela Fan, George Foster, Gholamreza (Reza) Haffari, Shahram Khadivi, Nanyun (Violet) Peng, Xiang Ren, Ehsan Shareghi, and Swabha Swayamdipta, editors, Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing. pages 117-125, Hybrid, July 2022. Association for Computa- tional Linguistics. doi: 10. 21118653/v11/222.depppl. 1.113. http::: https:://aclaaathooogy org/ 2022. deeplo-1.111
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL. ptttps://openrviiie net/ Forum?id=_ie _=iid===VVe____
15

---

[55] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, October 2020. Association for Computational Linguistics. URL https: /www://wwww..oo wwwwwblwwwbb.... org/ anthology/ 2020. wen.e-ddmosss
[56] Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182 2023.
[57] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sehastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. /7564, 2023.
[58] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 13711-13738 Toronto, Canada, July 2023. Association for Computational Linguistics. URL URLLatttps//aacaattooooooo org//023..2222 acl-long
doi: 10.18653/v1/200 767
[59] Yu Yang. Tian Yu Liu, and Baharan Mirzasoleiman. Not all poisons are created equal: Robust training against data poisoning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 25154-25165. PMLR, 17-23 Jul 2022. URL BoLLhttpa:/// httppp::///.ooceenngg... mlr. wttpa//........../11122222 press/v162/
yang22j html
[60] Yu Yang. Hao Kang, and Baharan Mirzasoleiman. Towards sustainable learning: Coresets for data-efficient deep learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 39314-39330. PMLR, 23-29 Jul 2023.
[61] Yu Yang. Aaditya K Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozière, Carole-Jean Wu, Ari Morcos, and Newsha Ardalani. Decoding data quality via synthetic corruptions: bneeddinggguider pruning of code data, arXiv preprint arXiv:2312.022411 02.44188.. 2023.
[62] Yu Yang, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 2953-2961. PMLR, 02-04 May 2024. URL httpps//pproceedings mlr press/v238/yang html
[63] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL /tttts:://ppprreeeeeee
[64] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Tweifth International Conference on Learning Representations, 2024. URL https:// openreview www.:e/oooooiiddlllll...
[65] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL HRRLhttss: :tttps://openreviert net wtt.://oooo..tt--tetttttttCC
16

---

[66] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Chinn Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. /277, 2023.
[67] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Oner Levy. LIMA: Less is more for aligmment. In Thirty seventh Conference Neural Information Processing Systems, 2023. URL httpttt:////eeree////
net nnn/eeetuuuuee for//ormmm? id KBBKKNOKXK2h
[68] Haotian Zhou, Tingkai Liu, Qianli Ma, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Lobass: Gauging learnability in supervised fine-tuning data. arXiv preprint arXiv:2310. 13008, 2023.
17

---

A Proofs
A.1 Proof of Theorem 4.1
Proof.
From the assumption that the loss trajectories of examples on the proxy and target models are
close:
|L prany
L
|
S 6, 4i.
(9)
Since i and j are in the same cluster Cj based on the proxy model, we have:
|
peoky S
(10)
Using the triangle inequality:
Iageet
L target
L
farget
Mooyy|
proxy ||
[ proey
L,
S 28 +
é
(11)
Therefore, at any iteration l:
Charget (0()) - (0))) sé', 1L.
(12)
Assuming that the loss functions can be approximated by:
ptarget
2 d H,d0 + g. d0 + Ci2
(13)
where &; is the loss of example at the beginning of fine-tuning, and d0 is the distance between the parameters of the pretrained model and those during fine-tuning Similarly for C) (0). The loss difference between and] j is:
strrgt
Td) C Caarget (do) = dd (H, H2)d0 + (g. - 9;) d0 + (c. - cj).
(14)
2
Given that C
L blangl (0)l S é', we can write:
do + (a - cy) sé' d (H: - dd + (g. To
(15)
Let us choose different values, ((1) and ((2), to generate two inequalities. For d0(1), we have:
(aO (H: H)) (g: );) To)T0 (1 + (a (q) Se's
(16)
and for (2) we have:
(d0l2)) (H: (g)) ()
T + +(c - c9) se'
(17)
Subtracting these two inequalities, get:
1 HH)O())) - (do(2))) (H, H2)d0(21) + (g. [8,=- ( - dd022)
$ 2
2
(18)
(do()) - -H ( H, (2)
< [H H,ll [[ 21
(1) H + HH (Icd0l))12 ||d 22))
4CD2
(19)
This gives us:
(g: T() - a0(2)) < + 2CD2
(20)
Assuming |10(1) a0(2)
d, we get:
26' + 2CD
|| - -gill
A.
(21)
d
18

---

A.2 Proof of Corollary 4.2
Without loss of generality, assume we select & example from each cluster and we have k Σ minje[R] [Cy]. Then the error of the subset in capturing the full gradient will be
s E(1g)) () + +A),
(22)
where 9j is the norm of the average gradient of the selected examples from Cj. In practice, weight elements of the subset by Tmin/k; which has similar effect to scaling the step size when training the subset. Let Imax max; Igill be the maximum gradient norm during training,
Tmax = max, CCy, Timin = min, Cyll. Then, we get
E' $ Σ (Tmin- - k) + (IC,1- -mmin (gr + A)
(23)
< K KK [min A
(Fmax min mmm)]
(24)
The first term RHS of Eq and the second term is due to selecting the same number of examples, k, from the larger clusters.
(23) is the error of the subset selected from C₁ to capture its full gradient
Using the above error and following the proof of Theorem in [35, for a constant step size S 1/c we get:
1 1-12 S ac) -0112
R/c2 + (Fmin///m/ (
(25)
where c <
K is the total size of the subset, R minfdo. Bgmax + E//c} and do 000 0'l is the initial distance to the optimal solution 0"
|. and
If k |C,l for any cluster Cj. one can simply add (Tmin/ /h:in//:: 1) 9j g' for the corresponding clusters, where ÿj is the norm of the total gradient of cluster Cj and we replace min in Eq (33) with the size smallest cluster that has larger than examples.
B Experiment Details
B.1 Models
Pythia. The Pythia models 66] are suite of large language models (LLMs) developed by EleutherAl licensed under the Apache License 2.0. These models range in size from 70 million to 12 billion parameters and are designed enable controlled scientific research on transparently trained LLMSs across various scales.
The Phi models (28 developed by Microsoft under the MIT License. Phi-1.5, a transformer- based model with 1.3 billion parameters, and its successor, Phi-2, with 2.7 billion parameters, have been trained on diverse set of data sources, including synthetic texts and curated websites. The Phi models underscore the potential of small yet powerful language models in understanding and generating human language, empowering a range NL.P tasks. Phi-2, in particular, has raised the har for reasoning and language understanding among foundation models, matching even exceeding the performance of models 25 times its size on complex benchmarks.
Phi.
The LLaMA 2 models 550 released by Meta AI and licensed under the LLaMA 2 Community License Agreement, are designed for improved natural language understanding and generation. LLaMA 2-7B, the smallest in this series with 7 billion parameters, has demonstrated competitive performance across various NLP benchmarks despite its moderate size.
LLaMA 2.
B.2 Datasets
The MathInstruct dataset 64 is compiled from 13 diverse math rationale datasets, using both chain-of-thought (CoT) and program-of-though (PoT) rationales. It ensures compre- hensive coverage across various mathematical fields in the 262K training examples, making it popular resource for fine-tuning large language models (LLMMs) for general math provlem-solving. MathInstruct is licensed under the MIT license.
MathInstruct.
19

---

Table 3: A synthetic radiology report (MRI of the brain), generated by the GPT-4 model [ to demonstrate the typical data format and content used in the clinical text summarization task. It is not suitable for clinical or diagnostic use.
Findings
The brain parenchyma demonstrates normal morphology with no evidence of mass effect midline shift. No acute infarcts seen diffusion-wiighted weighted images. There are signs of intracranial hemorrhage. Mild generalized cerebral atrophy is noted. The ventricles and sulci appear within normal limits
patient's age. The pituitary gland and sella turcica are unremarkable. There are no abnormal signal intensities within the brain parenchyma. The orbits, paranasal sinuses, and mastoid air cells are clear.
Impression
Normal MRI the brain. Mild cerebral atrophy, likely age-related. No acute intracranial pathology.
The MIMIC-III (Medical Information Mart for Intensive Care III) dataset [21 is comprehensive collection of de-identified health data associated with over 40,000 patients who stayed in critical care units of the Beth Israel Deaconess Medical Center in Boston, Massachusetts. This large dataset includes information such as demographics, vital signs, laboratory tests, medications, and more, making it an invaluable resource for wide range of research in healthcare, including clinical decision support systems, medical procedure efficacy studies, and patient care optimization strategies.
MIMIC-III
The MIMIC-III dataset is made freely available to the research community under the Health Insurance Portability and Accountability Act (HIPAA) compliance, ensuring patient confidentiality and data protection. Access to the dataset is granted under a data use agreement (DUA) to individuals affiliated with institution that approves the use of the data for research purposes. Rescarchers seeking to utilize the MIMICIII dataset must complete a required training course on human research protections, which ensures that all researchers are aware of the responsibilities involved in handling sensitive patient data.
B.3 Implementation Details
S2L The training trajectories for both MathInstruct and MIMIC-III are gathered from training Pythia-70M model, the smallest model in the Pythia model suite, recorded every 500 training iterations. We utilize the Faiss library [17 to perform efficient K-means clustering loss trajectories with Euclidean distance with K 100 and 20 iterations. The hyperparameter K is tuned in the range of { (50, 100, 200} based on the average accuracy of the model trained on 30K selected data. We found K 100 worked the best for both datasets studied in this paper. Ablations studies the length and the best time in the training to record the trajectories can be found Section 5.4
Comparing Reference Models for the Baselines For one-shot selection methods (excluding S2L), we use representations from either step 1000 or the end of fine-tuning Pythia-410M on MathInstruct and reported the better result in Figure 4 and Table [ In Table 4 we include the complete comparison between using carly-fine-tume Facility Locations, we further compared using the first hidden states as the feature representation as suggested in [4] using the last hidden states (56) for the tasks studied,TTh ranges for confidence, perplexity, and learnability are chosen according to the best rest-performing intervals reported in prior research (Section 5.1))
vs. end-of-fine- fifffnnttutngg model checkpoints the inference model. For
Due to memory and computational constraints, for Facility Locations, we calculate pairwise similarity and perform greedy selection per-data-source basis. We found this per-source selection approach also yields benefits for S2L as different data sources within MathInstruct exhibit distinct common patterns in their training trajectories. Therefore, implement S2L also per-source basis for MathInstruct, and recommend applying S2L per source when dealing with datasets composed of multiple data sources.
Hyperparameters Following the setup used in 64 we adopt a training regimen with learning rate of 2e-5, a batch size of 128, a maximum length of 512, and a cosine scheduler with a 3% warm-up period.
20

---

Experiments Compute Resources We fine-tune all the models with the Huggingface transformers library [55 with Fully Sharded Data Parallel (FSDP) [66 on 48G NVIDIA RTX A6000.
Table 4: Complete results used for selecting the best reference model for each one-shot data selection baseline. The choice of early-fine-tuning (step 1000) and end-of- offfiinettuning checkpoint follows [3 The best results selected for Figure 4 are highlighted in cyan.
SELECTION
MODEL
SIZE
GSM8K
MATH
NUMGLUE
AVG
SVAMP
MATHEMATICS
SIMULEO
AVG
30K
15.5
6.5
13.6
5.8
EARLY
50K
1.7
2.6
20.5
8.3
16.0
4.0
1.8
7.8
LEAST
100K
3.9
2.7
22.5
9.7
19.2
8.0
3.3
9.9
CONRIENNEE
30K
2.7
13.7
END
50K
2.1
1.7
21.0
8.3
14.3
3.5
1.0
7.3
100K
2.5
3.3
23.5
9.8
20.8
6.3
3.7
10.0
30K
3.3
3.8
17.5
8.2
11.8
1.2
1.2
6.5
EARLY
SOK
2.9
4.1
19.6
8.9
15.6
7.6
2.9
8.8
30K
5.3
3.7
16.2
8.4
14.2
8.7
1.2
8.2
END
50K
3.2
5.9
20.5
9.9
18.1
11.3
5.1
10.7
7.2
20.9
30K
6.1
1.6
19.1
8.9
10.7
9.9
1.4
8.1
8.9
HIGH
100K
7.4
9.2
29.8
15.5
20.7
19.4
10.3
16.1
LEARNABILITY
30K
3.0
1.4
14.7
6.4
6.8
5.0
END
50K
1.3
2.1
16.0
6.5
6.9
3.1
5.7
100K
4.3
7.2
23.0
11.5
16.7
16.1
4.3
11.9
(FIRST)
50K
FACILITY LOCATION
EARLY (LAST)
50K
5.7
9.1
12.4
9.1
15.4
18.6
1.6
10.5
19.2
9.9
END (LAST)
50K
5.2
9.7
11.8
8.9
12.4
18.2
1.0
9.7
B.4 Evaluation
B.4.1 MathInstruct
We utilize 6 diverse datasets with open-formed questions for evaluating the mathematical reasoning capabilities of models trained with both the full MathInstruct dataset and selected subsets. These datasets, detailed in Table span a range mathematical disciplines from early algebra calculus and linear algebra, covering various types of questions such multi-step reasoning, arithmetic word problems, and problems from mathematics competitions. This variety ensures comprehensive assessment across both in-domain and out-domain tasks.
Datasets.
designed to first determine whether the model can be prompted to generate code snippet. This code snippet, successfully generated, should be executable and produce the correct answer when run. This code-based evaluation is also used for Phi models 228 In cases where model does not directly produce a viable code solution, we employ "think step-by-step" prompting strategy (54). This method prompts the model to break down its reasoning process, technique that has been widely proven effective fully exploiting the model's problem-solvine capacity.
Pipeline. We utilize the pipeline provided by 64
B.4.2 MIMIC-III
Following 15), we include the most common modalitylanatomy pairs: CT head, CT abdomen, CT chest, MRI head, CT spine, and CT neck, and five less common pairs in the text data: MRI spine, CT sinus, MRI abdomen, MRI pelvis, and MRI neck the evaluation. There are total 13.7K test examples after data preprocessing and train-test splitting.
21

---

Table 5: Types of questions in the evaluation datasets for the mathematical reasoning task.
DATASET
Size
LEVEL
TASKS
GSM8K
1319
Early Algebra
Multi-step reasoning using basic arithmetic operations
MATH
5000
Early Algebra. Intermedi- Algebra, Algebra, Prob- ahility, NumTheory, Calcu-
Problems from mathematics competitions, in- cluding the AMC 10, AMC 12, AIME
lus, Geometry
NumGL.UE
1042
Farly Algebra
Commoneense, Domain specifie, Arithmetic Reasoning- Quantitative Comparison, Fill- in-the-blanks Format, Reading Comprehen- sion, Numerical Rensoning, Quantitative NLI, Arithmetic Word Problems
SVAMP
1000
Early Algebra
Arithmetic
Mathematics 1000
Early Algebra, Intermedi- ate Algebra, NumTheory, Calculus
Arithmetic Reasoning
SimulEq 514
Linear Algebra
Single and multiple equation word peoblems
C
Examples in Different Clusters
compare data points the same and different clusters based on training trajectories, Figure [5] Figure 16 and Figure 17 We can observe that examples with similar training trajectories have the same question format. Therefore, balanced sampling from all clusters can ensure different types of examples can be covered in the selected subset of training data.
22

---

Relative Accuracy Full Data
1.28
Relative Accuracy Rull Data
Accuracy Full
1.06
1.13
0.85
0.94
0.90
0.64
0.60
0.43
0.21
0.30
0.38
38%
76%
38%
76%
38%
76%
Data Size
Data Size
Data Size
(a) GSM8K
(b) MATH
(c) NumGLUE
Relative Accuracy Data
1.28
Relative Accuracy Full Data
Relative Accuracy Full Data
1.07
1.06
0.86
0.85
2.16
0.64
0.64
1.62
0.43
0.43
1.08
0.21
0.21
0.54
Data Size
Data Size
Data Size
(d) SVAMP
(e) Mathematics
(I) SimulEq
1.26
Relative Accuracy to Full Data
1.29
Random
1.13
Least Confidence
0.97
Middle Perplexity
High Learnabliity
0.84
0.81
Facil itylity Locations ons
0.70
Confidence Currccluum
(Ours)
0.56
Pretrained
0.42
0.32
Full Data
0.28
0.16
Data Size
Data Size
(g) In-domain Avg
(h) Ave
Break- accuracies (f) on in-domain and out-of-domain datasets using Pythia-410M. Data size refers to the total number of unique training examples used. All experiments in this table share the same total training steps and learning rate schedule (see Section 5.2
Figure 14:
D Topic Distribution of Data Selected by S2L
Beyond qualitative examples from different clusters, study how S2L changes the data distribution to outperform using the full fine-tuning dataset as well as using random subsets of the same size that have the same distribution as the original dataset. In Figure we observe that S2L not only guarantees thorough and balanced coverage across the spectrum topics but also ensures sufficient representation of foundational topics, such as pre-algebra, which lays the groundwork for tackling more complex subjects.
E Broader Impacts
This paper introduces data selection method for large language models (LLMMs), aiming to enhance the data efficiency in the supervised fine-tuning (SFT) of these models.
23

---

Comden went swimming 16 times Morch and Susannch went 24 times. the number of times they went throughout the month was divided equally among à weeks, how many more times a week did Susannah swim than Comden?
Camden went swimming 16/4 = times a week Susannah went swimming 24/4 6 times week Susannah The answer 1s
week than Camden
What the perimeter of Koch sowflake curve after five iterations, if the initial side length of the equilaterol triangle used to construct the curve cm?
The Koch snowflake a smaller equilateral al triangle, After each Iteration, the number of sides increases by foctor of and the length of each side reduced by foctor 3.
curve constructed iteratively replocing
equilateral triangle
After five Iterations, the number of sides in the Koch snowflake glieen bg:
number of sides 33 (5*5) =3 1024 3072
The length of each side after five inerations is
side length Initial side length / (3^5) 8/243
The perimeter of the Koch snowflake ofter five Iterations is the product of the number of sides and the length side:
perimeter = number of sides side length = 3072 (8/ 243) 101.9933 cm
Lilly and Flona are cleaning a room. Between them, It takes 8 hours to clean the room. A quarter of the time spent cleaning was by Lilly and Plona wos responsible for the rest of the cleaning. How long, In minutes, was Flona cleaning?
8 hours is 8 6600-880 minutes.
Lilly cleaned for 080/s-120 minutes.
This meons Flona was cleaning for 480 120-360 minutes.
The answer is 360
Figure 15: Examples in the cluster shown in Figure 7a open-formed algebra. Questions are in black and answers are in cyan.
Passage: The Roiders began their 2011 campaign duel with Denver Broncos the second game Monday Night Football's doubleheade: Oakland rrolled early in the first quarter Broncos kicker Matt Proter got gard field gool. The Rolders answered in the second quorter quarterback Joson Compbell found tullback Morcel Reece 37-yard, a 21-gard, and an NFL, record tying 63-jard field goul from kicker Sebastian Janikowski. Janikowski' leg helped put the Raiders 16-3 at halftime. Denver answered in the third quarter as wide recetver Eric Decker punt 90 yards touchdown, followed bg Proter getting a 30-jard the fourth quarter with Campbell's 1 1-yard touchdown. The Broncos tried to rally with quarterback Ryle Ortan completing 9-gord touchdown pass to running bock Lance Boll, yet the Roiders' offense was able run out the clock. With the win, only did Oakland begin their season 1-0, they snapped 8-straight opening day losing streak. Question: How mony yords was the second longest field goal? Let's write progrom second print(second))
Authority Field Mile
Week AFC
3-gard touchdown poss, followed
goal. Oakland struck
Requirements (EARs) and Recommended Dietory Allowances (RDAs) for iron 2001. The current EAR iron women ages 14-18 is mg/day, &1 for ages 19-50 and 5.0 thereafter (post menopuuse) For men the EAR is 6.0 mg/day for ages 19 and up. The EDA is 15.0 mg/ddy for
Passage:
The Institute of Medicine (1OM) updated Estimated
mg/doy for ages and up. higher than EARs so as identify amounts that will cover people with higher than average requirements. RDA for pregnancy is 27 mg/day and, for Inctation, mg/dny. For children ages 1-3 years mg/day, 10 for ages 4-8 and ages 9-13,
ages
for 19-50 6.0 thereafter For
sofety, the evidence is sufficient. in the case of iron the UL Is set at 45 mg/day, Collectively the EARs, RDAs and UlLs are referred to Dietory Reference Intokes, Question: How mony years does RDA Jost for children? write Python program solve child 4 print(child)
Tolerable upper intake levels (ULs) vicamins and minerals when
Figure 16: Examples in the cluster shown in Figure 7b are in black and answers are in cyan; instructions are highlighted in orange.
reading comprehension coding. Questions
Positive Impacts: Our method, by reducing the data requirements for training LLMs, can make fine-tuning LLMs more effective and accessible. This could lead to broader participation AI research and application development across various fields, including healthcare and education.
Negative Impacts: Our method does inherently involve or encourage applications with direct negative societal impacts. The focus is on a generic improvement in the field of machine learning, particularly in the training of LLMs.
24

---

roje Answer Choices: (A) 2237 (B) 5777 (C) 2002 (D) 5772 (E) 8622 Let's reason about the multi-hoicee question.
possible
same k 2006 then n 2002.
equal
type as
The answer is
Stephanie, Regine, exactly 2 hours, nobody ron faster than à millss per hour, who could have won the roce?
race. Stephanie Regine's
by
1. Regine
Stephanie
I.. Brian
Choices: only only
choice guestion step step.
Given that S+R=B+2, where S, B, and B are times which Stephanic, Regine, and Brian completed the race. the fastest rate, would complete the roce 2.5 hours, so combined time needed Stephanie and Begine would be S+R=B+2-0.5 hours, which not possible rum of two must be more than equal the twice the least time:
complete
Let's Brian could
2"2.5e5,
could
There is reason to distinguish Stephanie and Regine so one could have won the roce, another alse could. both could have won the roce.
If i le the overage (arithmetic mean) of the first 10 positive multiples of and Ms the median of the first 10 positive multiples what the value of mmmmm
Answer Choices: (A) (B) -5 (D) (E)
Let's think about the multi-choice guestion step by step.
The first positive multiples
important properties of evenly spoced set (aka arithmetic progression) isin any evenly spaced set the arithmetic mean (average) is equal to the median.
evenly spaced
Hence Mam => M-m=0.
Figure 17: Examples in the cluster shown in Figure 7c multiple-choice + multi-step reasoning.
Questions are in black and answers cyan; instructions are highlighted orange.
Intermediate Algebra
Prealgebro
Other
Algebra
Number Theory
Probability
Geometry
Calculus
OK
20K
40K
60K
BOK
100K
120K
(a) Topic distribution of the full MathInstruct dataset.
Preaigebra
Preaigebra
Intermediate Algebra
Intermediate Algebra
Other
Other
Algebra
Algebra
Number Theory
Number Theory
Probabiiity
Probability
Calculus
Geometry
Geometry
5K
10K
5K
10K
15K
20K
(b) Topic distribution of 30K data selected by S2L.
Topic distribution of 50K data selected by S2I..
Prealgebra
Intermediate Algebra
Other
Algebra
Theory
Probability
Geometry
Calculus
OK
SKC
10K
15K
20K
25K
30K
35K
40K
(d) Topic distribution of 100K data selected by S2L.
Figure 18: Compared to original topic distribution, S2L prioritized easier topics (e.g., pre-algebra over intermediate algebra, algebra over other more advanced topics) while always ensuring complete and more balanced coverage of all topics.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The theoretical results provided in Section (and Appendix A and the experimental results provided in Section support the main claims made in the abstract and introduction. We also conducted experiments with both different data scales and model scales (Section 5.2.ID. in both the math and medical domains (Section [5.2 and Section (.3)) and ablation studies (Section 5.4). to reflect how much the results can be expected to generalize to other settings. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The complete proof for each theoretical result is provided in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification; All information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper is provided in Section 5.22, Section [.. and Appendix Additionally, the anonymized code is provided with the instructions in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
27

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The anonymized code is provided with the instructions in the Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
cublic/guides/CCe pag.lddss/cceeeSubmmmisiiioooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nipscco wwtppppcccggpuiceccocccoocccccce Bttpi///gudde//ooeeeieee htt////ooSuuiissseeeee htipbbmisssooooooiooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings for diffferent tasks we considered are presented in the corresponding sections in the main paper (Section 5.2 and Section 5.3) More details are provided in
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive. We ran every experiment independently from the data selection to the target model training without pre-defining random seeds to avoid over-fitting to one selected subset or cherry- picking the seeds.
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information about the GPU including the memory is provided in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://wwrrrppscccpupcleee/ee/ee
Question:
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and confirmed that the research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]

---

Justification:
The broader impacts of the work are discussed in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks because it does not release new data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in this paper are properly cited. The license for each model and dataset is provided Appendix B.I and Appendix B.2 respectively. The code used for evaluation is introduced in Appendix B.4 with the URL provided in the footnote.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
30

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
31

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ZA5XE36hEy", "reviewer": "Reviewer_mU1m", "summary": "This paper addresses the challenge of data selection in supervised fine-tuning (SFT) of pretrained language models by introducing a novel method called SmallToLarge (S2L). The S2L method involves collecting loss trajectories from a smaller model, clustering these trajectories, and resampling the SFT data to ensure balanced representation across clusters within the given data budget. The authors tested S2L on tasks such as mathematical reasoning and clinical data summarization, demonstrating superior results compared to several existing offline and online data selection methods.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this study is clear, and the proposed S2L method enhances data efficiency in supervised fine-tuning (SFT) for specialized domains.\n2. The writing is overall clear and the paper is easy to read.\n3. Extensive experiments are conducted to validate the efficacy of the proposed approach.\n4. The authors provide detailed analysis about why this method works.", "weaknesses": "1. The proposed method may require training the model for multiple rounds, which may not scale up to datasets with millions of examples.\n2. The proposed method introduce several additional hyperparameters such as the K in k-means clustering, the training epochs, but the effect of these components are not studied. \n3. The connection between the theory and the proposed method is not clear. More explanation on this matter would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your positive comments on the motivation, clarity, and extensive experiments of our study. We address your concerns and provide clarifications below:\n\n#### **1. Scalability Concerns:**\nThe S2L method requires only one additional round of 3-epoch training on a smaller model (e.g., Pythia-70M) to select a subset of training data. Training the 70M proxy on the MathInstruct dataset with 262K examples takes only 30 minutes with a batch size of 128. During this training, the loss trajectory of each example is saved, which is then used to cluster and select representative data. This selected subset can be reused for multiple fine-tuning sessions on larger models, significantly reducing computational costs compared to multiple rounds of full-scale training. For example, training the 7B model (Llama-2-7B in Table 2) with full data for *1 epoch* took 10 hours with a batch size of 64 and a maximum sequence length of 512, consuming 48GB of memory per NVIDIA A6000 GPU with 8 GPUs. As shown in Table 2, with the cost of 30 minutes of training for a small 70M reference model, we can reduce the training time of the target 7B model by half without losing performance. The cost of full-scale training goes up significantly when scaling to millions of examples, making S2L a more efficient and scalable approach.\n\n**New results:** To further improve scalability, we conducted experiments by training the proxy on a smaller sample of the data (100K examples) for the same number of epochs (3 epochs) and saving the loss for all examples. Training the proxy on this smaller sample reduces the training time to approximately 12 minutes while maintaining performance, demonstrating S2L’s efficiency and scalability for large datasets. Please refer to Figure 1 in the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf)) for detailed per-dataset and average accuracy results. We'll add the results to the paper.\n\n#### **2. Hyperparameter Analysis:**\n**New results:** We conducted a detailed analysis on the effect of K (number of clusters). **The findings summarized in Figure 2 of the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf)) demonstrate that S2L maintains high performance across different values of K,** indicating the method is not sensitive to the choice of K. Based on our analysis, we chose K=100 for our experiments because it provided the best average accuracy over the math evaluation datasets and use it for the medical dataset without further tuning, which confirms the robustness/stability of our method. \n\n#### **3. Theory and Method Connection:**\nThe theoretical foundation of S2L is based on the observation that, with a small curvature (which is typically the case during fine-tuning), examples with similar loss trajectories have similar gradients during training. Based on this observation, we cluster the examples based on their loss trajectories and then randomly sample examples from each cluster. This approach ensures that the subset's gradient captures the full data's gradient during training.  Because the gradients of the subset and the full data are similar, each gradient update on the subset closely approximates an update on the full data. This similarity guarantees that training on the subset using (incremental) gradient descent maintains similar training dynamics and converges to a solution comparable to training on the full dataset. Our empirical results show that sampling an equal number of examples from different loss-trajectory clusters improves performance by reducing redundancy in the data. This method ensures that the selected data is high-quality and representative, enhancing the overall efficiency and effectiveness of the training process.\n\nWe hope the additional explanations and analyses address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WOZFwvkmM", "reviewer": "Reviewer_5vbK", "summary": "This study presents an innovative technique aimed at improving data efficiency in the supervised fine-tuning of large language models for niche domains. The method leverages the training trajectories of smaller models to inform the data selection for larger models, thereby maximizing the utility of the available data. Experimental outcomes on tasks such as mathematical problem-solving and clinical text summarization demonstrate that this technique can significantly reduce the training dataset size while delivering better performance than other standard approaches.", "questions": "1. I am still not sure why using a small fraction of data is better than the performance of using full data on NUMGLUE and MATHEMATICS. Can authors provide more insights on this?\n2. How to determine the number of K for clustering?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper addresses a critical objective: enhancing the efficiency of large models through data selection, utilizing a smaller model to streamline the selection process.\n- Extensive experiments are conducted to demonstrate the effectiveness of this approach.\n- The paper is well-structured, with the method and results clearly presented and supported by theoretical analysis.", "weaknesses": "- Overclaim: The paper showcases experiments on only two specialized domain datasets, yet it is presented as if the method is universally applicable. While the introduction offers some context, the title remains overly broad, and the methodology lacks domain-specific adaptations. The authors should refine the framing of their contribution to more accurately reflect its actual scope.\n\n- Marginal Novelty: Data selection using proxy models is not a new concept, as highlighted by reference [12]. Additionally, the use of training trajectories has been explored in prior studies, such as [1].\n\n- Limited Evaluation: It would be beneficial to conduct experiments with larger language models beyond the 7B scale to validate the proposed method's effectiveness. The results only suggest that the method may be useful for smaller language models. Also, it would be better to calculate the win-rate for the summarization dataset.\n\n[1] LESS: Selecting Influential Data for Targeted Instruction Tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your positive comments on the objective, experiments, structure, and theoretical analysis. We address your concerns and provide clarifications below:\n\n#### **1. Overclaim:**\nS2L is generally applicable to various domains without requiring specific adaptations, ensuring its broad usability. As long as the fine-tuning data is in the form of pairs of questions and answers, our method is expected to work without any further modification. Converting domain-specific data (e.g., EHR records) into question/answer pairs has been studied in the literature and is beyond the scope of our work. Our study focuses solely on the \"fine-tuning step\" after the input data is formatted into question/answer pairs. We will clarify this point in our revised manuscript.\n\nOur title specifies that the method is aimed at improving data efficiency in supervised fine-tuning for specialized domains, which accurately reflects the scope of our work. Our experiments were conducted on more than one specialized domain (mathematical problem-solving and clinical text summarization) to demonstrate the potential versatility of S2L. We agree that further validation in additional domains is necessary and will ensure that the revised manuscript emphasizes this point.\n\n#### **2. Marginal Novelty:**\nWhile proxy models are used by prior work such as [12], those approaches _do not translate well to LLMs_. They employ metrics like forgetting scores, uncertainty, or representations of a proxy model for data selection. Forgetting scores track the number of times a model correctly classifies examples and then incorrectly classifies them in the subsequent training iteration. While this approach is well-defined for image classification, it does not translate well to autoregressive LLMs because they do not do sample-level classification. Uncertainty metrics, which measure the model’s confidence in its predictions, as well as using representations were included as our baselines and we showed that S2L is more effective than these methods. Moreover, these methods rely on heuristics that do not provide theoretical insights.\n\nDue to the above-mentioned reasons, **using a proxy model to select data for LLMs requires a different approach that works and scales better for LLMs than existing ones, which is the main contribution of our work.** Unlike prior heuristics, we have provided a **theoretically justified framework** by leveraging the theoretical observation that examples with similar loss trajectories have similar gradients during training and empirically demonstrating that loss trajectories can effectively guide data selection.\n\nRegarding LESS, it _addresses a different problem and is not applicable to our setting._ LESS selects training examples whose gradients are most similar to the gradients of a validation set, focusing on targeted instruction tuning. In contrast, **S2L does not assume access to a validation set and works without the need for clean validation data.** This makes the problems S2L can address potentially more difficult. Moreover, LESS requires projecting gradients into a low-dimensional space, which is computationally expensive and slow. S2L, on the other hand, leverages loss trajectories (which are **easier and faster to compute**) collected from training a smaller model and uses clustering to identify representative subsets, focusing on balanced representation across clusters to ensure data diversity and efficiency. \n\n#### **3. Limited Evaluation:**\nAs an academic research lab, our computational resources are limited, and models at the scale of 7B parameters represent the largest scale we can currently experiment with. Our 7B experiments with a batch size of 64 and a maximum sequence length of 512 require 48G memory per GPU to host on 8 48G NVIDIA A6000 GPUs. Hosting models up to the 70B scale is much beyond our computational capacity. According to `accelerate`'s estimate-memory tool, training `meta-llama/Llama-2-70b-hf` with a batch size of 1 using Adam optimizer and fp32 requires 1TB GPU memory, while fp/bp16 requires 512G GPU memory, far exceeding our resources. Nevertheless, we believe that our results on models up to 7B are significant and provide a strong indication of the method’s effectiveness. \n\n**New results:** In response to your suggestion about calculating win-rates, we used GPT-3.5-turbo as an automated judge to evaluate the clinical summaries. We presented the judge LLM with the original findings and impression from the radiology reports and two anonymized summaries (one generated by our model trained with S2L selected data, and another generated by a model trained with full data).\nOur prompt instructed GPT-3.5-turbo to act as an expert radiologist, evaluating the summaries based on accuracy, completeness, relevance, and coherence, while using the original impression as a reference. The win-rate was calculated as the percentage of times the model was preferred. Our S2L model achieved a 54.8% win-rate against the full-data model.\n\n#### **Questions:**\n\n1. **Performance with a Small Fraction of Data:**\nAs discussed in the introduction section of our paper, training more on smaller, higher-quality data can work better than training on larger, lower-quality datasets [48,67]. Our method is theoretically guaranteed to remove redundancy (i.e. examples with highly similar effect on training) to allow the model to learn more effectively from a diverse and representative subset of training data during fine-tuning. By ensuring that the selected data is non-redundant and representative, S2L allows the model to focus on the most informative examples, thereby enhancing the overall training efficiency and performance.\n\n2. **Determining the Number of Clusters (K):**\nPlease see Figure 2 in the global rebuttal ([link](https://openreview.net/forum?id=K9IGlMQpif&noteId=lyJkoUGNPM)). \n\nWe hope these clarifications address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsE7w4R9Xx", "reviewer": "Reviewer_Ho1K", "summary": "This paper introduces a data selection method called Small to Large (S2L), which uses the training trajectory of a small model to build clusters and select data based on these clusters. The method is shown to be effective in two domains, math and medicine, with superior performance than other data selection methods and, oftentimes, better than training on the full dataset. A theoretical guarantee is also provided for the convergence of training based on this data selection method. Extensive analysis demonstrates S2L's effectiveness under various compute budgets and across different model series.", "questions": "Please refer to weaknesses.", "limitations": "The work addresses the limitations of the limited domain tested and the limited size (7B) of the model used for evaluation.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The contribution of this work is indeed unique, as most data selection methods in the LLM era are focused on initial or continual pretraining and IFT, with relatively sparse attention given to SFT data in specific domains. This is important for many real-life LLM applications.\n- The idea is intuitive, as the training trajectory is informative of data characteristics, and effective. Additional theoretical support is provided for the similar effects during training by in-cluster data and the convergence analysis of data selected by S2L. Extensive experiments and analysis demonstrate the overall effectiveness of this method across various compute budgets (Figure 4) and for various model families like Pythia, Phi, and Llama.\n- Extensive analysis is also being conducted to better understand the S2L method, specifically the synergy between the cluster and fine-tuned model's embedding in terms of data similarity, as well as the robustness across the length and time frame of the training trajectory.", "weaknesses": "- As shown in Figure 4, it is clear that for many methods, the SFT has not yet converged. It would be interesting to see how each method affects the convergence of the SFT process and what the relative accuracy looks like, i.e., whether better performance can be further achieved.\n- The work studies the Pythia model as the small reference model and multiple models from different model families for task evaluation. It is intriguing to know whether other small models can serve as good reference models, and how the alignment between the pretraining data of the small and large models will affect the effectiveness of this data selection strategy.\n- In the LLM era, the size of SFT data scales up quickly. It will be interesting to see how this method scales with the SFT data size, for instance, at M/B token scale.\n- It is still tricky to determine whether the effect of such clusters will be influenced by different learning rate schedulers and optimizers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback on our paper. We appreciate your positive comments on the uniqueness, intuitiveness, theoretical support, and extensive experiments of our study. We address your concerns and provide clarifications below:\n\n#### **1. Convergence of SFT:**\nThe x-axis of Figure 4 is the size of the subset that we trained on till convergence. We conducted additional experiments extending our analysis to larger data sizes. Due to time and resource constraints, we focused on two data sizes larger than 38% (57% and 76%) for this extended analysis and 3 best-performing baselines in our original submission. The results are presented in Figure 3 in the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf)), which extends Figure 4 from our main submission. \nWe observe that:\n- Random monotonically approaches full data performance as the data size increases, showing expected behavior.\n- High Learnability does not show improvement when scaling beyond 38% of the data.\n- Both Facility Locations and our S2L method continue to improve when increasing from 38% to 57% of the data.\n- As we approach 100%, the performance of all methods drops and converges to the dashed line (full data).\n\nMeanwhile, S2L consistently outperforms other approaches across all data sizes, both for in-domain and overall average accuracy. \nIn our revision, we plan to include the remaining methods from Figure 4 that are not present in this new Figure 3. \n\n#### **2. Using Other Small Reference Models:**\n**New results:** We used GPT-2 (2019), which has 124M parameters, and compared it to Pythia-160M (2024) as the reference model to select data to train Pythia-410M. GPT-2 was pretrained on the WebText dataset, which consists of 8 million web pages scraped from outbound links on Reddit, and cleaned to remove Wikipedia pages and duplicates. The Pythia models, on the other hand, were trained on the Pile dataset, a curated collection of English language datasets that is popular for training large autoregressive transformers. The Pile dataset contains approximately 300 billion tokens and includes diverse sources such as books, Wikipedia, and various web texts. Results in Figure 5 in the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf)) show that both proxy models perform comparably in guiding the data selection for training Pythia-410M, demonstrating that different small models can be effectively used as reference models.\n\n#### **3. Scaling with SFT Data Size:**\n**New results:** The MathInstruct dataset, which we used in our experiments, contains approximately 60M tokens. Our method has shown strong performance and scalability with this dataset size. \n\nTo further improve scalability, we conducted experiments by training the proxy on a smaller sample of the data (100K examples, ~30M tokens) for the same number of epochs (3 epochs) and saving the loss for all examples. Figure 1 of the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf)) presents the results of these experiments,  demonstrating its efficiency and scalability for large datasets.\n\n#### **4. Effect of Different Learning Rate Schedulers and Optimizers:**\nWe tuned hyperparameters for fine-tuning the target model, specifically learning rate $\\in$ {2e-5, 5e-6} and number of training epochs $\\in$ {2, 3, 4}. Standard values were used for warmup ratio (0.03), weight decay (0), and learning rate scheduler type (cosine) as per [64]. The same hyperparameters were applied for all data selection methods, including S2L and the baselines, ensuring a fair comparison. For S2L, these hyperparameters were consistently used for both training the proxy model and fine-tuning the target model.\n\nWe hope these clarifications address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z83Sq8PTLr", "reviewer": "Reviewer_dErU", "summary": "This paper proposes a data selection approach to reduce the sample size required to conduct supervised fine-tuning (SFT) of LLMs for specific domains. The method achieves it by approximating the training gradients on the full data using only a subset of data.\n\nThe experiments were done for SFT tasks, including (1) math problem-solving and (2) clinical text summarization. It was found that with 11% of the original dataset, the yielded model can be comparable to the one trained on the full dataset.", "questions": "From Figure 4, it is surprising that the proposed method and some baselines can outperform the full data performance with around 38% of the samples. How much benefit can the method bring if we continue to scale the data?", "limitations": "do not apply", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides a clear rationale for selecting data for Sparse Fine-Tuning (SFT) by estimating the full gradients with a subset of samples. It conducts a thorough analysis of various variants of the proposed method and includes experiments across diverse scenarios of its application.", "weaknesses": "1. Data selection for machine learning has been explored in the literature. Some related works which this paper omits but may need to compare and discuss include:\n\n- influence function based approaches [1,2]\n\n- reinforcement learning based approaches [3]\n\n- data Shapley [4]\n\n2. The generalizability of the proposed theory remains elusive. A small model is used to approximate the gradients. There are two gaps\n\n- The small model can have a distinct loss landscape from the large model. Approximating full gradients of a small model not necessarily extrapolate to a large and different models.\n\n- There is no theoretical guidance on how this small model should be like, which architecture, which parameter size, to be competent of approximating the gradients of the large model. \n\nConsidering the black-box nature of deep learning, especially for LLMs, it seems very hard to build such theoretic foundation for the proposed method.\n\n\n[1] Wang, Z., Zhu, H., Dong, Z., He, X., & Huang, S. L. (2020, April). Less is better: Unweighted data subsampling via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 6340-6347).\n\n[2] Kong, S., Shen, Y., & Huang, L. (2021, October). Resolving training biases via influence-based data relabeling. In International Conference on Learning Representations.\n\n[3] Yoon, J., Arik, S., & Pfister, T. (2020, November). Data valuation using reinforcement learning. In International Conference on Machine Learning (pp. 10842-10851). PMLR.\n\n[4] Ghorbani, A., & Zou, J. (2019, May). Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning (pp. 2242-2251). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your positive comments on the rationale and thorough analysis of our method. We address your concerns and provide clarifications below:\n\n#### **1. Omitted Related Works:**\nOur paper includes related works in the data selection domain, but we will expand this section to incorporate a discussion of the specific references you mentioned in our future revisions.\n\nInfluence function-based approaches, such as those by Wang et al. (2020) and Kong et al. (2021), are not directly applicable to the problem we address with S2L. These methods rely on the availability of a validation set that is a very good representative of the test set, as they find training examples most similar to the validation examples. This is a different problem from ours, which does not assume the availability of a good validation set. \n\nSimilarly, reinforcement learning-based approaches, such as Yoon et al. (2020), also address a different problem from ours. It aims to quantify the value of data by learning data values that improve model performance on a small validation set, focusing on tasks like domain adaptation. This approach also requires a clean validation set that is a good representative of the test set. \n\nIn contrast, S2L focuses on dropping redundancies in the data, ensuring data diversity without the need for a clean and representative validation set. Thus, **influence-based and reinforcement learning-based methods [1,2,3] are not applicable to our problem** and are not considered as baselines.\n\nCalculating Data Shapley values requires evaluating all possible subsets of the dataset, **(for a dataset with $ |D| $ instances, the time complexity of directly calculating Shapley values is $ O(|D| \\cdot 2^{|D|}) $,) making it computationally infeasible for large datasets typically used in training LLMs.** Approximations to Data Shapley values reduce the computational burden but still remain impractical for large-scale applications​. Typically, estimating Shapley values accurately might involve several hundred to thousands of permutations, where each permutation involves training a model on different subsets of the data. In contrast, S2L’s training complexity is only a single additional training round of a smaller proxy model and its inference complexity is linear with respect to the proxy dataset size. \n\n#### **2. Generalizability of the Proposed Theory:**\nIn our paper, specifically in the \"Small-to-Large Data Selection\" paragraph in Section 4, we provided references and experiments supporting the use of small proxy models. In Figure 3 of our paper, we show that examples in the same loss trajectory cluster of a small model also have similar loss trajectories on a large model. This demonstrates that the clustering of loss trajectories from a small model can effectively capture the data characteristics relevant for training larger models, thereby validating our approach empirically. \n\nThe reviewer is correct that in our proof, we assumed finding the loss trajectories using the target model used for fine-tuning. Assuming that the distance between loss-trajectories of examples on the proxy and target models is bounded by some constant, this can be incorporated into our theory to bound the distance between the gradient of the subset and the full data at every step of (incremental) gradient descent, and can be used to bound the size of the neighborhood around the optimal solution (found by training the target model on full data) that the target model converges to when trained on the subset. For the fine-tuning setting where we assume a bounded curvature for the proxy and target models, the above assumption is reasonable. We thank the reviewer and will incorporate this discussion.\n\nAdditionally, Xia et al. (2022) analyzed the training trajectories of differently sized OPT models, ranging from 125M to 175B parameters. They showed that models of different sizes within the same architecture family pre-trained with the same data exhibit similar learning patterns and behaviors at equivalent levels of training perplexity, supporting the idea that smaller models can provide valuable insights for larger models. However, theoretically bounding the distance between loss trajectories of such models is not trivial and requires future investigation.\n\n\n### **Questions:**\n**New results:** Figure 3 of the one-page PDF attached in the global rebuttal **([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf))** shows that S2L first increases in relative accuracy compared to the full data approach and then its advantage over training on full data decreases as data size continues to scale up. Ultimately, as data size approaches 100%, the accuracy will converge to that of training on the full dataset. \n\nNote that we kept the total training iterations/steps consistent for all results in Figure 3, including training on the full dataset. As discussed in the introduction section of our paper, training more on smaller, higher-quality data can be more effective than training on larger, redundant datasets. Even if we train on the full data for more epochs, our experiments show that training on S2L selected data yields higher accuracy (Figure 4 in the one-page PDF attached in the global rebuttal ([link](https://openreview.net/attachment?id=lyJkoUGNPM&name=pdf))) than training on full data with the same number of epochs, even though in this case full data takes more training budget/iterations compared to training with S2L selected subsets. By clustering based on loss trajectories, S2L ensures that the selected data is of high quality and representative, allowing the model to focus on the most informative examples. Results in Figure 3 indicate that **S2L can bring significant benefits in terms of data efficiency, making better use of the available data.**\n\nWe hope these clarifications address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your valuable feedback and for recognizing the strengths of our work, including its clear motivation, extensive experiments, and unique contributions to enhancing data efficiency for large language models (LLMs). We have conducted additional experiments and analyses to address the concerns raised and provide new insights, summarized in the attached one-page PDF.

**Figure 1. Scalability:**
To further demonstrate the scalability of the proposed S2L method, we conducted experiments by training the proxy on a smaller sample of the data (100K examples) for the same number of epochs (3 epochs) and saving the loss for all examples. The results, shown in Figure 1, confirm that S2L remains effective when the proxy model is trained on a smaller subset of training data and therefore is scalable to larger datasets without a proportional increase in computational costs.

**Figure 2. Robustness to Clustering Parameter (K):**
We conducted detailed experiments varying the clustering parameter K, as shown in Figure 2. The results demonstrate that S2L maintains high performance across different values of K, highlighting the robustness of our method to different clustering parameter choices. We chose K=100 for our experiments as it provided the best average accuracy across the evaluation datasets for the math reasoning task.

**Figure 3 & 4. Convergence and Data Efficiency:**
Figure 3 extends the analysis presented in the main submission, showing the relative accuracy to full data as the data size increases, with consistent total training iterations/steps for all results. Both in-domain and overall average accuracy are shown. S2L consistently outperforms other methods, such as Random, High Learnability, and Facility Locations, which were the best-performing baselines based on the results we presented in our paper, in terms of relative accuracy to full data. This underscores the efficiency and effectiveness of S2L in achieving comparable or superior performance with fewer data and fewer training iterations.

Figure 4 in the one-page PDF illustrates the relative accuracy to full data across different epochs, comparing S2L-selected data and full data with the same number of epochs. Both in-domain and overall average accuracy are shown. S2L demonstrates superior performance with fewer data and fewer training iterations.

**Figure 5. Proxy Models:**
Reviewers also expressed interest in understanding whether different small models could serve as effective proxies. We used GPT-2 (124M) and Pythia-160M as proxy models for data selection to train Pythia-410M. The results, illustrated in Figure 5, show that both proxy models perform comparably in guiding the data selection, demonstrating the versatility and effectiveness of using different small models for S2L.

We hope these additional explanations and analyses address your concerns and demonstrate the robustness, scalability, and effectiveness of the S2L method. Thank you once again for your valuable feedback.

Meta-review: This paper provides a method for dataset selection for fine-tuning small LMs by looking at the the training trajectory. The experimental evidence is quite compelling (especially the extra results show in the rebuttal) which shows that gains from their method seem to improve/maintain with scale. While it would be interesting to see this method applied to pretraining dataset selection, but I understand this is out of scope for this work. The reviewers agree that this method is novel and interesting, and the experimental evidence is compelling, additionally the most negative review seems to ask for baselines that are known to not work well in the literature. For these reasons I vote to accept the paper.
Decision: Accept (poster)