Harnessing Diversity for Important Data Selection in Pretraining Large Language Models
Authors: Chi Zhang
Huaping Zhong
Kuan Zhang
Chengliang Chai
Rui Wang
Xinlin Zhuang
Tianyi Bai
Qiu Jiantao
Lei Cao
Ju Fan
Ye Yuan
Guoren Wang
Conghui He
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLMs
data selection
influence function
diversity
Primary area: foundation or frontier models, including LLMs

Abstract: Data selection is of great significance in  pretraining large language models, given the  variation in quality within the large-scale available training corpora. 
To achieve this, researchers are currently investigating the use of data influence to measure the importance of data instances, $i.e.,$ a high influence score indicates that incorporating this instance to the training set is likely to enhance the model performance. Consequently, they select the top-$k$ instances with the highest scores.  However, this approach has several limitations. 
(1) Calculating the accurate influence of all available data is time-consuming.
(2) The selected data instances are not diverse enough, which may hinder the pretrained model's ability to generalize effectively to various downstream tasks.
In this paper, we introduce $\texttt{Quad}$, a data selection approach that considers both quality and diversity by using data influence to achieve state-of-the-art pretraining results.
To compute the influence ($i.e.,$ the quality) more accurately and efficiently, we incorporate the attention layers to capture more semantic details, which can be accelerated through the Kronecker product. 
For the diversity, $\texttt{Quad}$ clusters the dataset into similar data instances within each cluster and diverse instances across different clusters. For each cluster, if we opt to select data from it, we take some samples to evaluate the influence to prevent processing all instances. Overall, we favor clusters with highly influential instances (ensuring high quality) or clusters that have been selected less frequently (ensuring diversity), thereby well balancing between quality and diversity.  Experiments on Slimpajama and FineWeb over 7B large language models demonstrate that $\texttt{Quad}$ significantly outperforms other data selection methods with a low FLOPs consumption. Further analysis also validates the effectiveness of our influence calculation.

Published as a conference paper at ICLR 2025
HARNESSING DIVERSITY FOR IMPORTANT DATA SE- LECTION IN PRETRAINING LARGE LANGUAGE MOD- ELS
Chi Zhang +1.3 Huaping Zhong"? Kuan Zhang', Chengliang Chai Rui Wang². Xinlin Tianyi Bais, Jiantao Qiu², Lei Cao", Ju Fan Ye Yuan', Guoren Wang', and Conghui He 3
Xinlin Zhuang?
Beijing Institute of Technology
Sense' Time Research
Shanghai Artificial Intelligence Laboratory
University of Arizona
Renmin University China
edu. zhonghuaping senset
sensetimeeeeeeeeeeeeeeiieiiiiiit ime. com caoleiBarizona.co
fan fanjerue.ede edu.
heconghui @p wrg.aa......o....o.oo org.en
ABSTRACT
Data selection is of great significance in pretraining large language models, given the variation in quality within the large-scale available training corpora. To achieve this, researchers are currently investigating the use of data influence to measure the importance data instances, i.e., a high influence score indicates that incorporating this instance to the training set is likely to enhance the model per- formance. Consequently, they select the top-k instances with the highest scores. However, this approach has several limitations. (1) Calculating the accurate in- fluence of all available data time-consuming. (2) The selected data instances are not diverse enough, which may hinder the pretrained model's ability gener- alize effectively to various downstream tasks. In this paper, we introduce Quad, data selection approach that considers both quality and diversity by using data influence to achieve state-of-the-art pretraining results. To compute the influence (i.e., the quality) more accurately and efficiently, we incorporate the attention layers to capture more semantic details, which can be accelerated through the Kronecker product. For the diversity, Quad clusters the dataset into similar data instances within each cluster and diverse instances across different clusters. For each cluster, we opt to select data from it, we take some samples to evaluate the influence to prevent processing all instances. Overall, we favor clusters with highly influential instances (ensuring high quality) or clusters that have been se- lected less frequently (ensuring diversity), thereby well balancing between qual- ity and diversity. Experiments on Slimpajama and FineWeb over 7B large lan- guage models demonstrate that Quad significantly outperforms other data se- lection methods with a low FLOPs consumption. Further analysis also validates the effectivenesss of our influence calculation. Our code and data are available Iht=ps://ancnymovi 4open. science. x /uadd/)
INTRODUCTION
Recently, large language models (LLMs) have significantly advanced the field of artificial intelli- gence (Zhao al. 2023 Hadi et al. 2023 Minace al. 2024). Due to the unprecedented number of parameters (model size) and the pre-training on huge amount of training data, LLMSs are gener- alizable broad spectrum of downstream tasks. However, in practice, the computation resources limit both the model size and the volume of data used in pre-training. In this situation, judiciously
"qqual Contribution.
"Correspondence to: {cc1 1 @bi t edu.. en, heconghui @pjlab. pejlab.org.cm org. This work done during the internship Shanghai Artificial Intelligence Laboratory.

---

Published as a conference paper at ICLR 2025
0.000
0.000
G
0.015
C
10
0.002
0.005
0.000
Co
G
C.
+0.005
-0.002
-10
-0.010
-20
00.00
high-influence
low-influence clusters
-20
-10
clusters
(a) Influence scores of data instances
(b) Influence scores in different clusters
Figure Distribution of influence scores of some sampled data instances.
selecting train datasets critical for producing highly performance LLMs (Brown 2020 Du et al.
2022 Gururangan et al., 2020 Zhang et al. [2024a] Hoffmann et al. 2022 Raffel et al. 2020 In particulan lar, the quality of the training datasets vary dramatically, while the LLaMA-3.1 report (Dubey
ct al. 2024) shows that the use of high quality data in later training stages greatly improve model performance.
Typical straightforward data selection approaches include rule-based data filtering clean the data (Raffel et al., 20200] Rae et al., (021). querying high-performance models (e.g., GPT-4) (Wet- tig et al. 2024] Sachdeva et al.) (2024), surrogate models (Lin et al., 2024 Shao al. 2024). elc. Although these methods have achieved success on some datasets and models, they rely on simple heuristics to select training data. Without explicitly measuring the impact of the selected data on the model, these methods tend to produce sub-optimal pretraining results. To address this issue, some researchers (Xia et al., 2024] Chai et al. [2023 Yu et al. 2024) start evaluating each data instance by assigning it a score that reflects its impact the model. Frequently used scoring methods include the influence function (Xia al. 2024 Liu et al., 202T), early loss (Albalak al., 2023 Deng al.), and perplexity (Chen et al. 2024). Among these methods, the influence function consistently delivers state-of-the-art results by effectively approximating the impact adding each instance to the training set. A higher score signifies higher priority for selecting data instance, and hence the
top-k (or gumble top-k) instances with the highest scores are chosen (Xie et al. 2023 Wettig et al., 2024
Yu et al. 2024).
However, the above methodologies have the following limitations.
Prohibitive Computation Cost. First, accurately calculating the influence score of one data in- stance is expensive, because it involves the computation of the Hessian matrix. However, in the LLM pre-training, the number of the candidate data instances is extremely large. It is thus pro- hibitively expensive to compute the scores for all of the candidates.
Lack of Diversity. Figure Ia] We can see that the top-k instances (e.g., some high-score instances in C1) tend to be closely distributed in the feature space because the influence computation closely related to the data features. That is, the training instances selected in this way are lack of diversity (e.g., other instances in C3 with high influence are also worth selecting), while confirmed by some studies (Abbas et al. [022 Tirumala et al. al. (002), diversifying training samples mitigates overfitting, thereby enhancing the generalizability of the model. Therefore, an effective data training selection method should take both the influence scores and the diversity into consideration.
Second, assume that all influence scores have been calculated, as shown in
We thus propose Quad, scalabe and effective data selection approach, which successfully ad- dressing above challenges, achieves state-of-the-ar pretraining results. Initially, Quad organizes the given dataset into clusters where the data instances within each cluster are similar, and those in different clusters exhibit diversity. Hence, the accurate average influence of the cluster, so as to represent the cluster quality (e.g. domain relevance) w.r.t the model performance.
sample data subset from cluster estimate
Next, leveraging the property of the attention based Transformer architecture which is widely adopted by the L.LMs, we design novel method accurately compute the influence of instance
2

---

Published as a conference paper at ICLR 2025
on LLM pre-training. More specifically, rather than solely relying the MLP layers to compute the influence (Koh & Liang/ 2017] Yu et al. 20224) Grosse et al. 2023: Engstrom et al. 2024). we incor- porate the attention layers such that the ifluunnee computation considers more semantic information. In addition, given that calculating the Hessian matrix is time-consuming, particularly for attention layers with complex interactions, incorporate the Kronecker product approximate the Hessian matrix, thereby greatly expediting the computation. This successfully addresses the computation cost challenge.
To improve diversity, we apply the Multi-Arm Bandit (MAB) technique, where each cluster is re- garded as an arm of the MAB. Upon selecting an arm, we draw samples from the cluster to calculate influence scores. Subsequently, Quad iteratively samples from clusters, taking into account both the influence score and data diversity, e-g., whether the cluster has already been sampled. Moreover, because this sampling strategy effectively avoids calculating the influence of all instances, it further speeds up the data selection process.
We summarize main contributions as follows:
To balance the quality and diversity, we incorporate an iterative MAB solution to first cluster the data instances and select data instances from these clusters.
We propose a novel method to compute the influence function in attention-based Trans- to preciscly measure the data quality in LLM pre-traie pre-training.
former architecture,
Experiments on the widely-used dataset Slimpajama, FineWeb and 16 popular downstream tasks demonstrate that Quad significantly outperforms state-of-art data selection methods by 1.39% in zero-shot accuracy (train and test on 1.3B and 7B models respectively), also with low computation resources consumption.
2 RELATED WORK
Rule-based Methods. Initially, researchers often relied on intuition to design hand-crafted heuris- ties (Soldaini et al., 2024) and (Penedo et al., (023). aiming to improve data quality. Deduplication another typical approach for selecting pretraining data, such as (Penedo al., 2023) and SemD- edup (Abbas et al. 2023) which use keyword- based and semantic deduplication, respectively. Ad- ditionally, certain approaches employ r-gram similarity (Gao et al., 2020: Xie et al. (023) to assist in choosing corpora that is semantically aligned with the validation set data (Chai et | (2022). Al- though these methods effectively filter out noise and redundant data from web sources, they rely on simple heuristics and cannot be well generalized.
LLM As a Selector. Although large models such as GPT-4 can effectively assess data quality due to their semantic comprehension capacity, the metrics utilized to rate data (e.g., writing style, educational value etc.) heavily rely on human intuition (Wettig et al.] 2024, Penedo et al. [024]] Zhang et al.] 2024b Gunasekar al., 2023 Peng al. 2025). This often leads to a mismatch between the selected data and the data desired by the model.
Surrogate Models. MeepSeekMath (Shao al. 2024 proposes active learning strategy train a web data classifier. Similarly, in MATES (Yu et al. 2024). a surrogate model was developed to estimate the influence scores of the data instances. RHO-1 (Lin et al. 2024) used surrogate model trained with high-quality data to perform token-level data filtering. However, these surrogate models are not trained over large-scale data, and thus their generalization ability limited.
Perplexity serves as a metric for selecting high-probability data in language model. In CChen al.) 2024 Marion al. 2023 Muennighoff et al., 2024, Wenzek al. (209)), perplexity (PPL) utilized to filter data. As also discussed in Qurating (Wettig et al. (024), we observe that this method often incorporates a significant amount of simple and redundant data, because they are easy for the model to predict.
2023 Choe et al. 2024) demonstrates that influence function can reveal the impact of training data on the performance of large models. Consequently, LESS (Xia et al. 2024) and MATES (Yu et al. 2024) utilize influence functions for selecting data during the SFT and pretraining phases, respectively. For large models, computing influence functions is com- putationally expensive. (Grosse et al. (222). Hence, given the large amount of data handled during pretraining, directly using LESS (Xia al. 2024) for data selection this stage poses considerable
Influence Function (Grosse et al.
3

---

Published as a conference paper at ICLR 2025
difficulties. To overcome this, MATES YYu et al. 2024) employs proxy model approximate the influence score across the full dataset. However, the limited capacity of this small proxy model hin- ders its ability to provide accurate influence scores. Furthermore, relying the influence to select data solely often leads to a lack of diversity in the chosen data.
METHODS
First, present problem statement in $3.1 Next, in $3.2 we explain how method achieves
the balance between quality and diversity in selecting pretraining data. Finally, in $3.3 we introduce how we compute the influence with attention layers more accurately and efficiently.
4. update Cluster Score
Clustering
1. sample from top-k clusters
3. select data
CS
high influence score
C52
Data Pool
1>shreshold
2. calculate influence score
Cluster Score(CS)
CS
$10.8.)
9(0,0)
00.8.
Js
exploitation
exploration
Module 1xx Drex
X iHVP
2InT
Attention
=s
CS,
1
((0,))
Figure 2: Overview of Quad
3.1 PROBLEM DEFINITION
In this paper, we study the problem of data selection from large pool D, of candidate data for pre-training large language model (LLM). Formally, given data pool De and a reference dataset D,., the problem of the updated model M' on the reference dataset D,. Note that our method is applicable to both pre-training M from scratch, and continuous pre-training scenarios where M starts as a pre-trained checkpoint.
select subset D, C De to pre-train an LLM M, aiming minimize the loss
3.2 BALANCE BETWEEN QUALITY AND DIVERSITY
As shown in Figure lb there are significant variations the distribution of influence scores among different clusters. To achieve the quality-diversity balance, it is necessary to know the precise av- erage influence score for instances each cluster. However, Figure Ib shows that the influence scores for each cluster also fluctuate around the average, indicating certain level of uncertainty. Estimating the average with small sample size will not be accurate enough, while taking a large number of samples to compute the average influence is costly.
Hence, we propose to use the MAB (Vermorel & Mohri 2005) technique that capable of making decisions iteratively under uncertainty. At high level, each cluster represents an arm of the MAB, and during each iteration, cluster with high average influence score tends to be selected and sam- pled. We will then compute the influence of data instances to update the average. Moreover, clusters that are not visited often present significant opportunities for sampling to balance the diversity.
The overall process of this approach is illustrated in Figure 2 Specifically, our method can be di- vided into the following four steps: First, we sample the top-k clusters with the highest cluster scores (denoted by CS) computed by MAB. Here, the cluster score is determined by both the fluence score and the sample frequency. Then we calculate the influence scores for the samples in each cluster (Section 3.3). At this point, we select high scoring samples to be added for training and use their scores update the cluster score for each cluster. Throughout the iterative process, the MAB algorithm focuses on frequently sampling high-quality clusters that have high influence scores, which also enhances the accuracy of their quality estimation (i.e., updating the average in- fluence ID). Simultancously, it ensures diversity by also sampling less- less-visited clusters. Next, we discuss how to compute and update the cluster score in details.
4

---

Published as a conference paper at ICLR 2025
Cluster Score (CS). The Upper Confidence Bound can effectively balance exploration (i.e., data diversity) and exploitation (i.e., data quality), so we use it as the cluster score to evaluate each clus- ter, as shown in Equation (I). Specifically, the cluster score is determined by the average influence score I, and the exploration score sampled from cluster C₁, and E T(C,) denotes the total times of samples taken from all clusters.
22 L, T(C)) where T(C;) denotes the frequency of instances T(C,)
2 InTT(O)) T(C,)
CS = -1=++ar
Update the cluster score. During each iteration, a subset of data B; is sampled from each cluster with high cluster (CS). The sum of their influence score I can be used denote the impact of the samples from the cluster C, on the model.
R(C+)+ = Σ 2DDr.), T(C)+=1
ZEB;
where R(C₃) denotes the total reward accumulated by cluster C, over several iterations.Ther Then the average influence score I for cluster C; can be represented as I, R(C,) As the sample size grows, 1 for each cluster Cs steadily approaches the exact average influence of the cluster, which can be used to update the cluster scores for all clusters.
T(C:)'
Data selection. During each iteration, we pick small proportion(7) of data instances from selected clusters. We also require that these instances have influence scores higher than the threshold т, otherwise we will not select them, which are then added into the training dataset.
Algorithm 1: Quad Algorithm
Input:
Candidate data pool De, reference set D,,, the model Output: Selected data D2
1 C = Cluster((D)):
2 while do
3
Crop top-k clusters with the highest Cluster Score(CS); Biop.t amini-batchi sampled from Ctop.k
for C. in Crop & do
R(C,) += ExEB, Io(D..2, T(C,)+= +=
7
end
for
C, in C do I, R(C:) T(C:) if 7, Whreshold then D,+ = 7C:
10
11
end
2hn Σ, T(G) T(C:)
CS, = T, + ov
12
end
14 return Dp:
3.3
INFLUENCE CALCULATION WITH ATTENTION LAYERS
Instead of retraining the large model with each data sample z, the impact of the model M can be estimated by calculating the influence function for each instance. this section, we extend the influence caleulation to multi-head attention layers and provide acceleration techniques.
1s (D
( (H LL)
In the above equation, ls (D,,,))) 2) denote the influence function of data model 0. L D,) and L(, 2) denote the gradient of reference set D, and data 2, respectively. Since the training of the large model does not often fully converge, resulting in non-invertible Hessian matrix H, regularization term AI is introduced (Bae et al.] (2022). Equation (3) is typically divided into the following two stages to speed up the computation:
5

---

Published as a conference paper at ICLR 2025
1. Approximate the multiplication of the gradient of the validation set VL(B, and the inverse Hessian matrix H using the inverse Hessian vector product (iHVP).
2. Compute the dot product between the iHVP and the gradient of each training data point (L(0,, 2).
While this framework can accelerate the computation the influence function, scaling it up to large language models (LLMs) with massive parameters is still expensive. Hence, K-FAC (Martens) & Grosse 2015 Ueno et al,, 2020) can be used to accelerate the iHVP computation by using the Kronecker product to decompose the Hessian matrix.
The K-FAC approximate the parameters of different MLP layer 0, 8g and 0g as independent. That's because, during the gradient computation and update process, there are usually only minimal direct dependencies between the gradients of different MLP layers. This is particularly evident during back propagation, where the weight updates for each MLP layer are primarily influenced by the parameters of that specific layer. Therefore, the influence function 02.0 (Dr, in K-FAC method can be expressed as:
(4) In attention mechanisms, there exist complex connections between the Query, Key, and Value layers. As the right-upper corner of Figure 33 shows, separately calculate the hessian matrix of Query, Key and Value layers, will miss massive information. Consequently, it is essential to consider the QKV layers as a unified layer Bykv when computing the influence function. Therefore, the influence function To (Dr, 2) can be expressed as:
02, (D,,,, = Le, (D,, + (D,,,) + Le, (D,,
Io [Dr, 2) Io (Dr, (D,,
(5)
Then, as the right-lower corner of Figure shows, by decomposing the Hessian matrix into
Add & Norm
Linear
HQ
Concat
Hk
HoKV
Feed Forward
Hy
Scaled Dot product Attention
D0
x
DO ghr
Add & Norm
6,
6,
8
x
6
6,
8.
Multi-head Attention
Wo
WK
Wv
8
6
6
6.
A
Figure 3: Kronecker Product in calculating iHVP
kronecker product of smaller matrices and computing the inverse of each smaller matrix, we can avoid directly inverting the entire Hessian matrix, significantly reducing computational cost, and accelerate this process:
Forward Propagation:
QKT Vdz
Allention Q K, V) = so soflmar
(6)
Backward Propagation:
Do = vec( (De) 80
(7)
Here, & denotes the Kronecker product, and vec() represents the vectorization operation. Thus, the gradient of 9 gku can be written
Tvec(D (eecDWWar Q) vee DW K) vec ( DWv) 8
D8gku
(8)
[6g 6g 6
Let 6yge
Then, the Hessian matrix Hako can be estimates by:
Aite E(D0 Do 7) F. ykv Tik
(9)
E(dyro (phoo) 2 E( (to)) Aguu 8 Xakev

---

Published as a conference paper at ICLR 2025
Also, Ho Do @ Xo. Thus, the iHVP of the attention layer can be estimated follows:
Hate Batt
H
qkv V
Vgkv
AQkv 8 xgkk) Vggvv
H Vo
(Ao @ xx) -vv
(10)
- gkv
(4 -1
gkv
Vakv
vec(A A -1
gkv qkv ggvv)
(A & X
Vo
vec A xxxo0
where Vatt, Egke, v represent the gradient of reference set D, on parameters Putt, gyku 8o. respec- tively. Thus, the influence score of attention layers can be written as: To -VI 2) H Vatt-
To avoid the excessive memory usage of validation set gradients, we apply the Johnson- Lindenstrauss Lemma JJhnson Lemna[ohnson (1984) to reduce the dimensionality of both the iHVP computation results and the training data gradients =)), We achieve this by projecting Har Vatt onto a d-dimensional space, resulting Q" Hat Patt- In the Lemma, each element of Q is drawn from R Pxd, with P representing the original dimensionality and d denoting the reduced dimensionality.
4
EXPERIMENT
4.1 EXPERIMENT SETUP
Dataset Preparation. We use the entire 627B-token SlimPajama dataset (Soboleva et al. 2023 and 15000B-token FineWeb dataset (Penedo et al.) 2024) the candidate pool De. In the clustering process, the BAAI wwtpl//eeaageeeeeeeee input data, and approximately 600 million data points from the candidate pool D. clustered into 10,000 groups using the k-means algorithm. We use LAMBADA (Paperno et al.] (016), Open- webmath (Paster et al. (202)) and FLAN (Chung et al.
5 model is employed to generate embeddings for the
2024) as our reference (validation) set D,,
which are all widely used language modeling tasks and
often serves as a validation benchmark for
language model pretraining. et al. 2024
Xie al.
2023 Hoffmann al. 2022
Experimental Settings. We train two transformer-based decoder-only language models, one with 1.3B parameters and the other with 7B parameters, both utilizing RoPE embeddings (Su et al. 2023) and a maximum context window of 2048 tokens (Touvron et al. (023). Following the setting of MATES (Su et al., 2023), 30B tokens out of the 627B are selected for training using Quad and compare with baselines. The learning rate is set to 5 x 10-5 for 1.3B model and x 10-5 for 7B model, the batch size is set to 4096, and the Adam optimizer is employed with hyperparameters 31 = 0.9, B2 0.95, 10 8, As for Multi-Armed Badit, we set the = 0.002, sample proporation 0.05 and the sample threshold т as 0.0025.
Baselines. We compare methods with several baselines. (1) Random samples data from the entire candidate dataset randomly. (2) Qurating uses the large language model to select data. (3) DSIR selects data instances that are similar to the LAMBADA, Openwebmath FLAN dataset. (4) PPL uses perplexity- based data selection, é.e., selecting data instances with the lowest perplexity scores. (5) MATES trains surrogate model to evaluate the influence of each data instance the target model.
Evaluation Datasets. To comprehensively evaluate the capabilities of pretrained models, we duct experiments on various downstream tasks covering three significant categories:
General Knowledge: ARC-C, ARC-E (Clark et al. 2018). and SciQ (Welbl et al. 2017)
Commonsense Reasoning: HellaSwag
(Zelerrs et all
2019). SIQA (Sap et al. 2019).
Wino-
Grande (Sakaguchi et al.) 2021), Logiqa
Liu et 2020).
Reading Comprehension: OpenbookQA
MMhaayoo et al. 2018), and BoolQ (Clark et al. 2019
Math: GSM8K (Cobbe al. 2021). MATH (Hendryeks et al. 2021). OCW (Lewkowyez al.
2022 SAT (Azerbayev et al. 2023). and MMLU STEM (Hendrycks et al. 2020)
Long-text Generation: WikiText (Merity al. 2016), and HelloBench Que at 2024
Evaluations are conducted using the ineevlluaioonhaanes. (Gao et al. 2023 framework and the average accuracy (i.e., Overall Score) is reported for comparison.

---

Published as a conference paper at ICLR 2025
Table Overall Performance
General Knowledge (3 tasks)
Commonsense
Reading Comprehension (2 tasks)
Selection Method
Reasoning (4 tasks)
Overall
FLOPs
Random
50.33
36.19
39.09
41.55
7.66
DSTR
50.37 10.01
34.01
42.18
38.80
11.29
40.53
41.02
7.66
PPL
48.71
$1.62
37.72
11.53
38.57
J0.52
41.57
10.02
9.51
Semdedup
50,99
10.66
36.
10.08
39.44
41.81
10.26
8,
Qurating
51.56
35.93
J0.26
39.70
10.61
42.01
10:46
13.66
MATES
50.45
70.12
36.06
10.13
39.83
10.74
41.93
10.38
9.81
Quad(ours)
52.08
11.75
37.03
10.84
41.07
42.94
9.15
MAB
Random
Threshold
Overall score
43
Random
Overall score
42.5
Correlation
Overall score
43.0
42.0
42.5
42
41.5
41
150
1000
No-Hess.
No-Hess.
Indep.
1.5e-3
2.5e-3
(a) MAB vs. top-k clusters
(b) Influence accuracy
(c) Influence correlation
(d) Sample threshold т
Figure 4: (a) shows the effectiveness of the MAB method; (b) shows the accuracy of calculating the influence function on MLP and attention layers; (c) shows the correlation between Query, Key, Value layers impact lot on the accuracy of influence calculation; (d) shows the model performance of varying sample threshold т.
4.2 RESULTS
Overall Performance. As demonstrated in Table our method surpasses all the baseline meth- ods in downstream tasks with zero-shot evaluation. To specific, we can observe that on General Knowledge and Reading Comprehension tasks, Quad has the improvement of 1.75% and 1.98% respectively compared with Random. Quad outperforms DS OSIR and Semdedup because they use rule-based heuristics to select data without considering the model. Although PPL and MATES con- sider the model, they do not perform well because the former one always selects some simple and duplicated instances, and the surrogate model of the latter one is small and lacking of enough train- ing data. gurating generally performs the best among other baselines, but still worse than our approach, and it incorporates the highest FLOPS((el9)) because of the usage of L.MMs for data selec- tion. In terms of the FLOPs, we can observe that except the methods (i.e., DSIR, SemDe up) that use simple heuristics, we consume minimal computation resources because we sample from clusters without considering the entire candidate dataset like PPL, Qurating and MATES.
Effectiveness of MAB. This section evaluates the effectiveness of the MAB approach for data lection in contrast to the straightforward method of choosing the top-k clusters with the highest influence scores for model training. To be specific, we randomly select an equivalent number of data points from the top 150, 500, and 1000 clusters. Figure 4a illustrates the trade-off between data quality and diversity: clusters with higher influence scores do not necessarily enhance model performance downstream evaluation sets because of their lack of diversity. Hence, the multi- armed bandit method can more effectively capture the trade-off between quality and diversity across clusters, resulting in superior performance, opposed to merely choosing the top-k clusters.
Effectiveness of Influence Calculation. This experiment studies the effectiveness of our influence calculation method. In this section, we select the top 500 clusters with the highest scores using three methods: (1) soo-Hessian (i.e., computing the gradient similarity between training data and reference data (Pruthi al. 2020)) without considering the Hessian matrix; (2) MI.P(i.e., calculat-
8

---

Published as a conference paper at ICLR 2025
12
Ransam
Random
30
42.0
Ausurayy (36)
43.0
62.0
62.5
1000 rodo
1000 2000 5000 1000
4 RAMAANS DESGAN
GVM
Venntben
5000 10000 41.5
(a) Training loss
Validation loss
(c) Clustering algorithm
(d) number of clusters
Figure 5: (a) training loss; (b) validation loss; (c) model performance on varying clustering algo-
rithm; (d) model performance varying number of cluster
ing influence function on MLP layers) and (3) Ours (i.e., calculating influence function on both MLP and attention layers). From each cluster, uniformly sample data to train the large model. As shown in Figure 4bb our solution (MLP -AtttAttetttt ion) performs better than MLP because the attention layer considers more semanties. no no-Hessiare performs the worst because does not precisely capture the impact of training data instances on the model without the Hessian matrix.
Also, we conduct experiments verify the relationship between the Query, Key, Value matri- ces, which is shown in Figure 4c] We compare Pearson correlation coefficients of three meth- ods with a baseline that computes the attention layer's influence score without acceleration. (1) No-Hess sss. ani..., computing the gradient similarity between training data and reference data) without considering the Hessian matrix; (2) Independent. (i.c., calculating the Hessian matri- ces of the query, key, and value layers independently) and (3) Ours (i.e., calculating the Hessian matrices the query, key, and value layers as a whole).
4.3 CONTINUOUS PRETRAIN
In this experiment, we Slimpajama (Soboleva al. 2023) the candidate dataset and Open- webmath (Paster et al.
2023) as the reference set on 7B model. For downstream tasks, we use
GSM8K (Cobbe al. 2021). MATH (Hendrycks al. 202I). OCW (Lewkowycz et al. 2022), SAT (Azerbayev et al.
2023), and MMLU STEM (Hendrycks et al. (020), calculating accuracy selecting 400B random data instances performs worse than 100B selected by Quad because the larger set has irrelevant information, degrading the performance and Quad improves selection by identifying beneficial data for the specific domain.
by averaging their scores. As shown in Figure 7
4.4 FURTHER EXPERIMENT
Generalizability of Quad To demonstrate the generalizability and robustness of our method, we include FLAN (Chung et al. 2024) (a mix of multiple NLP tasks) as reference datasets We use FLAN to train a transformer- based, decoder-only language model with 1.3B parameters on new candidate dataset, Web (Penedo et al., 2024). From this dataset, 100B tokens are selected for training using Quad alongside other baselines. As shown in the Figure Quad exhibits superior accuracy compared to all baselines like DSIR, Semdedup. PPI., Qurating. and MATES. Notably, Quad shows 0.97% accuracy improvement over Qurating, leading baseline.
Scalablity of Quad. We increase the model size from 1.3B to 7B and choose 100B tokens from Slimpajama dataset. As shown in the Figure a),,, Quad outperforms other baselines in accuracy due to high-quality, diverse data selection. Additionally, Quad demonstrates good scalability com- pared to state-of-the-as th-the-art baselines (Qurating and MATES) by using the K-FAC technique for faster influence computation and the MAB framework to quickly identify beneficial data instances.
Effciency of Quad. To highlight the importance of data selection, compare Quad and direct training with clean data on 7B modes for downstream tasks. As shown in the Figure [ Quad- 7-100B surpasses Clean-7B-1000 because Quad selects data beneficial for the current training process, unlike the latter's random selection from SlimPajama. At the same time, Quad-7B-100B offers performance comparable to Clean-7B-400B, reducing computation by 75%.
4.5 ABLATION STUDY
Sampling Threshold of Influence (r). Figure 4d illustrates that setting the threshold too high or low will both degrade the model performance. This is because the selected data instances tend to

---

Published as a conference paper at ICLR 2025
Semdedup
Qureting
38
49
53
37
48
Accuracy (g))
Arcuracy
35
46
51
34
33
49
32
43
(a) LAMBADA-78
(b) Openwebmath-75
(c) FLAN-1.38
Figure 6: (a) 7B model use LAMBADA reference set; (b) 7B model use Openwebmath as refer-
ence set; (c) 1.3B model use FLAN as reference set
43.5
43.5
54.3
1200
37.5
1200
FLOPs
FLOPs
1000
36.3
43.0
43.0
52.5
Accuracy(S))
Accuracy/%%
35.5
800
51.5
42.5
50.3
42.0
42.0
49.5
46.5
60h
(a) Influence threshold
(b) Exploration weight a
(c) Effciency of Quad
(d) Continuous pretrain
Figure 7: (a) model performance with different influence threshold r; (b) model performance with
different or; (c) the effciency of Quad; (d) continuous pretrain
exist in few clusters with high influence scores, resulting in poor diversity. In contrast, when the threshold is set too low, the sampled instances will be from many clusters with low influence scores, which also degrades the model performance. Considering that the model's performance is highly sensitive to the influence threshold, a quick selecting method proposed in Appendix F.
Number of Clusters We use the Elbow (Syakur et al. 2018) method identify the optimal cluster number(k: 10000) of the Slimpajama dataset. Figure 5c We plot the effect of different cluster numbers, which shows the model consistently performs well with around 10000 clusters. Too few clusters (i.c., = 1000) cause high variance and poor representation, while too many clusters (i.c., k 500000) result in redundancy and hinder diverse exploration, degrading performance.
Clustering Algorithm Moreover, we evaluate the performance of several typical clustering methods including GMM (Figueiredo & Jain 2002). DBSCAN (Ester et al. 1996), and MeanShift (Cheng, 1995). The details of selecting optimal clustering parameters can be found in the Appendix D. As illustrates in Figure 5d Quad is robust clustering algorithms downstream tasks.
for Quality-Diversity Balance. Our approach employs to balance the diversity and quality in the MAB framework. As shown figure 7b when a is small, the MAB framework prioritizes hhghh-iggh high-influence clusters and risks local optima due to reduced diversity. Conversely, when is large, overemphasizes diversity the expense of quality, limiting model performance gains.
CONCLUSION & LIMITATIONS
This paper presents Quad, a method balance diversity and quality in pretraining data selection. Quad employs influence functions to identify beneficial data by clustering and selecting represen- tative subsets. Each cluster is treated as an arm in an MAB framework due to uncertain influence scores, allowing sampling from quality clusters to estimate influence scores accurately while main- taining diversity. We also adapt the influence function for attention layers and enhance calculation efficiency for better data impact assessment. As exploratory research work, the ability of Quad to generalize to larger model sizes needs to be further explored, which a challenging and significant issue for the method to be used in practice.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Ju Fan supported by the NSFC (62436010 and 62441230), the Beijing Natural Science Foundation (L244010 and L222006), and the Research Funds of Renmin University of China. Chengliang Chai supported by National Key Research and Development Program of China (2024FF33302000, NSFC (62472031), CCF-Baidu Open Fund (CCF-Baidu22200)) and Huawei. Ye Yuan is supported by Beijing Natural Science Foundation (1.241010), the National Key R&D Program of China (Grant D1.2022YFB2700 100), and the NSFC (Grant Nos. 61932004, 62225203, U21A20516) Guoren Wang is supported by the NSFC (62427808, U2001211) 1), the Liaoning Revitalization Talents Pro- gram (Grant No.XLLC2204005)

---

Published as a conference paper at ICLR 2025
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint a0x5522303.09540, 2023.
Ken Aho, DeWayne Derryberry, and Teri Peterson. Model selection for ecologists: the worldviews of aic and bic. Ecology, 95(3):631-636 2014.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
P Auer. Finite-time analysis of the multiarmed bandit problem, 2002.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.1063 2023.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953 1591799--7967. 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv: 14165, 2020.
Chengliang Chai, Jiabin Liu, Nan Tang, Guoliang Li, and Yuyu Luo. Selective data acquisition in the wild for model charging. Proc. VLDB Endow. 15(7):14661144 2022. doi: 10.14778/ 3523210.3523223. URL https::/:
vldb. http:///wwwoooo////////////e/e//
Chengliang Chai, Jiayi Wang, Nan Tang, Ye Yuan, Jiabin Liu, Yuhao Deng, and Guoren Wang. Efficient coreset selection with cluster-based methods. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (eds.), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA. USA, August 6-10, 2023, pp. 167-178. ACM, 2023, doi: 10.1145/ 3580305.3599326. URL http:: dow. org/ 10. 1141111448888 3599326
Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, et al. Towards effective and efficient continual pre-training of large language models. arXiv preprint arXiv:2407.18743 2024.
Yizong Cheng. Mean shift, mode seeking, and clustering. JEEE transactions on pattern analysis and machine intelligence, 17(8):790-79,, 1995.
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, al. What your data worth to gpt? scale data valuation with influence functions. arXiv preprint arXiv:2405. 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instructionfiintte lan- guage models. Journal of Machine Learning Research, 25(70):1 13,, 2024.
Christopher Clark, Kenton Lee, Ming-
Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try the ai2 reasoning challenge. arXiv preprint 1803.00533 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1416 14168, 2021.
12

---

Published as a conference paper at ICLR 2025
Yuhao Deng, Chengliang Chai, Lei Cao, Nan Tang, Ju Fan, Jiayi Wang, Ye Yuan, and Guoren Wang. Misdetect: Iterative mislabel detection using early loss.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-expert. In International Conference Machine Learning, pp. 5547- 5569. PMLR, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Logan Engstrom, Axel Feldmann, and Aleksander Madry, Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pp. 226 226-231, 1996.
Mario A. T. Figueiredo and Anil K. Jain. Unsupervised learning of finite mixture models. IEEE Transactions pattern analysis and machine intelligence, 24(3):381-392 2002.
Gao, J Tow, Abbasi, S Biderman, Black, DiPofi, C Foster, I. Golding, Hsu, A Le Noac'h, et al. A framework for few-shot language model evaluation, 12 2023. URL https://zeeodo. org/records/112566866 7, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 00027 2020.
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions, arXiv preprint arXiv:2308.033296 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 2004:, 2020.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Scyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 000:20009.03300. 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of computc-optimal large language model training. Advances in Neural Infor- mation Processing Systems, 35:30016-30033 2022.
William B Johnson. Extensions of lipshitz mapping into hilbert space. In Conference modern analysis and probability, 1984, pp. 189-206, 1984.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pp. 1885-1894, PMLR, 2017.
13

---

Published as a conference paper at ICLR 2025
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke -Sloo et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.0796. 07966, 2024.
Jiabin Liu, Fu Zhu, Chengliang Chai, Yuyu Luo, and Nan Tang. Automatic data acquisition for deep learning. Proceedings of the VLDB Endowment, pp. 2739- Jul 2021. doi: 10.14778/ 3476311. 1,.47633..33... URLhttpp://// /t/.:/c.....oo0/00117///////1110 .11.3111..373333
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08/ 00.00007.8//444 2020.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining ilms at scale. arXiv preprint arXiv:2309.04566 2023.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp.
2018.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint 46X1v:2402.06196, 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alcksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: 2016.
Word prediction requiring a broad discourse context. arXiv preprint
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arX:v:2310.06786, 2023.
Guilherme Penedo. Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16, 2023.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.11 17557, 2024.
Jiahui Peng, Xinlin Zhuang, Qiu Jiantao, Ren Ma, Jing Yu, Tianyi Bai, and Conghui He. Unsu- pervised topic models data mixers for pre-training language models, 2025. URL https:
//arxiva farxiv.org/al arb. 2202. 16802
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
14

---

Published as a conference paper at ICLR 2025
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang. Jian Yang, Ge Zhang, et al. Hellobench: Evaluating long text generation capabilitiies of large language models. arXiv preprint arXhv:2409.16191, 2024.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 67, 2020.
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms. arXiv preprint wxXiv:2402.09666 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 2019.
Language Processing (EMNLP-IJCNLP),, pp.
Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS), 42(3):1-22) 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, SlimPajama:
Jacob R Steeves, Joel 627B token cleaned and
Hestness, and Nolan deduplicated version of
Dey.
A
RedPajama.
https///errbraa.
slimpaj jamaajaaaaaaesa a- token cleaned and o redpaa oo/////ttett//errrrra///11aaanap
2023. URL htttps::/hugginngaaac.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining rescarch. arXiv preprint arXiv:2402.00159 2024.
J Su, Y Lu, S Pan, A Murtadha, B Wen, and Y Liu Roformer. Enhanced transformer with rotary position embedding.. 2021. DOI: hutp:://di. org/10.1016/j. neucom, 2023.
Muhammad Ali Syakur, B Khusnul Khotimah, EMS Rochman, and Budi Dwi Satoto. Integration k-means clustering method and elbow method for identification of the best customer profile clus- ter. IOP conference series: materials science and engineering, volume 336, pp. 012017. IOP Publishing. 2018
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving I1m pretrain- ing via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36:53983-53999 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0m 092888, 2023.
Yuichiro Ueno, Kazuki Osawa, Yohei Tsuji, Akira Naruse, and Rio Yokota. Rich information is affordable: A systematic performance analysis of oedond-order optimization using k-fac. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2145-2153, 2020.
15

---

Published as a conference paper at ICLR 2025
Joannes Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical evaluation. In European conference on machine learning, pp. 437-448. Springer, 2005.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprini wrX.vv:707.062000 2017.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint 1..0v99 2019.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. In Forty-first International Conference Machine Learning, 2024. URL httpttts///ppereeeee
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning, 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint arXiv:2406.06046, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Meihui Zhang, Zhaoxuan Ji, Zhaojing Luo, Yuncheng Wu, and Chengliang Chai. Applications and challenges for large language models: From data management perspective. In 40th IEEE International Conference
Data Engineering, ICDE 2024, Utrecht, The Netherlands, May 13- 16, 2024, pp. 5530-5541. IEEE, 2024a. doi: 10.1 16.1099/CDEE6114222444400. //doi. org.or////// 10. 11 /9/ TCDE601 46. 2024 00441
URL https:
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C Yao. Autonomous data selection with language models for mathematical texts. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303, 18223, 2023.
16

---

Published as a conference paper at ICLR 2025
A DETIALED RESULT
Table 2: Performance Comparison
Selection Method
General Knowledge
Commonsense Reasoning
Reading Comprehension
Overall
arcie
arc-c
sc34
logiga
hellaswag
siqa
winogrande
openbookga
boolg
Rundom
Semdedup
51.35
20.73
80.90
50.99
19.05
34.56
39.30
$1.54
36.11
18.80
60.09
39.45
41.81
MATES
38.69
PPL.
45.41
20.82
79.90
48.71
20.43
35.92
39.92
54.62
37.72
18.80
$8.35
38.58
41.57
Qurating
52.10
23.29
79.80
51.56
20.43
33.57
39.05
50.67
35.93
18,00
61.41
39,71
42.04
Quadi
topi-claster
isp-150
42.04
top-500
51.015
21.25
79.70
50.67
22.73
34.40
39.2 39.20
52.41
37.19
18.80
62.76
40.78
42.48
top- 1000
Table 3: Ablation Study of Threshold
Threshold
General Knowledge
Reasoning
Reading Comprehension
Overall
arC-C
seig
logiqa
hellaswag
winogrande
openbookqa
boolg
0.0015
51.26
21.16
80.20
50.87
21.51
33.92
39.00
$1.07
36.38
19.60
61.74
40.67
42.16
22.27
80.70
50.20
40.35
42.45
0.0025
52.27
21.76
82.20
52.08
22.89
34,41
38.74
52.09
37.03
20.00
62.14
41.07
42.94
0.0030
50.25
19.62
80.80
50.22
22.27
33.96
38.96
53.28
37.12
20.60
59.20
39.90
42.10
Table 4: Effectiveness of Influence Calculation
Method
General Knowledge
Commonsease Reasoning
Reading Comprehension
Overall
scig
logiqa
hellaswag
siqu
winogrande
openbookqa
boolg
Random
50.27
20.31
80.40
50.33
21.20
34.11
38.49
50.99
36.20
17.60
60.58
39.09
41.55
No-Hessian
49.03
20.99
80.50
50.20
22.58
33.40
38.89
52.41
36.82
19.20
61.50
40.35
42.06
MLP
50.63
21.50
78.90
50.34
22.89
33.32
38.74
$2.57
36.88
19.60
61.77
40.69
42.21
Ours
51.05
21.25
79.70
50.67
22.73
34.40
39.20
52.41
37.19
18.80
62.76
40.78
42.48
Table 5: Model Architecture
Hyperparameter
Value
Vocabulary Size
32,000
MLP Ratio
8/3
Hidden Dimension Size
2048
Number of Layers
24
Number of Attention Heads
16
Number of KV Attention Heads
RoPE Base
10,000
Maximum Context Window Length
1024
Number of Parameters
1 1.345,4233366(1.3B)
B EFFECTIVE SOLUTION SUITABLE SAMPLE RATIO
For this hyperparameter, explore the extreme cases in the 100B scenario.
[Extreme cases.] The extreme cases of y are very small and large sampling ratios. As shown in the table below, very small sampling ratio (i.e., 7 0.1%) leads to low model accuracy (2.77% lower than the accuracy the most appropriate sampling ratio, i.c., 5%) due to the inaccurate estimates of influence scores. We observe that large sampling ratio (i.e., 7 20%) does not improve the model performance much because a relatively accurate influence estimation is sufficient, but incorporates high computational costs.
17

---

Published as a conference paper at ICLR 2025
[Metric and Criteria.] At high level, we compute the most appropriate sampling ratio by sampling several clusters. For each sampled cluster, we compute the influence scores of its data instances and calculate the true average score. Then, we use the difference between the estimated score and the true score as the metric to select the best sampling ratio т. The high-level idea is to compute the true average influence scores of several sampled clusters and then identify the appropriate sampling ratio to accurately approximate the average.
(Specific hypermarameter selection strategy. Specifically, after clustering the data instances in the candidate dataset, we randomly sample several clusters. Initially, we sample 1% data instances from the cluster and compute the average score. if the difference within 10%, we set this proportion (1%) as the suitable sampling ratio for that cluster. Otherwise, we increase the ratio by 1%, to 2% compute new average, and repeat this process until the difference within 10%. Finally, we average the suitable ratios from the sampled clusters to find the overall appropriate sampling ratio.
In this way, although we sample once, there would be fairly accurate estimation of the average influence score. Since we are likely to sample multiple times, the estimation will be even more accurate. Besides, as data instances within each cluster exhibit similarity, by sampling a small fraction, we can have an estimation that is precise enough to select high-quality data for good model performance, thereby keeping computational costs low.
In experiments, based the Slimpajama dataset, have determined that the optimal ratio is about 5%, as indicated the paper. Here, we add another ablation study to demonstrate the effectiveness of this strategy. Specifically, algorithm respectively, and report the model accuracy training with the selected data using the sampling ratio. We can observe from the below table that with the ratio increasing from 1%, the model accuracy increases because the influence estimation is more accurate. When the ratio exceeds 5%, the accuracy remains stable because that ratio is large enough have an accurate estimation. Hence, it is not necessary to keep increasing the ratio, which will consume more computational costs.
increase the sampling ratio from 1 1,, run the Quad
Table 6: Ablation Study of Sampling Ratio
Sampling Ratio
0.1%
1 1%
2 %
3 %
4 %
5.5
6.%
7 7%
8 %
9%
10 %
20 %
Accuracy
44.97
46.11
46.92
47.36
47.63
47.88
47.93
47.96
47.97
47.97
47.97
47.99
C STATISTICAL SIGNIFICANCE TESTS
We run the experiment six times and report the statistics significance tests. The results are as follows:
Table 7: Statistic Significance of Model Performance
Exp-1 1
Exp-2
Exp-3
Exp-4
Exp-5
Exp-6
Avg + std
Random
41.52%
41.32%
41.68%
41.72%
41.57%
41.55%
41.56%+1.113%
DSIR
40.59%
40.47%
40.72%
40.29%
40.37%
40.53%
40.49%
PPL 41.35%
41.72%
41.55%
41.60%
41.61%
41.57%
41.57%:10.119 %
Semdedup
41.60%
41.94%
41.87%
42.04%
41.91%
41.81%
41.86%-40.14%
Qurating 42.03%
41.89%
41.81%
42.21%
42.13%
42.01%
42.01 +.13%
MATES 42.10%
41.76%
42.20%
41.70%
41.81%
41.93%
41.92%%+10.18%
Quad(ours) 43.11%
42.91%
43.03%
43.02%
42.87%
42.94%
42.08%=+0.08%
Based on the above results, we perform significance test utilizing t-tests. We make the following two hypothesis. (I) Null hypothesis (Ho): The average accuracy of Quad does not exceed the av- erage baselines; and (2) Alternative hypothesis (H1)): The average performance of Quad exceeds that of the baselines. We set the confidence level as 99%, with significance level of 0.01. The p-value represents the probability of observing the test statistic under the assumption that H, is true. A small value indicates lower likelihood of observing the current result if Hn holds.
For each baseline, we can observe p < a, and thus we reject Ho and accept H,, indicating that the average performance of Quad is greater than that of baselines. Since the confidence level of
18

---

Published as a conference paper at ICLR 2025
Table 8: T Test of Different Selection Methods versus Quad
Llest
Random
DSIR
PPL
Semdedup
Qurating
MATES 2e-07
p-value
7e-10
12
2e-10
le-08
4e-08
I-test surpasses 99%, we can conclude that the superiority of Quad compared to other baselines statistically significant.
D ABLATION STUDY OF CLUSTERING NUMBERS AND ALGORIHTMM
In this part, we conduct experiments to explore extreme cases of selecting 100B data.
[Extreme cases.] As shown in the table below, very small number of clusters (i.c.. k = 100) leads to poor accuracy (2.77% lower than the accuracy of the best cluster number) due to the high variance of influence scores in each cluster. Thus, the sampled instances cannot well represent the cluster. Similarly, when the cluster number too high (i.e., k = 1, 000, 000), there will be many clusters that are in fact contain similar data instances, and thus it is relatively hard to explore diverse clusters, thereby leading to the performance degradation (1.37% lower than the accuracy of the best cluster number).
[Metric and Criteria.] We use the metric Within-Cluster Sum of Squares (WCSS) to select the best cluster number using the well-known Elbow [1] algorithm. WCSS is the sum squared distances between each data instance and its cluster center, i.e., WCSS= Ezec; || - -4 ||- a high level, the criteria should be that within each cluster, data instances are close to each other, based on which it is better for different cluster centers to be far away from each other. Based on the criteria, the Elbow algorithm leverages the WCSS as a mcasurement to iteratively select appropriate cluster number, as follows.
[Specific hypermarameter selection strategy.] To be specific, Elbow begins with small k and with increasing, WCSS first decreases rapidly and then slows down. Then, we identify the "elbow point" where the decreasing rate becomes slow the best k. Thus, within each cluster, data points are sufficiently close to one another. Furthermore, given that k remains modest, different cluster centers tend to maintain distance from each other. From the following table we can observe that the model consistently performs well when the cluster number is close to 10,000.
Table 9: Ablation Study of Cluster Numbers
Cluster Numbers Accuracy
100
1,000
5,000
10,000
20,000
50,000
100,000
1,000,000
45.11 %
46.93%
47.21%
47.88%
47.73%
47.73%
47. 47.06%
46.51%
Clustering algorithms. In terms of the clustering algorithms, we also added experiments show that Quad is not sensitive to clustering algorithms mainly because different algorithms have their own strategies to select appropriate parameters, which follows the criteria mentioned above. Under the criteria, in general, Quad can perform well by considering both the quality and diversity.
evaluate the performance of several typical clustering methods including GMM (Figueiredo & Jain 2002), DBSCAN (Ester et al., 1996) and MeanShift (Cheng 1995). Considering that the clustering results are affected by the parameters of clustering algorithms, we use diffeernnt methods to select proper parameters. For GMM, we can use the AIC score (Aho et al.) 2014 to determine the appropriate number of components. For DBSCAN, there are 2 key param- eters: (1) eps(the radius of a neighborhood w.r.t. some data points) and (2) Pts (a data point considered as a core point if at least minPts data points are within eps of it). They can be set using the method in (Schubert et al. 2017). Mean-Shift is centroid-based method that updates the centroids to be the mean of the points within a given region. The size of the region is controlled by bandwidth, which can be set by the estimation of the bandwidth. In this set of experiments, the experimental settings of pretraining 1.3B model are consistent with those reported in our paper. The above result shows that Quad is not sensitive to the clustering algorithms.
Specifically,
19

---

Published as a conference paper at ICLR 2025
Table 10: Ablation Study of Clustering Methods
Clustering Methods
K-MEANS
DBSCAN
GMM
MeanShift 47.71 %
Accuracy
47.88
47.79 %
47.69 %
E FLOPS CALCULATION
FLOPs is the number of floating point operations performed by GPUs. Many state-of-the-art meth- ods [1,2.3] use it to measure the consumption of GPU computing resources. In our experiments, FLOPs collected directly the data selection process using the Python code: import torch
import torch.nn as nn from worrch.profi... import profile, ProftlerActivity
model = nn nn. Liner (1024, 512) c)
input.data torch. (128, 1024) 1024).cudal ()
with
profile (activities- (ww.feeeerrrrrrttitttyycooo
Profileercctiiiyy.CUAA. with.flops=True) pref:
model (input_data)
print (prof. key averages () table (sort by=oooos==""""s" row low.iimt==1))
F EFFECTIVE SOLUTION OF INFLUENCE THRESHOLD
Recap that the sampling threshold of influence (r) is utilized to determine whether data instances sampled from each cluster should be fed into LLMs for training. For this hyperparameter, we also explore the extreme cases in the 100B scenario.
[Extreme cases.] ] If is very large (e.g., 2.92e-3, close to the smallest influence score the candidate dataset), only data instances with high influence scores selected. This typically results in reduced diversity among instances, which in turn can negatively impact the model performance. enough (e.g., 1.76e-3, close to the largest influence score in the candidate dataset), many low-quality data instances with low influence scores are selected. This also hurts the model performance.
small
Table 11: Ablation Study of Influence Threshold
Influence threshold(7)
Bucket 1-th 46.87%
Accuracy
2.92e-3
2.60e-3
3-th 47.30%
2.47c-3
4-th 47.88%
2.38e-3
5-th 47.65%
2.17
6-th 47.19%
1.76e-3
10-th
45.26%
[Metric and Criteria.] We use the model performance on the validation (reference) set as the evalua- tion metrie. The criteria is that we sample some data instances from the candidate dataset. Based on the samples, we try different thresholds, evaluate the validation test and select the best threshold.
[Specific hypermarameter selection strategy:] The strategy of selecting an appropriate 7 consists of the following steps. (1) Considering the efficiency issue, we sample about 20% data instances to form a new candidate dataset from the original one to tune the parameter. (2) We cluster over the candidate dataset, sample some instances from each cluster and compute their influence scores, which are utilized to capture the distribution of influence scores. (3) We rank these scores in de- scending order and assign them to 20 buckets, the best one. For example, the first threshold corresponds to the highest influence score among the top 5% instances (the first bucket). The last threshold is the lowest influence score among all in-
to derive 20 thresholds, among which select
20

---

Published as a conference paper at ICLR 2025
stances of buckets. (4) Finally, select data instances via the Quad algorithm using the above thresholds, evaluate on the validation set, and select the best one.
G SCALABILITY OF QUAD
In this response, we first discuss the advantage of using Quad to select data instances. We then run experiments to validate the argument.
Overall, we agree with the reviewer that data cleaning is likely to enhance the model performance. However, the obtained data after cleaning is still large-scale, which is prohibitively expensive to train, but not always necessary. This is because a subset of judiciously selected data can achieve competitive or even superior model performance compared with training with all clean data. The reasons are two-fold. First, Quad is model-aware data selection method, which uses the data influence measure the impact of each data instance on the current training process, but coarse- grained data cleaning methods are not customized to the target tasks. Therefore, given a user- specific task, simply using all available data to train may not the optimal choice. Second, Quad also considers the diversity for data selection, and thus the selected small part of data instances that effective for the current pretraining process.
Overall, the model-aware data selection is of great significance because (1) it is customized to the reference set, which can much improve the model performance with a small number of data in- stances; and (2) the model can be trained over these selected small amounts of data, thus greatly improving the training efficiency.
Next, run two new experiments validate this.
First, we would like to clarify that the SlimPajama dataset used in our experiment has already un- dergone multiple rounds of coarse-grained data cleaning processes SSbollva et al. (2023). including short documents filtering, deduplication, minhash generation, etc. Therefore, to demonstrate the significance of the data selection method, compare the performance of Quad and directly train- ing with clean data with different data sizes with 1.3B and 7B models, evaluating on the long-text generation tasks mentioned in the last response. As shown in the below table, we can observe that Quad-113B--
outperforms Clean-1.3B-Il -00 because Quad judiciously selects data that ben- efits the current training process while the latter one just selects 100B data randomly from the clean SlimPajama dataset without considering the current model state. Because of the similar reason, Quad- d- 100B achieves comparable model performance with C1 11ea--133B-0000 which saves 75% of computation costs. We can also observe that training on 7B model has similar result.
Table 12: Model Performance and Flops with Different Method and Training Tokens
Methods
Accuracy
FLOPss
Clean-1.3B-100P
44.41 %
58.1
Clean-L.BB-400B
47.66%
232.5
Quad-1.3B-100B
47.88
72.3
Clean-7B-100B
49.73%
263.1
Clean-7B-400B
53.11%
1063.7
Quad-7B-100B
53.19%
307.6
Secondly, we add another dataset Openwebmath (Paster et al.) 2023) as our new reference dataset. is based on the Slimpajama dataset, which aims at selecting data to improve the mathematical skill of the model, i.e., a specific domain. We use GSM8K (Cobbe al. 2021), MATH (Hendrycks al.) 2021). OCW (Lewkowycz et al., 2022). SAT (Azerbayev et al. (023), MMLU STEM (Hendrycks) et al. 2020) as the downstream tasks and the final accuracy is computed the average score these downstream tasks. As shown in Table 12, we can observe that random selection of 400B data instances does not perform better than selecting 100B by Quad on downstream mathematical eval- uation datasets. This is because the 400B data instances contain significant amount of information irrelevant to mathematics. which degrades the model performance, while Quad is a model-awann method that accurately identifies the data that benefits the specific domain(i.e., the math domain).
21

---

Published as a conference paper at ICLR 2025
Table 13: Model Performance in Math Evaluation
Methods
Accuracy
Clean-7B-100B
33.20
C Clean-7B-4008
35.01 %
Quad-7B-100B
36.76
Moreover, we train model and evaluate all methods on the SlimPajama dataset (Soboleva al, 2023), using LAMBADA as the reference dataset. In this experiment, 100B tokens out of 627B are selected for training using Quad. As shown in table below, Quad still demonstrates superior model performance on 7B model compared to all baselines, such the DSIR, Semdedup, PPL, Qurating and MATES.
Table 14: Model Performance with a Larger Model Size
Methods
Accuracy
FLOPs
Random- 100B
49.73%
263.1
Random-400B
53.11%
1063.7
DSIR-100B
49.97%
263.1
FPL-100B
51.18%
325.3
Londedup-100B
51.97%
275.7
Qurating-100B
52.18%
375.6
MATES-100B
51.97%
313.3
Quad-100B
53.19%
307.6
Note that our CS function is balance the exploration (i.e., diversity) and exploitation quality) inspired by the upper confidence bound score (Auer 2002). which is typical solution of MAB. However, certainly our approach can use other CS functions. Here we discuss two intuitive alter- natives, i.e., the exploitation-oni, and exploration-ons us. The former one samples some data instances from each cluster, computes their influence scores and selects instances from clusters with high influence scores. The latter just randomly samples from different clusters without considering the data quality. As shown in the table below, we can observe that for both IB and 7B models, our method outperforms the two baselines because consider both the data quality and diversity to select data instances.
methods to demonstrate the superiority of
H CS FUNCTION
Note that our CS function is to balance the exploration (i.e., diversity) and exploitation (i.e., quality) inspired by the upper confidence bound score (Auer) 2002). which a typical solution of MAB. However, certainly natives, i.c., the exploitation-oniy and exploration-only methods to demonstrate the superiority of The former one samples some data instances from each cluster, computes their influence scores and selects instances from clusters with high influence scores. The latter one just randomly samples from different clusters without considering the data quality. As shown in the table below, we can observe that for both IB and 7B models, our method outperforms the two baselines because we consider both the data quality and diversity to select data instances.
approach use other CS functions. Here we discuss two intuitive alter-
Table 15: Model Performance with Different Size and Sampling Method
Model
Quad Score
exploitation-onle
exploration-onle
1B
42.94 %
42.36%
41.60
7B
53.19 %
52.56 %
49.33%
22

---

Published as a conference paper at ICLR 2025
I GENERALIZABILITY
To show the generalizability and robustness of our method, we add FLAN (Chung al. (024) (the mixture of multiple NLP tasks) and Openwebmath (Paster et al. 2023) (a math-related dataset) as reference datasets, associated with 7 new downstream tasks. We also add one candidate dataset and a 7B downstream model.
Specifically, we first use FLAN as new reference dataset to train 1.3B model dataset (i.c., FineWeb (Penedo et al. 2024). approximately 15000B tokens). In this experiment, 100B tokens out of 15000B tokens shown in the table below, Quad still demonstrates superior accuracy compared to all baselines, such the DSIR, Semdedup, PPL. Qurating and MATES. For example, we can observe that Quad has an improvement of 0.97% on model accuracy compared with the Qurating, which is a state-of- the-art baseline. addition, we add two downstream tasks (i.e., WikiText (Merity et al. 2016), HelloBench (Que et al. (224). and for WikiText, the lower the score, the better the model per- formance) about long-text generation tasks. As shown the table below, on the candidate dataset Slimpajama, for the 7B model, we can observe that Quad still outperforms other baselines because we simultaneously consider the quality and diversity in data selection.
candidate
selected for training using Quad and other baselines. As
Table 16: Model Performance with Reference Set FLAN
Methods
Random
DSIR
Semdedup
PPL
Qurating
MATES
Quad
Accuracy
44.41%
44.47%
45.38%
45.92%
47.03 %
46.96 % 47.88%
Table 17: Model Performance on Long- Text Generation Tasks
Random
Qurating
MATES
Quad
WikiText 16.73
15.36
15.51
14.81
HelloBench
3.81
5.37
4.39
6.33
Moreover, we also add the Openwebmath dataset another reference dataset train 7B model on the Slimpajama dataset, which aims at selecting data to improve the mathematical skill of the model. Correspondingly, we also inCludeGSMSK (Cobbe
(021), MATH (Hendrycks al., 2021 OCW (Lewkowycz et al., 2022). SAT (Azerbayev et al. (023). MMLU STEM (Hendrycks) al. 2020 new downstream tasks evaluate the model's 's mathematical capabilities, with the final accuracy calculated as the average score across these tasks. As shown in the table below, Quad still demonstrates superior model performance compared to baselines.
Table 18: Model Average Performance on 5 Math Tasks
Methods
Random
DSIR
Semdedup
PPL
Qurating 35.31%
MATES
Quad 36.71%
Accuracy
33.20%
33.05%
33.70%
34. 79%
35.20%
This experiment shows that our method can generalize well to various types of reference datasets and downstream tasks.
J RELIABILITY OF OUR EXPERIMENTAL RESULT
For our baselines Qurating & MATES, the performance improvement comparing with the best base- line their papers are 1.9% and 0.6% respectively. In our work, Quad surpasses the best baseline (i.e., Qurating) by 0.93%, the improvement is not marginal. Specifically, although MATES also selects data considering the current training process, it does not perform well because the surrogate model is not accurate enough due to lacking of enough training data. Qurating generally per- forms the best among other baselines, but still worse than our approach because it does not consider the varience of model state during the training process, and it incorporates the highest FL.PPs (1e19)
23

---

Published as a conference paper at ICLR 2025
because of the usage of LLMs for data selection. In terms of the FLOPs, observe that sume minimal computational resources because our MAB solution samples from clusters without iterating the entire candidate dataset like Qurating and MATES.
For DSIR, we would like to clarify that in the literature, i.c., the papers of our basclines, DSIR also performs worse than random selection. This is consistent with results. For example, in Qurating, random selection achieves a 2% higher accuracy than DSIR. In MATES, DSIR also does not perform better than random selection. The reason is that DSIR selects data instances whose n-gram features are similar to the instances in the validation set, which cannot capture the data semantics and the downstream model performance, leading to poor generalization ability. In addition, purely relying on the similarity of the strings to select data will incorporate many duplicated instances, which may even hurt the model performance.
To better demonstrate the benefits of our method Quad, we enlarge the model size from 1.3B to 7B, using LAMBADA as the reference dataset select 100B tokens from Slimpajama dataset. As demonstrated in the following table, Quad outperforms other baselines in terms of accuracy and has good scalability (achieving low FLOPs).
Table 19: Model Performance with Larger Model Size
Methods
Random
DSIR
Qurating
MATES
Quad
Accuracy 49.73%
49. .9.
12%
51.97%
53.19%
For the training loss, we find that Quad converges successfully at rate similar to the random selection. is notable that for the validation loss, Quad converges faster than random selection because can discover data instances that are beneficial for the current training process.
24


Reviews:
[{"review_id": "WLuiS0wBTr", "reviewer": "Reviewer_1sUi", "summary": "This paper explores the significance of data selection in the pretraining of large language models (LLMs) and the limitations of existing methods, including high computational costs and a lack of diversity. To address these issues, the authors propose a new method called Quad, which aims to balance data quality and diversity. Quad clusters the data and samples from each cluster to assess its influence, reducing computational burden while ensuring diversity. Additionally, Quad leverages attention mechanisms and the Kronecker product to efficiently compute data influence, further enhancing computational efficiency. Experimental results show that Quad significantly outperforms existing methods on the Slimpajama dataset and nine downstream tasks, achieving a 1.39% improvement in zero-shot accuracy with lower computational resource consumption. Overall, Quad provides an effective and scalable data selection approach, offering a significant breakthrough for LLM pretraining.", "questions": "1. Clustering Quality and Sensitivity: How sensitive is the clustering quality to the choice of the number of clusters (10,000 in this case)? What criteria were used to determine the optimal number of clusters, and how might different numbers of clusters affect the performance of the MAB approach?\n\n2. Generalizability and Robustness: The experiments were conducted on a specific set of downstream tasks and a particular reference set (LAMBADA). How well does the method generalize to other types of tasks or datasets? Have you tested the method on a broader range of tasks or datasets to ensure its robustness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Clear Problem Definition: The paper clearly defines the problem of enhancing large models by selecting a subset of data from a large pool to fine-tune the model, aiming to minimize the loss on a reference set. This provides a concrete and well-defined goal for the research.\n\n2. Balanced Quality and Diversity: The authors use Multi-Armed Bandit (MAB) techniques to balance quality and diversity in data selection. By treating each cluster as an arm and using the Upper Confidence Bound (UCB) to determine cluster scores, the method ensures frequent sampling of high-quality clusters while also exploring less-visited clusters to maintain diversity.\n\n3. Efficient Influence Calculation: The paper introduces an efficient method to estimate the influence of individual data points on the model without full retraining. By extending the influence function to multi-head attention layers and using K-FAC for acceleration, the authors provide a scalable solution for large language models, significantly reducing computational costs.", "weaknesses": "1. Trade-off Between Sample Size and Accuracy: Although the MAB technique can make decisions under uncertainty, there is still a trade-off between sample size and accuracy when estimating the average influence score. Small sample sizes may lead to inaccurate estimates, while large sample sizes increase computational costs. Finding the optimal sample size in practical applications remains a challenge.\n\n2. Complexity and Computational Resource Requirements: Despite the introduction of K-FAC and other acceleration techniques, the computational complexity and resource requirements remain high when dealing with large-scale language models. In environments with limited resources, this method may be difficult to implement or may require significant computation time.\n\n4. Assumptions and Model Dependence: The method relies on certain assumptions when calculating influence scores, such as the approximation of the Hessian matrix and the independence of gradients. These assumptions may not hold in some cases, affecting the effectiveness and reliability of the method. Additionally, different model architectures may require different adjustments and optimization strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0H0ehal1Ka", "reviewer": "Reviewer_yV6J", "summary": "The paper presents a method for data selection, namely Quad, which (i) uses the influence function with multi-head attention layers to estimate the potential impact of individual samples on the model's performance; (ii) uses clustering and multi-armed bandits (where each cluster is an arm) to balance between quality and diversity. The paper further proposes acceleration techniques to reduce computational cost and memory usage. The proposed method is applied on the SlimPajama dataset by training a 1.3B model, evaluated on a set of common text benchmarks, and compared with a set of data selection baselines.", "questions": "Could the authors report the results of statistical significance tests?\n\nHow does the selection of clustering algorithm and number of cluster affect the performance of the proposed method?\n\nCan the authors clarify how they compute the FLOPS column in Table 1?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the important challenge of data selection via a novel method and practical acceleration techniques. Influence function is an expensive operation and the proposed techniques make it more accessible for LLMs where computational overhead is an important factor. \n\nThe paper presents a comprehensive evaluation on downstream tasks as well as ablations and statistics to help understand the proposed method. \n\nDespite the method's complexity, the paper is overall clearly written and easy to follow. The authors also attach the code, which helps the community to adapt this method for custom usecases.", "weaknesses": "The proposed method requires a complex set of steps and hyperparameters (e.g., $\\alpha$,  $\\gamma$, and $\\tau$ for multi-armed bandit) and, even though there are ablations for some of these, it \nmight be hard and resource-intensive to develop and tune in practice for diverse applications.\n\nThe experimental findings show that all evaluated methods have limited difference in downstream performance and it is not clear whether the observed improvements are statistically significant. It is also concerning that baselines like DSIR show no benefit compared to random selection, which might be contradicting with findings from previous work. Therefore, I would recommend a more thorough discussion behind the empirical benefits of the proposed method and the connection of the findings compared to the previous literature. To better demonstrate the benefits, it might be worth (i) considering more experimental settings, for example with models of varying sizes; (ii) plotting the training and validation losses for each data selection method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tukeiTvZe1", "reviewer": "Reviewer_R56B", "summary": "This paper presents a new method for selecting LLM pretraining data. The core idea is to measure data quality with diversity. The authors introduce a new approach named Quad, which utilizes attention layers for influence calculation, and clusters data to maintain diversity. Experimental results on the Slimpajama dataset show that Quad surpasses other data selection methods in performance.", "questions": "1. Explain figure 1(a) better, what is the x- and y-axis?\n2. How do you apply the “JohnsonLindenstrauss Lemma” (line 343)\n3. Why only select “30B tokens out of the 627B” (line 360), what if we just do a coarse-grained filtering of the original 627B to leave around 400-500B data (e.g., via near-duplicate removal) and use all of them to do LLM pretraining, will the model performance better?\n4. Is the Quad method robust to the clustering algorithms? What features are used in the clulstering?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a framework that is overall reasonable.\n2. The influence calculation with attention layers technique is interesting and could potentially be useful in other settings.\n3. Experiments (of the current limited settings) are complete and the authors do release the code for better reproducibility", "weaknesses": "The main limitation of this study is its experiment scale is too small (in terms of claiming the method for LLM pretraining). Only 30B out of 627B data are selected for training a small 1.3B decoder-only LLM. In practice, most LLMs will start with at least 7B so how the technique can generalize to reasonably sized LLMs is questionable. Also related to this limitation is the diversity of testing datasets, little (long-text) generation tasks are evaluated.\n\nThe second limitation of this study is that the authors do not fully motivate the problem setting. While cleaning noisy data and removing near-duplicate examples are useful for LLM pretraining, I am not fully convinced why you have to select a subset of \"quality data\" from the full dataset. For example, if we do a coarse-grained filtering of the original 627B to leave around 400-500B data and use all of them to do LLM pretraining, will the model performance better?\n\nFinally, there are a few presentation issues such as the unclear method descriptions (see the below questions) and typos/grammatical issues (just list a few below).\n\n1. Line 042: ”LLMs are generalizable a broad spectrum of downstream tasks”\n2. Line 095: “..., which successfully addressing above challenges, achieves” \n3. Line 289, “.. will miss massive information of Consequently, …“", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8foQ0SbmDM", "reviewer": "Reviewer_Q54u", "summary": "This work proposes to balance the diversity and quality in data selection for LLMs pre-training. The author adopts the MAB method to dynamically estimate and sample data points from various clusters. Additionally, they introduce a novel influential calculation approach to estimate the data point more accurately while improving processing efficiency. Their experimental results on extensive benchmarks validate the effectiveness and efficiency of the proposed method.", "questions": "* How do you choose the sampling threshold hyper-parameter in the main experiment? Have you tuned it in a validation set?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The introduction of MAB to LLMs' pre-training data selection is reasonable and interesting, addressing a practical problem about diversity and efficiency.\n* The experiment results demonstrate that their proposed method significantly improves the pre-trained LLM's performance while maintaining efficiency.", "weaknesses": "* The experiments are not thorough. The experiment and analysis only take up 2 pages, which fails to thoroughly discuss many important components and designs of the proposed method. For example, whether the CS function could be designed in another way? Also, while the evaluation is conducted in various benchmarks, the authors only adopt experiments in a single LLM. This also makes people doubt the generalization of their method. I suggest the author to fully utilize the 10-page space and conduct further experiments.\n\n* According to ablation study results, the model's performance is quite sensitive to the sampling threshold of influence. And there are no hyper-parameter selection strategies introduced in the paper. All these make me worry about the robustness of the proposed method to apply in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)