Group-Level Data Selection for Efficient Pretraining
Authors: Zichun Yu
Fei Peng
Jie Lei
Arnold Overwijk
Wen-tau Yih
Chenyan Xiong
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Pre-training
Data Selection
Data Influence
Primary area: deep_learning
TL;DR: We present Group-MATES, an efficient group-level data selection method that optimizes the speed-quality frontier for LLM pretraining. On DCLM, Group-MATES achieves up to 9.4% relative accuracy gain and 1.75× faster training than random selection.

Abstract: The efficiency and quality of language model pretraining are largely determined by the way pretraining data are selected. In this paper, we introduce *Group-MATES*, an efficient group-level data selection approach to optimize the speed-quality frontier of language model pretraining. Specifically, Group-MATES parameterizes costly group-level selection with a relational data influence model. To train this model, we sample training trajectories of the language model and collect oracle data influences alongside. The relational data influence model approximates the oracle data influence by weighting individual influence with relationships among training data. To enable efficient selection with our relational data influence model, we partition the dataset into small clusters using relationship weights and select data within each cluster independently. Experiments on DCLM 400M-4x, 1B-1x, and 3B-1x show that Group-MATES achieves 3.5\%-9.4\% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improvements achieved by state-of-the-art individual data selection baselines. Furthermore, Group-MATES reduces the number of tokens required to reach a certain downstream performance by up to 1.75x, substantially elevating the speed-quality frontier. Further analyses highlight the critical role of relationship weights in the relational data influence model and the effectiveness of our cluster-based inference. Our code is open-sourced at https://github.com/facebookresearch/Group-MATES.

Group-Level Data Selection for Efficient Pretraining
Zichun Yu' Fei Peng" Jie Lei?
Arnold Overwijk²
Wen-tau Yih? Chenyan Xiong'
Language Technologies Institute, Carnegie Mellon University Meta
Abstract
The efficiency and quality of language model pretraining are largely determined by the way pretraining data are selected. In this paper. we introduce Group-MATES, an efficient group-level data selection approach to optimize the speed-quality frontier language model pretraining. Specifically, Group-MATES parameterizes costly group-level selection with relational data influence model. To train this model, we sample training trajectories of the language model and collect oracle data in- fluences alongside. The relational data influence model approximates the oracle data influence by weighting individual influence with relationships among training data. To enable efficient selection with our relational data influence model, we partition the dataset into small clusters using relationship weights and select data within each cluster independently. Experiments on DCLM 400M-4x, IB- 1x, and 3B-1x 6x show that Group-MATER achieves 3.5%-9.44 9.4% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improve- ments achieved by sfate-of-th--is individual data selection baselines. Furthermore, Group-MATES reduces the number of tokens required reach certain down- stream performance by up to 1.8x, substantially elevating the speed-quality frontier. Further analyses highlight the critical role of relationship weights in the relational data influence model and the effectiveness of our cluster-based inference. Our code is open-sourced at hathttps:/// //gi htw.e//ww..ooaaccoooooeeeeeeeeee Saapt//rrouuuMMMMMMMM MATES
Introduction
Improving the speed-quality frontier is es- sential for making large language models (LLMs) more efficient, scalable, and ac-
100%
Group-Level vs.
Avg. Downstream Ace
0.18
Group
80%
Individual Selection
0.16-
37.6%
Random Overlap
cessible across world applications [10]
Overlap
60%
0.14
Indiv
+15,0%
15] 20). Pretraining data selection [3] 25
40%
provides practical path to achieve that
0.12
Random
20%
by identifying high-quality data 31 44
0
1000
2000
3000
optimizing domain mixtures 45 and con- structing adaptive training curriculum [50]
Pool Size
(a) Selected data overlap.
(b) Evaluation results.
Effective data selection approaches can nearly double the PLOOos-prffrmance scaling efficiency of language models
Figure Misalignment (a) and performance gap (b)
[TT]
between brute-force group and individual selection.
50]. or enables smaller models to outperform larger counterparts [6].
Prevailing selection methods often evaluate the utility of cach training data point individually (1]. implicitly assuming that the overall utility of a data group is the sum its elements. However, theoretical analyses 4 34 reveal that the influence of a data group is shaped by complex interactions among data points rather than their isolated contributions. This discrepancy particularly pronounced when selecting pretraining data. As shown in Figure Ta in typical pretraining data selection setting (25) [50), individual data selection quickly diverges from brute-force group selection after merely hundred selected data points-les than single batch in modern pretraining workflows.
Part of the work done during internship Meta.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In Figure [b training with group selection exhibits significantly higher downstream performance than individual selection, doubling the efficacy of pretraining data selection. Although group-level selection demonstrates tremendous potential, directly optimizing it is computationally prohibitive, which requires enumerating an exponential search space of all possible data subsets [38
In this paper, we introduce Group-MATES, an efficient group-level data selection approach to optimize the speed-quality frontier of pretraining. Specifically, we parameterize costly group-level selection with relational data influence model. To collect its training data, we sample group training trajectories of the language model and compute oracle data influences [50] alongside. The relational data influence model approximates the oracle data influence by weighting individual influence with relationships among training data. To enable efficient selection with our relational data influence model, we partition the dataset into small clusters using relationship weights and select data within each cluster, preserving essential relationships while reducing the computational cost.
We empirically verify the effectiveness of Group-MATES on DCLM [25]. a standard pretraining data selection benchmark. DCLM is designed to assess the utility of data selection methods in enhancing pretraining, beyond the effects of basic cleaning and denoising that it already includes in data preprocessing. On DCLM 400M-4x, 1B-1x, and 3B-1x, Group-MATES achieves 3.5%-9.4% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improvements achieved by state-of the-art individual data selection baselines, including Fine Web- Edu Classifier [3I], WebOrganizer [45]. MATES [50], and Quad [3T] Furthermore, Group-MATED reduces the number of tokens required to reach a certain downstream performance by up 1.8x compared to random selection, substantially elevating the speed-quality frontier. Additional results confirm the effectiveness of Group-MATES, in approaching group-level data selection and the necessity of having relationship weights in our relational data influence model. Further analyses validate that our blaster-based inference facilitates a more efficient data selection procedure while preserving crucial relational information.
We summarize the highlights of our work as follows:
1. We propose Group-MATES, group-level data selection framework designed for efficient pretraining by parameterizing costly group selection with a relational data influence model.
2. We train our relational data influence model with sampled trajectories and enable its fast inference for data selection with influence-aware clustering.
3. Group-MATES sets new tf-te- on DCLM and significantly elevates the speed- quality frontier. Further analyses highlight the essential role of relationship weights.
2 Related Work
Improving the speed-quality frontier is essential for making large language models (LLMs) more efficient, scalable, and accessible (10 20). Pretraining data curation provides a practical path to achieve that by identifying and leveraging the most valuable data (3). Standard approaches for
data curation include: (1) Domain reweighting adjusts the mix of data from various sources (c.g., Wikipedia, GitHub) by determining optimal weights that work best for small proxy models 26
45
(2) Synthetic data generation employs generative models to rephrase [2]
128 or transform
49 52
existing data, thereby augmenting or refining datasets. (3) Data selection encompasses various
metrics to identify high-value data, ranging from rule-based filtering [30 32
deduplication of
semantically similar data [ 35), proximity to high-quality corpora 25|
47 L.L quality
scoring [31 44), and data influence attribution [1 40 50) [5]. The benefits of data selection
are signifiggam significant recent techniques can double the speed-quality scaling of LLMs [31 (5), or enable
smaller models to outperform larger counterparts trained on uncurated data (6].
The ideal goal of data selection to identify the optimal subset of training data that maximizes model performance under resource constraints (3]. However, directly finding optimal subsets has been shown computationally prohibitive 122 24], as requires retraining the model on all possible subsets. To circumvent this challenge, a common assumption is that the most influential data points will also constitute the most influential subsets 23]. Based on this, prior data selection methods A typical approach for which utilizes first-order Taylor expansion to estimate how model parameters would change if training point were infinitesimally
primarily focus on evaluating the influence of individual data points 40 50
approximating individual data influence is influence function 222 43

---

up-weighted. Beyond influence functions, DsDm [19] employs linear regression model to estimate individual influences from subset training runs, while MATES [50] proposes data influence model to parameterize individual influences, Both approaches have demonstrated notable success in improving the efficiency and effectiveness of pretraining.
While approximating group-level influences by individual influences can be computationally efficient, often introduces substantial inaccuracies, as data points rarely contribute model performance in isolation [17]. In particular, theoretical analyses 4 34 show that group-level data influences contain relationship terms that cannot be captured by individual influences, while empirical stud-
ies 17 I8 reveal that interactions among data points can either cancel out or amplify individual influences. To mitigate this gap, ZAMinfluence [5] iteratively selects the most influential point to approximate the maximization of group-level influences, i.c., the greedy algorithm 22). Building upon this work, researchers effectively applied group-level influences in data pruning
enhancing
trustworthiness
33 39
L.M fine-tuning [14], and data selection 37 38
3 Preliminary
In this section, we first introduce the formulation of pretraining data selection and then standard approaches to evaluate oracle data influences. Finally, we empirically illustrate the gap between group-level and individual data influence oracles.
Pretraining Data Selection. Given size-N pretraining dataset D and training budget of n data points, data selection approach aims to to identify the optimal size-n subset (in) C DD that yields the best pretrained model. In general, large-scale pretraining operates in a data-rich, compute-constrained regime, where the available data pool is much larger than what can be used for training given practical computational budgets. Thus, data selection is typically performed without replacement.
Formally, the optimal size-n training subset P(n) is the set that minimizes the loss over a reference data D, after optimizing the model M on ((m))
Pin) = arg min C(D, MD
(())
arg min Eaw)) E(y xy )
where MD denotes the model trained to converge on the data subset ((n) using an optimizer like Adam (21] and l denotes the function to compute the model loss input-output pair (z,y). Prior works optimize Din)) with retraining- based data influence oracles and their approximations.
Retraining- Based Data Influence Oracles and Approximations. The oracle group-level data influence of subset (i)) is normally quantified by leave-n-out retraining 4) In particular, leave- n-out retraining evaluates the influence I of subset ((n)) by measuring the difference in model performance when the subset is included in the training data versus excluded:
I(MG, (m)) L(D, MD) L(D, MD..
While this approach accurately captures complex interactions among data points, is computationally infeasible in practice, it requires retraining the model from scratch for every possible subset.
To make group-level influence computation more tractable, prior works [1 23 50
approximate
by decomposing the group influence into the sum of leave-one-out oracle individual influences:
I(M P(m))
Σ I(MD,,,,)
(n)
where I(M L(D, MD) L(D, (D)
(5)
Instead of working with converged models MM, MATES (50) introduces local probing to capture the dynamic nature of data influence as the model evolves during training. This technique calculates
-odel-aware oracle data influence by applying single gradient update to the current model M using data 24 and measuring the change in reference loss before and after this one-step update:
I(M,z;) L(D, A(M, - L(D, M))
(6)
3

---

where a() denotes the output of one-step optimization of model M on a data point x;. The theoretical connection between Eq. and influence functions can be found Appendix [.
To efficiently calculate oracle individual data influences, MATES trains a parametric data influence model Q Indiv that learns map data points their oracle individual data influences:
I(M,z;) x;) R Q Indiv (x;) = w. hxg
(7)
where wo denotes the regression weight that transforms the last hidden representation ha, of language model to the individual influence prediction.
Empirical Gap between Group and Individual Data Influence Oracles. Although the approxima- tion Eq. 4 makes group-level influence computation tractable, it can introduce substantial errors Prior studies indicate that these errors stem
when estimating oracle group-level influences [3 34
from the approximation's neglect of interaction effects among data points within the group [1 [8].
To illustrate the gap between group-level and individual data influence oracles, we conduct an empirical study with the following setup: we utilize an intermediate checkpoint A (specifically, DCLM [25] 400M-4x baseline model at step 12,288) during pretraining and continue training it on
the selected data for 100 steps, using the decay stage of the WSD scheduler [[6]. We then evaluate the
model's i performance on 22 downstream tasks from DCLM. Further details can be found in Section 5
We first select data that minimizes oracle individual influences, denoted DD indiv.
(m)
D indiv +- arg min (n) EDE I(M,
(8)
(x)
where arg min (n) EED denotes the set of the data points with the lowest-n oracle individual influences.
For group-level influences, direct optimization is infeasible as the search space grows exponentially with the dataset size. Instead, formulate group-level selection sequential process [8), greedily selecting the data point with the minimum oracle individual influence at each step:
For t = I,...,... Tt
arg min I(Ms
(9)
E,E' (t-1)
group D group
(zz). M = A(Me, (t-1)
(10)
starting from D group (0)
0, M1
(11)
IDidliDD decreases rapidly as increases. The overlap becomes close to random even for small n 100, indicating substantial divergence between the two selected sets. Furthermore, training M on D group doubles the perfor-
Figure shows that the overlap between 2 (n) and 2 preu (i.e..
mance gain over random compared to D indiv as illustrated in Figure lb This discrepancy aligns with
(x)
previous theoretical findings [ 34 and highlights the potential to approach group-level selection.
4 Methods
In this section, we introduce Group-MATES, a novel group-level data selection framework to advance the speed-quality frontier of pretraining. First, we propose relational data influence model to parameterize costly group-level selection ($4.1). Next, we describe how to train ($4.2) and efficiently Our overall pipeline illustrated in Figure
infer with the relational data influence model (4.3)
4.1 Parametric Approximation of Group Selection with Relational Data Influence Model
As greedy group selection outlined in Eq. 10 requires brute-force computation of all oracle data influences, it is prohibitively costly to apply throughout the entire pretraining process. To address this challenge, we propose a relational data influence model e to predict the oracle data influence of each candidate data point given the previously selected set Diel and thus parameterize the group selection procedure
(c-1)
For / = 1,..., n:
arg min
6rd D
D
rll rel - (t-1)
U {
(12)

---

Oracie influence
Parametric Appraximation
Cluster-Bosed Bosed Efficient Inference
Collection w/ Rollout
Miniriiee Group-Lever Influence
Train
Language Model
Relational Data Influence Model
Orocio
Grocie
Intuenoe
infuance
infuence
Dota
D
Dala
Bootstrap
Influence Aware Clustering
Group-Level Data Selection
Figure 2: Overview of Group-MATES, We collect oracle data influences by sampling training data trajectories and train a relational data influence model to approximate them. This model then selects data that minimizes group-level influences within each influence-aware cluster.
Specifically, the prediction of the relational data influence model is formulated as relationship- weighted individual data influence:
x4) = ard (ic-1))
Ch
Σ REiz * (w. hz,),
(13)
B
1SK
where R = sim heg hz,) ) is the relationship weight, and w. hg, is the predicted individual data influence.
(14)
(15)
and B3 are two trainable scaling factors initialized both from 1 and sim is the similarity function (e.g., cosine similarity) two embeddings, ranging within [-1, 1]. The relationship weight R
designed to capture the interactive effects among training data points [4] 17). We also provide theoretical analysis in Appendix A.2 2 to demonstrate the inherent connection between our relationship
weight and trajectory-specine influence functions 37
4.2 Training Relational Data Influence Model with Rollouts
To gather supervision signals for training relational data influence model, rollout policy to sample training data trajectories T and collect oracle data influences I(M,, 21) alongside:
T ~~a 21 It-1 (1PPa-))
3t ...
(16)
where D(n) + 1tt-1) (xr), M 2(,, I( L(D, M441) - -((+) M,).
(17)
(18)
where T is the rollout length, hyperparameter. We start with random rollout policy Taan train the initial data influence model Ocd by minimizing the mean squared error between its prediction @rel (xx D()-)) and oracle data influence I(Me,
re
arg min E ET grd
Σ (O D(--)) ( - I(M,,
(19)
The distribution of oracle data influences is typically Gaussian [50], so random sampling primarily focuses on the mean of the distribution. To better approximate the full oracle distribution, we introduce bootstrapping data inffuence model, targeted rollout policy Tboot based on Or that emphasizes the tail fractions of the oracle data influences. This policy explicitly samples data points corresponding to the lowest and highest predicted data influences:
Thoat DD (r-))
min
e (Z1 D(t-))) U max DIE (1-1) 6 (#c P(t-))) (20)
K lowest
highest
where is the rollout width at every step. We then combine the supervision signals sampled from both Trand and booo to train our final relational data influence model O m
rel Einal
arg min E Toad aond.
(Orel (ate D(--1) I(Me,
(21)
G
5

---

4.3 Cluster-Based Efficient Inference of Relational Data Influence Model
Directly plugging mmal into Eq. 12 can perform the group-level data selection. However, this iterative process still be computationally intensive. To select a subset of size n, the selection involves n steps. At each step 1, we compute @ (r; (D(e) for all N t remaining candidates from the original dataset of size N. This leads to a time complexity of O(N - n) calculation of relationship weights, which can be extremely slow for large datasets.
To speed up the selection, we propose a cluster-based based inference approach that significantly reduces the number of relationship weight calculations. We first partition the selection pool D into d clusters {C¹ , .e. ... C... Consequently, Eq. can be executed independently within each cluster, allocating the selection budget to each cluster proportionally to its size relative to the entire pool:
For i I....,::
C
For = 1, 1,...,
n
C(t)
+ C(-1))
of aig @ (xy G(-1))}
[D]
P(n)=
C ((m-1))1)
(22)
:E(1.....d)
This approach only computes relationship weights within each cluster rather than across the entire dataset. As a result, our cluster-based inference achieves a time complexity of O [N- enabling efficient group-level data selection for large-scale pretraining. In practice, running inference for each cluster independently with multiple threads can further reduce runtime.
Cluster-based inference can be viewed as an efficient approximation of brute-force inference in Eq. 12 which implicitly assumes independence between clusters and thus ignores relationships across them. To ensure that the most meaningful relationships are preserved during this process, we introduce influence-aware clustering. Specifically, our approach directly employs the relationship weight R as the similarity metric for clustering, grouping data points with strong relationship weights into the same cluster. As a result, the relationship weights computed within each cluster closely approximate those computed over the full dataset.
pretraining is conducted in S stages; after each stage we collect data influences with the current model M, train the relational data influence model Ore,, and utilize to select training data for the next stage + 1. This iterative process enables efficient, model-aware data selection throughout pretraining.
Group-MATHS
integrated into the pretraining pipeline in model-aware manner 50
5 Experimental Setup
Model and Data. We conduct our main experiments following standard setups in DataComp-LMM (DCLM) (25) a formalized competition benchmark the effectiveness of pretraining data selection. The data curation pipeline in DCLM integrates heuristic cleaning, deduplication, and model- -ased filtering, yielding stronger baseline performance compared other open-source datasets such as
C4 32,, Web [3I], and RedPajama 41 Beyond high data quality, DCLM also standardizes data loading, training hyperparameters, and evaluation tasks, making the competition strictly fair.
Specifically, we choose three experiment scales from DCLM, 400M-4x IB-1x, and 3B-1x. "400M" denotes the model size, and "4x" denotes the relative quantity of pretraining tokens for this model size, identified by the Chinchilla [55] optimum. We pretrain all models from scratch and evaluate pretrained models with 22 downstream tasks in either zero-shot few-shot manners. These tasks provide holistic assessment of the essential abilities of pretrained models, including commonsense reasoning, language understanding, reading comprehension. symbolic problem solving, and world knowledge. We use centered accuracy as the primary evaluation metric, where the accuracy per task transformed to 0 when it equals random guessing and 1 corresponds to perfect accuracy. The average centered accuracy across all tasks is denoted as "Core score". A comprehensive list of the evaluation tasks is provided in Table II
4400M 4x is not a predefined setup in the original DCLM, but we extend its 400M-1x Ix setup train for 4x longer (4x more tokens) for better evaluation stability.

---

Table 1: Benchmarking different data selection methods DCLM 400M-4x, 1B-1x, and 3B-1x settings. Dependencies on stronger LLMs (e.g.. LLa- Instructt are denoted by *. Best performances marked bold.
COMMONNENNE
LANGUAGE UNDERSTANDING (6.nsk))
READING COMPREHENSION (asks)
SYMBOLIC
WORLD
REASONING [3 tasks)
PROBLEM SOLYING
KNOWLEDGE (5 rasks)
CoRE mssss
(5 tasks)
400M-4X SETTING: 412M MODEL, 32.8B TOKENS
EDU
0.29401
0.28287
0.24732
0.21821
RANDOM
0.25335
0.28315
0.10477
0.15643
0.22858
0.21356
0.28176
0.28358
0.14225
0.16296
0.22179
0.22260
QUAD
0.33437
0.27733
0.12080
056664
0.22124
0.22358
GROUP-MATES
0.29190
0.28735
0.14997
0,18890
0.22908
0.23362
1B-IX SETTING: 1.4B MODEL, 28.0B TOKENS
EDU CLASSIFIER*
0.33713
0.37612
0.14689
0.20967
0.33590
0.29157
WHHORGANIZER"
0.36042
0.39132
0.20225
0.18162
0.30865
0.29488
RANDOM
0,34994
0,38584
0. 0.22059
0.18291
0.30784
0.29456
0.22548
0.19958
QUAD
0.34989
0.39913
0.16843
0.19864
0.30233
0,29340
0.39744
0.23922
0.20250
0.30793
0.30747
3B-1X SETTING: 2.8B MODBL, $5.9B TOKENS
RANDOM
0.44969
0.47816
0.27832
0,18070
0.37523
0.35603
MATES
0.44178
0.48263
0.30487
18497
0.37799
0.36139
GROUP-MATES
0.45874 0.48504
0.31094
0.19591
0,38146
0,36846
Baselines. We compare our method with (1) random selection (DCLM-Baseline); (2) edu clas-
sifier 31: educational valuation of data distilled from LLamm3--0B--Intruuet [TO]; (3) WebOrga-
nizer 45 domain construction with LLMs and mixture weight optimization via RegMix (26]. As
their selection code, we copy their results the same DCLM IB-1x setup; (4) MATES [50]: data influence estimation with individual data influence models; and (5) Quad [ cluster-level influence estimation and diversification with multi-armed bandit [36]. These baselines cover ttate-of-the-arr data selection techniques like LLM rating, domain mixtures, and individual data influence attribution. Some recent works, such as GREATS [X and TSLOO [37], have not open-sourced their selection code for pretraining or evaluation results, hindering direct comparison. We also compare our method with earlier baselines (DSIR 47 SemDeDup (1 DsDm [T and QuRating (4) in Appendix
WebOrganizer does not fully
Implementation Details. We sample size- 128 subset from FLAN as our reference data D, for its exceptional generalization abilities [8]. FLAN does not overlap with our evaluation tasks. We initialize all parameters of our relational data influence model O with sage-base-en--ee except wa. which is randomly initialized. The similarity function for calculating relationship weight R is cosine similarity, consistent with the original BGE design. We set the rollout length 7=10 and rollout width K=5, and collect 20,000 rollout trajectories to train our relational data influence model. In inference, we partition all data points into d=10,000 clusters (the optimal choice in Quad) using k-means [27]. The number of pretraining stages S is set to 2 and the selection ratio N is set to 50%. More implementation details can be found Appendix Ablations on key hyperparameters are presented in Appendix C.5
46
C.6
and C.8
Evaluation Results
In this section, we present our main results on DCLM ($6.1). Then, we analyze the training of relational data influence models ($6.2), demonstrate the effectiveness of influence-aware clustering (.6.), and finally present a case study ($6.4). Additional results can be found Appendix
6.1 Main Results
Overall Performance. Table summarizes the overall results on the DCLM benchmark. Group- MATES consistently outperforms random selection, achieving 3.5%-9.4% relative improvements Core scores across all three setups. Compared to our primary baseline, MATES, Group-MATES delivers superior performance on every subtask group, doubling its gain over random selection in the 400M-4x and 3B-Ix settings. Notably, for the 3B- setup, Group-MATES maintains consistent improvements over random selection, whereas MATES exhibits diminishing returns. highlighting

---

0.23
0.31
0.23
0.31
1337 Finter
Fewerr
1.5x Fasger
Core Score
0.22
Core Score
Core Score
0.22
Core Score
0.29
0.21
0.21
Group MATES
0.27
0.27
Quad
0.20
0.20
MATES
Random
0.19 30 40 Training Tokens (B)
0.25
20
25
0.1%0
0.19
80 100 120 140
150 250 300 350 0.25
Training Tokens (B)
Total H100 Hours
Total H100 Hours
(a) 400M-4x Tokens.
1B-1x //ok/okkss.
(c) 400M-4x Hours.
IB-Ix Hours.
Figure 3: Core score comparison between Group-MATES and baselines w.r.f pretraining tokens (a, b) and total H100 hours (c, d). Total H100 hours count both pretraining and data selection.
Table Ablation on the key components Group-MATES on DCLM 400M-4x setting.
COMMONSENSE REASONING
LANGUAGE UNDERSTANDING
READING COMPRERENSION (3ok))
SYMBOLIC PROBLEST SOLVING
WORLD KNOWLEDGE (5
CORE
ABLATION
(3 instri
(oskr)
(Sink
(22 kosky)
GROUP-MATES
0.29190
0.28735
0.14997
0.18890
0.22908
0.23362
ATIONSHPP
0.28451
0.14301
0.22733
w/o BOOTSTRAPING
0.28563
0.28139
0.14788
0.183044
0.2278.4
0.22924
w/ SEMANTIC CLUSTERING
0.28908
0.281172
0.14315
0.18524
0,23112
0.23042
RANDOM
0.25335
0.28315
0.10477
0.15643
0.22858
0.21356
the better scalability of group-level selection. These performance gains are substantial, as even the strong edu dlassifir-disttiiee from LLama--70B-Inttru and recognized for its effectiveness less curated datasets [31- [3ils to outperform random selection in the IB-1x setup. Edu classifier performs well on world knowledge tasks, since its selection strategy is optimized for educational value and thus favors knowledge-re related data. To demonstrate the generalization ability of our method, also conduct experiments on the C4 dataset in Appendix [C.4 where Group-MATES consistently matches or surpasses MATES on out of evaluation tasks. In summary. Group- MATES demonstrates significant advantage over individual data selection methods for pretraining, confirming the effectiveness of group-level data selection.
Frontier. Figure shows the evaluation results of Group-MATES and baselines with respect to pretraining tokens and total H100 hours. Token- based measurement reflects the compute cost of pretraining alone, as data selection can be easily parallelized with sufficient resources. Hour-based measurement accounts for both pretraining and data selection costs, representing the total compute used. In 400M-4x and IB-1x settings, Group Group-MATES MATES reduces the number of tokens by 1.8x and 1.4x needed to reach given Core score compared to random selection. Furthermore, Group- MATES achieves 31.5% % and 20.5% net efficiency gains (measured by H100 hours) in 400M-4x and IB- setups, respectively. By contrast, MATES only achieves 16.4% and 10.1 % net efficiency gains in 400M-4x and IB-1x setups. Therefore, Group-MATES nearly doubles the net efficiency gain of MATES, further validating the advantage of our group-level selection over individual selection.
These results demonstrate that Group-MATES substantially improves pretraining efficiency on the rigorous DCLM benchmark, achieving a state-of-the-arn art speed-quality frontier. A detailed breakdown of the compute overhead is provided in Appendix C.I and the pretraining results under equalized compute are presented Appendix
Ablation Studies. Table shows the ablation studies of three key components in Group-MATER namely relationship weight, bootstrapping data influence model, and influence-aware clustering. When we remove relationship weights during the selection process and consider only individual data influences, the performance gain over random selection drops by more than 30%, indicating that relational information plays a major role in the effectiveness of Group-MATES. In contrast, discarding the bootstrapping technique or replacing the influence-aware clustering with vanilla BGE semantic clustering also leads to noticeable performance degradation, though the magnitude of these drops is smaller compared to the effect of removing relationship weights. Overall, our ablation studies highlight the importance of incorporating relationship measurements between data points in our framework, extending the scope of data selection beyond the individual-level paradigm to account for inter-sample dependencies that better reflect the collective nature of the training data.

---

Reference Los
5.6
5.8
Group
Relational Individual
Core Soove
0.16
37.5%
Probubility Density
25-
20-
U+ Bootstrip
Random
Val. Spearman
0.50
0.75
5.4
Kandoea
0.14
15-
0.25
Random
5.2-
Relationhip
5.
0.00
5.0
100
Ramdon
Individudi
Relational
Group
3,00 3,05
3.10
3.15
2000 Training Steps
1000
Training Steps
Marginal Influence
(a) Reference loss,
(b) Evaluation results.
(a) Influence distribution.
(h) Influence modeling.
Figure 4: Reference loss (a) and evaluation results (b) of greedy group-level selection, data selected by relational data influence model, individual data influence model [50), and random.
Figure 5: Marginal influence distributions (a) and the performance of data influence models with random and bootstrap rollout policy, or random policy without relationship weight modeling (b).
Comparison with Greedy Group-Level Selection. This experiment compares the performance of our relational data influence model and the individual data influence model in MATES (Eq. 7 with greedy group-level selection, following the same experimental setup Section 3 As shown in Figure 4al the subset selected by our relational data influence model consistently achieves lower reference loss than the individual one after the initial steps. The evaluation results in Figure 4b further validate the superiority of our relational data influence model, with 5.8% relative performance gain compared individual selection after the training. We also emphasize the significant potential of group-level selection, which nearly doubles the performance gain even in the short decay stage. Nevertheless, our method represents a critical step toward efficiently tackling group-level selection and has demonstrated its effectiveness.
6.2 Analyses on Relational Data Influence Model Training
This experiment analyzes the training of our relational data influence models. As shown in Figure 5a using our bootstrapping rollout policy, the sampled oracle data influence distribution is more spread out. This demonstrates that bootstrapping effectively identifies more informative data points from the tails to train our relational data influence model. As result, our relational data influence model with bootstrapping better approximates the oracle, improving the upper bound of validation Spearman correlation by 0.18 compared to the random rollout policy alone, as illustrated in Figure 5b
We also demonstrate the necessity of having relationship weight in relational data influence model. As shown in Figure (5b] when the relationship weight is removed from the model formulation (Eq. [3] and only individual influence is considered, the Spearman correlation drops significantly near zero, indicating that the model fails to approximate the oracle data influence. This result suggests that the relationship weight is crucial for our relational data influence model to capture group-level influence, which aligns with previous theoretical findings 34
6.3 Effectiveness of Cluster-Based Inference
Probubility Densiy
This experiment demonstrates the advan- tages of using influence-aware clustering for more efficient inference with relational data influence models. First, we com-
Inference Time (h)
10
Intra
10
Cluster
No Cluster
Inter
6
pare the inference speed with clustering (Eq, (2) versus without clustering (Eq. [2). As shown in Figure ba], cluster-based inter- ence reduces inference time by several or- ders of magnitude, achieving over 10 x speedup compared to brute-force selection. This efficiency gain arises from comput- ing relationship weights within clusters in- stead of across the entire dataset. Consequently, our data selection procedure effectively scales to large pretraining datasets containing millions of samples.
12
1o4 1c6 1c8 lc10
0.3 0.3 0.9
Dataset Size
Relationship Weight
(a) Inference speedup.
(b) Relationship weight.
Figure 6: Inference speedup with clustering (a). Rela-
tionship weights in intra- and inter-cluster scenarios (b).

---

Table 3: Cancellation and amplification effects identified by our relational data influence model.
Relation
Data
Data
Cancellation
Let the schools teach history, science, arts... hope- filly allowing greater degree creativity and diversity to manifest. And purents should teach their children their philosophy, spiritual practices, and their wisdom as they see fit...
With tochnology, teachers are no longer going to be relevant, but on the contrary teachers are becoming more imporiant, have very different role, of an expert, a managor and a facilitator...
Amplification
the object is to find integers and z satisfying the Diophantine equation x-42=44 A) Inasmuch as god A, 4) = a divisor of 44, there is a solution to this equation. Upon multiplying the relation = 1 (-3) 4 1 by 44 to get 44= I(-132) 4-44...
Definition 2.2. Let a and b be given integers, with at least one of them different from zero. 'hhe greatest common divisor of and b, denoted by ged(a.b), is the positive integer d satisfying the following: (a) di a and di b. (b) If e la and el &, then << d...
To further validate that our influence-aware clustering can effectively approximate inference over the full dataset, we compare the distributions of the relationship weight It (Eq. 14) between data points within the same cluster (intra-cluster) or across different clusters (inter-cluster). As shown in Fig- ure the relationship weights are generally higher in the intra-cluster scenario, while inter-cluster relationship weights are distributed around Therefore, our influence-aware clustering preserves essential relational information by keeping intra-cluster relationship weights in the inference proce- dure. This analysis validates that influence-aware clustering effectively approximates relationship computation over the full dataset by focusing on the stronger intra-cluster relationship weights.
6.4 Case Study
Finally, we present case study in Table to illustrate how to interpret the relationship weights given by our relational data influence model. Specifically, we analyze two representative examples of cancellation and amplification effects, which are identified when the relationship weight is signifi- cantly greater than 0 and less than 0, respectively. The cancellation effect in the first example arises from misaligned perspectives on education, where data emphasizes parental influence and data highlights teachers' critical roles. In contrast, the amplification effect in the second example emerges from complementary concepts: data requires ged for its problem solution, while data provides formal definition of ged. Our study highlights the unique ability of our relational data influence model to capture complex interactions between training points, unlike semantic embedding models that focus solely on semantic similarity. We hope that our relational data influence model can serve as an analytic tool to discover and interpret more interesting interactions within pretraining data.
7 Conclusion
In this paper, we introduce Group-MATES, an efficient group-level data selection framework designed to optimize the speed-quality frontier of language model pretraining. On the DCLM benchmark, Group-MATES achieves 3.5%-9..4% relative performance gains over random selection, nearly doubles improvements from individual selection methods, and substantially reduces token and compute requirements for reaching target downstream performance levels. Further analyses show that modeling relationship weights is critical for accurately approximating oracle data influences.
Our work offers two key insights for data-centric pretraining. First, advanced data selection shall extend beyond the individual level, as model training is ultimately a collective effect. Second, modeling relationships among data provides a practical path toward scalable group-level data selection without extensive oracle data influence calculations. Future work can explore integrating Group- MATES into various training stages, extending relational modeling to further close the gap with greedy group selection, and investigating the theoretical limits of group-level selection efficiency. We hope that our work motivates shift toward group-level perspectives in data selection, paving the way for more scalable and efficient foundation model pretraining.
Acknowledgments
We thank CMU Foundation and Language Model (FLAME) Center for computational support.
10

---

References
[1] Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari Morcos. SemD:Dup: Data-efficient learning at web-scale through semantic deduplication. In Workshop on Multimodal Representation Learning: Perks and Pirfalls, 2023.
[2] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. ArXiv preprint, 2024.
[3] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. ArXiv preprint, 2024.
[4] Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In Proc. ICML, 2020.
[5] Tamara Broderick, Ryan Giordano, and Rachael Meager. An automatic finite-sample robustness metric: When can dropping little data make a big difference? ArXiv preprint, 2020.
[6] Aldo Carranza, Alvin Deng, Pratyush Maini, Muhammed Razzak, Jack Urbanek, Amro Abbas, Paul Burstein, Ning Cao, Priya Goyal, Joshua McGrath, Fan Pan, Josh Wills, Haoli Yin, Vineeth Kada, Vishwa Shah, Vishruth Veerendranath, Bogdan Gaza, Ari Morcos, and Matthew Leavitt. DatologyAI Technical Deep-Dive: Curating Our Way to a State-of-the-Ar Art-Art Text Dataset. Technical report, DatologyAI, 2024.
[7] Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. "what data benefits my classifier?" enhancing model performance and interpretability through influence-basee data selection. In Proc. of ICLR, 2024.
[8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. language models. JMLR, 2024.
Scaling instruction-finetune
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of HATC 2019.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle. Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. ArXiv preprint, 2024.
[11] Logan Engstrom, Axel Feldmann, and Aleksander Madry. DsDm: Model-aware dataset selection with datamodels. In Proc. of ICML, 2024.
[12] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In Proc. of ICML, 2019.
[13] Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. Proc. of ICLR, 2025.
[14] Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. ArXiv preprint, 2023.
[15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Proc. of NewrIPS, 2022.
11

---

[16] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. ArXiv preprint, 2024.
[17] Yuzheng Hu, Pingbang Hu, Han Zhao, and Jiaqi W Ma. Most influential subset selection: Challenges, promises, and beyond. In Proc. of NewrIPS, 2024.
[18] Jenny Y Huang, David R Burt, Tin D Nguyen, Yunyi Shen, and Tamara Broderick. Approxima- tions to worst-case data dropping: unmasking failure modes. ArXiv preprint, 2024.
[19] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry, Datamodels: Predicting predictions from training data. In Proc. of ICML, 2022.
[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv preprint, 2020.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
[22] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proc. of ICML, 2017.
[23] Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Perey S Liang. On the accuracy of influence functions for measuring group effects. In Proc. of NeurlPS, 2019.
[24] Dan Ley, Suraj Srinivas, Shichang Zhang, Gili Rusak, and Himabindu Lakkaraju. Generalized group data attribution. ArXiv preprint, 2024.
[25] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-LMM: In search of the next generation of training sets for language models. In Proc. of NeurlPS, 2024.
[26] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang. Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. In Proc. of ICLR, 2025.
[27] Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 1982.
[28] Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: recipe for compute and data-efficient language modeling. In Proc. of ACI., 2024.
[29] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 1978.
[30] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Refined- Web dataset for Falcon LLM: Outperforming curated corpora with web data only. In Proc. of NeurlPS, 2023.
[31] Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The Web datasets: Decanting the web for the finest text data scale. ArXiv preprint, 2024.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
[33] Prasanna Sattigeri, Soumya Ghosh, Inkit Padhi, Pierre Dognin, and Kush R Varshney. Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting. In Proc. of NeurIPS, 2022.
12

---

[34] Nikunj Saunshi, Arushi Gupta, Mark Braverman, and Sanjeev Arora. Understanding influence functions and datamodels via harmonic analysis. In Proc. of ICLR, 2023.
[35] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving LLM pretraining via document de-duplication and diversification. In Proc. of NewrIPS, 2023.
[36] Joannes Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical evaluation. In Proc. of ECML, 2005.
[37] Jiachen Wang, Dawn Song, James Zou, Prateek Mittal, and Ruoxi Jia. Capturing the temporal dependence of training data influence. ArXiv preprint, 2024.
[38] Jiachen Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. GREATS: Online selection of high-quality data for Ilm training in every iteration. In Proc. of NeurlPS, 2024.
[39] Jialu Wang, Xin Eric Wang, and Yang Liu. Understanding instance-level impact of fairness constraints. In Proc. of ICML, 2022.
[40] Xiao Wang. Weikang Zhou, Qi Zhang, Jie Zhou, Songyang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, and Tao Gui. Farewell to aimless large-scale pretraining: Influential subset selection for language model. In Findings of ACL, 2023.
[41] Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhang. RedPajama: an open dataset for training large language models. In Proc. of NeurlPS, 2024.
[42] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. In Proc. of ICLR, 2022.
[43] Sanford Weisberg and R Dennis Cook. Residuals and influence in regression. 1982.
[44] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high-quality data for training language models. In Proc. of ICMI, 2024.
[45] Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. In Proc. of ICML, 2025.
[46] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Nie. C-Pack: Packed resources for general chinese embeddings. In Proc. of SIGIR, 2024.
[47] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Proc. of NeurlPS, 2023.
[48] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. In Proc. of ICLR, 2023.
[49] Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Synthetic continued pretraining. ArXiv preprint, 2024.
[50] Zichun Yu, Spandan Das, and Chenyan Xiong. MATES: Model-aware data selection for efficient pretraining with data influence models. In Proc. of NeurIPS, 2024.
[51] Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang. Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, et al. Harnessing diversity for important data selection in pretraining large language models. In Proc. of ICLR, 2025.
[52] Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale. ArXiv preprint, 2024.
13

---

A Theoretical Analysis
A.1 Connection between Oracle Data Influences and Influence Functions
We start from the oracle individual data influence, defined as the change in reference loss after locally probing the model .M with z; [50]:
I(,, L(D, ((M, L(D, MM)
(23)
where A(M, mii) denotes the model after training on Zi; To approximate this, we consider the optimal model M after upweighting z; by small E:
Mr.,. arg min c(a MM cC(a; M).
1
(24)
M
71
and let M' Mo be the original optimum. The influence function estimates the change in reference loss
I(M, [(D, M
(25)
c=0
Applying the chain rule,
d My
a. d.M
[(,
= MV MM
(26)
de
de
c==0
c==0
To compute d.M.
we differentiate the optimality condition:
c=0
M
((ry M el(or MM
(27)
n
Differentiating both sides with respect to and evaluating at e = 0 yields
dM
HM
VM IM*) = O.
(28)
de
c=0
d.Mi..,
- - VML(I M*),
(29)
de
c=
where HM- = 1 ΣE Eay IM is the Hessian. Substituting back to Eq. 26 we obtain the
influence function approximation:
I(M, -V M³) VMLCi, M*)
(30)
A.2 Connection between Relationship Weight and Trajectory-Specifi Influence Function
Trajectory-speciff influence function 337 approximates a data point's conditional influence within
the training trajectory, providing an intermediate approach to capture group-level influences. Formally, given training batch sequence {B,, B2..... BB J. we estimate the influence of downweighting training data point æ; from batch B, by a small amount on the reference loss L(D, - )-t.),-DD,,,
M where M and Mi,,, are the final converged model state before and after the downweighting.
The model updates with Stochastic Gradient Descent (SGD) as:
M++1 M. T/ Σ
VML(w Mc)
(31)
zEB.
Downweighting x; by modifies the update at step t:
M441(c) = M, - -
Σ
VMC(I
M,) + Mc
M,)+
(32)
14

---

The change in reference loss is approximated around E = with first-order Taylor expansion:
[(D, M,,,)) L(D, = Rc OL(D, MM
(33)
de
c=0
O.
Re I
6,
(34)
a.M
Now we derive
Lidg de
At t, differentiating the modified update:
l==0
(M)+
Tre MM M,)
(35)
de
c=0
For subsequent steps j = t I,..., T 1,i (M)+1(E) de
(,)((E)
VM MM(()) aMj(c))
(36)
de
de
zEB;
(M,((E) de
= (1 - n1 Hj)
(37)
c==0
Unrolling from ++ 1o
U. a,
aMr(c)
T-1
II (I (1- (1- 1=
(n VoVor
(M))
(38)
c=0
l==0
Substituting into Eq.
T-1
L(D, M2 - - L(D, M²) = en VMC(D, MM) IM)
II (1 mHH1,) VmD(Is Mi) -1+1
(39)
When z; is totally removed from batch B,, i.e., e = 1:
C(D, Mi M*) ~ n VMD, (M))
II (T (T- -=
Vmnlla MM.) (40)
Our relationship weight R in Eq. 14 serves similar purpose IT = mT H42 by reweighting
the influence of a data point based on its relationships with other points in the training trajectory.
B Experimental Details
Table 4: Training hyperparameters.
HYPERPARAMETER
400M-4X
IB-1X
3B-1X
RELATIONAL INFLUENCE MODEL
STEPS
31403
54923
107610
3086
BATCH SIZE
512
256
256
128
SEQUENCE LENGTH
2048
2048
2048
2048 (512 ( 4)
MAX LEARNING RATE
3F-3
3F-3
3F-3
SB-5
OPTIMIZER SCHEDULER
ADAMW
ADAMW
ADAMW COSINE
COSINE
COSINE
COSINE
Language Model. We pretrain all decoder-only language models from scratch following DCLM setups. The training employs cosine learning rate scheduler and Adam optimizer 21]. All experiments are conducted on GPUs, with detailed training hyperparameters provided in Table 4
Relational Data Influence Model. Our relational data influence model fine-tuned from rre-base-en-vi... [66), which takes the last hidden state of the first token (i.c., (CLS)) as the
15

---

sentence embedding h E R As our base model only supports a maximum imput sequence length of 512, but our pretraining sequence length extends to 2048, we split each sequence into four chunks and process them separately. The hidden states of four chunks embedding h. This vector is then multiplied by regression weight w. E I 768 to predict individual influence W, h. For relationship weights, the sim function the cosine similarity between two embeddings, consistent with the original BGE design. The model trained using the mean squared error loss between the predicted and Z-score normalized oracle data influences. The validation set consists of 1,000 sampled oracle influences. All training hyperparameters are listed in Table
averaged to compute the final
C Additional Results
In this section, we provide detailed compute breakdown (SC.) results on different setups CCC2 C.4). and extensive ablation studies ($C.5
SC.3
SC.6 C.7
to support our findings.
C.1
Compute Breakdown
Table 5: #FLOPs and H100 hours breakdown of Group-MATES.
PROCESS
#FLOPS +1R19
#FLOPS RATIO H100 HOURS
HOURS RATIO
400M-4x SETTING: 412M MODEL, 32.8B TOKENS
MODEL PRETRAINING
8.00
87.8%
104.0
82.9%
ORACLE DATA INFLUENCE COLLECTION
0.29
3.2%
4.5
3.6%
DATA INFLUENCE MODEL TRAINING
0.05
0.5%
2.7
2.2%
DATA INFLUENCE
0.77
8.5%
14.2
11.3%
TOTAL
9.11
100%
125.4
100%
1B-1x SETTING: 1.4B MODEL, 28.0B TOKENS
MODEL
24.00
93.3%
240.0
90.0%
ORACLE DATA INFLUENCE COLLECTION
1.01
3.9%
12.1
4.5%
DATA INFLUENCE MODEL TRAINING
0.05
0.3%
2.7
1.0%
DATA INFLUENCE MODEL INPERENCE
0.65
2.5%
12.0
4.5%
TOTAL
25.71
100%
266.8
3B-1X SETTING: 2.8B MODEL, 55.9B TOKENS
MODEL PRETRAINING
94.00
96.6%
740.0
94.2%
ORACLE INFLUENCE
1.98
2.0%
18.7
2.4%
DATA INFLUENCE
0.05
0.1%
2.7
0.4%
DATA INFLUENCE MODEL INFERENCE
1.30
1.3%
23.9
3.0%
TOTAL
97.33
100%
785.3
100%
We provide detailed breakdown of compute used by Group-MATES in Table measured either by #FLOPs or H100 hours. Notably, the data selection procedure of Group-MATE only accounts for 12.2%, 6.7%, and 3.4% of the total FLOPs in 400M-4x, 1B-1x, and 3B- 1x setups, respectively. The relative selection cost in larger setups is generally smaller because their pretraining FLOPs dominate the total computation, while the training and inference costs data influence model remain nearly stable. Considering the net gains achieved by Group-MATES, its compute overhead can be negligible.
C.2 Equalized Compute Setup
To evaluate the effectiveness of Group-MATES under equalized compute setup, compare its performance against random selection using the same total FLOPs, as presented Table Although random selection utilizes more tokens for pretraining, Group-MATES consistently outperforms across different scales. Specifically, Group-MATES achieves relative gains of 7.4%, 4.1%, and 3.0% the 400M-4x. 1B-1x, and 3B-1z 1x setups, respectively. These results highlight that merely increasing the number of training tokens does not yield comparable improvements method. Notably, the computational overhead of Group-MATES diminishes relative to the total pretraining cost as model and data scales increase. Furthermore, the selection process can be efficiently parallelized and decoupled from the pretraining. Our results underscore the scalability and efficiency of Group-MATES. making it an attractive preliminary step for large-scale pretraining.
selection
We observe that in some sub-categories, the performance of random selection slightly decreases despite increased #FLOPs. We hypothesize that this is due to the stochastic nature of long-
16

---

Table 6: Comparison with equalized compute DCLM 400M-4x, 1B-1x, and 3B-1x settings.
SYMBOLIC
UNDERSTANDING
COMPREHENSION Nanks)
AFLOPS/
RANDOM
+18199
MATES
#
0,28358
0.14225
0.16296
0.22179
0.22260
GROUP MATES
9.11
#1E19/ 32.3B
0.29190
0.28735
0.14997
0,18890
0.22908
0,23362
18-1x SETTING:
1.48 MODEL
RANDOM
24.00 / 28.08
0.34994
0.385%4
0.22059
0.18291
0.307 84
0.29456
RANDOM
25.71 +1819/ 50 OB
36643
0.37954
0.22403
0.18115
0.30665
0. 20530
MATES
25.71 +1119/
0B
0.36 0.36311
0. 1060640
0.225548
19958
0.30415
0. 30288
GROUP MATES
41E91 28.00
0.36997
0.39744
0.23922
0.20250
0.30793
0,30747
3B-18 SETTING:
2,88 MODEL
RANDOM
9.4 +1401 +5.20/55.9B 55.99
0.44969
0.47816
0.27832
0.18070
0.31525
0. 0.35602
RANDOM
9.7 0.2-11:20/ 57.778
0-45261
0,48056
0.28435
0.18126
0.37431
0. 35782
MATES
41220 55,99
0:44178
0.48263
0.30487
0,18497
0.37799
0.36139
0.45874
0.48564
0.31094
0.19591
0.36846
Table 7: Results on DCLM IB-3x setting.
COMMONSENSE REASONING
LANGUAGE UNDERSTANDING (askk)
READING COMPREHENSION (3 tosks)
SYMBOLIC PROBLEM SOLVING
WORLD KNOWLEDGE tuska)
CORE
METHOD
(3 tarks)
(5 taska)
(22 tosks)
1B-3X SETTING: 1.4B MODEL, 84.0B TOKENS
RANDOM
0.42433
0.44592
0.27744
0.17986
0.35294
0.33840
MATES
0.40872
0.45021
0.29262
0.20697
0,34452
0,34376
GROUP-MATES
0.43300
0.44907
0.29663
0.21737
0.363220
0.35391
LLM pretraining optimization, where simply adding more tokens does guarantee consistent improvements across all tasks, especially when comparing similar computational budgets [31]. For instance, in the official DCLM leaderboard, 7B-2x models do not universally outperform 7B-1x models despite doubling the training tokens. As highlighted in the DCLM benchmark, Core score provides a more reliable and comprehensive metric that mitigates individual task variability.
C.3 1B-3x Setup
This experiment illustrates that larger models quently require more extensive training to fully manifest the benefits of data selection. In Table 7 we further run a set of experiments in the 1B-3x setup, where the training tokens are 3 times more than IB-1x. Our results show that the absolute Core increases from 1.3% (IB-Ix) to 1.6% (IB-3x), doubling the gains achieved by MATES. Therefore, Group-MATES consistently maintains its advantage even as model size and training data scale up.
robust to data quality variations and conse-
improvement of Group-MATES over random
C.4 MATES Setup
In this section, compare Group-MATES with previous pretraining data curation baselines, follow- ing the same setup as MATES [50]. These methods include (1) DSIR [47] proximity to Wikipedia
based on n-gram features. (2) SemDeDup [ deduplicating semantically similar data. (3) DsDm [[]] static approximation of influence functions by converged proxy model. (4) QuRating [44] ranking with educational values distilled from GPT-3.5. As shown Table [8 Group-MATES achieves the best average downstream results with minimal additional costs, highlighting the potential of optimizing group influences in data-efficient pretraining.
C.5 Selection Ratio
In Table we explore the impact varying the selection ratio of Group-MATES to 10%, 25%, and 50%. A 10% selection ratio does not perform as effectively as the other two, likely due the loss of diversity in a high-quality corpus like DCLM when the selection is too aggressive. Both 25% and 50% achieve comparable results; however, 50% produces more training tokens, making the preferred choice for our final selection ratio.
17

---

Table Zero-shot evaluation of pretraining 1B models with different data selection methods on C4. We report per-task accuracy and the total #FLOPs for each method. All results except Broup-MATES are directly copied from the original MATES paper [50]. Best performances are marked bold.
METBOD uPLoPS wirito
SciQ
ARC-E
ARC-C
LoGIQA
OBQA
BooLQ
HELLASWAG
WINOGRANDE
AVERAGE
SETTING: MOUEL,
RANDOM 17.80)
65.8
43.7
25.6
27.5
31.8
60.2
43.8
68.9
50.7
46.4
DSIR
65.8
42.6
24.7
28.7
29.2
59.7
44.2
68.3
53.2
46.3
SEMDEDEP
66.8
25.3
27.6
30.6
60.2
45.3
69.7
52.5
47.1
c2.00
QURATING (3767)
67.1
25.6
26.9
60.3
70.2
46.9
MATES
67.3
44.9
25.9
28.7
32.2
61.9
45.3
69.5
52.
47.5
GROUP-MATESS
67.8
45.0
25.5
28.9
32.6
60.9
47.4
70.5
52.4
47.9
Table 9: Results on DCLM 400M-4x setting with different selection ratios.
COMMONSENSE REASONING
LANGUAGE UNDERSTANDING
READING COMPREHENSION (3 taska)
SYMBOLIC PROBLEM SOLVING
WORLD KNOWLEDGE
CORE
RATIO
(3 tasks)
(6 tasks)
(5 tasks)
(5 taska)
(22 taska)
10%
0.27575
0.27950
0.13437
0.19149
0.22776
0.22743
25%
0,28573
0.28654
0.15217
0.18646
0.22830
0.23212
(OURS)
0.14997
0.18890
0.22908
C.6 Design of Relational Data Influence Model
In this section, we vary the design choices of our relational data influence model, including replacing the model backbone with BERT-base [9]. choosing dot product FFN model the sim function. As shown in Figure 7a BERT demonstrates weaker abilities to approximate oracle data influences than BGE, as the latter been specifically optimized for sentence embeddings. Taking FFN as the sim function does not significantly decrease the approximation performance but introduces additional parameters; choosing dot product, the performance dramatically drops. This validates our choice to align the similarity measurement with the original BGE, i.e., cosine similarity.
C.7 Number of Collected Trajectories
In this section, examine the effects of number of collected trajectories the approximation performance our relational data influence model. As shown in Figure (7b scaling up the number of collected trajectory consistently elevates the performance, but with diminishing returns. Considering the effectrivenes--effciinny trade-off, we finally choose 20k as the number of collected trajectories.
0.7
0.7
Val. Spearman
Val. Val.S0 Spearman
0.6
BERT
0.1
0.5
Dot
0.1
FFN
Ours
0.4
0
1000
2000
3000
2k
Sk 10k 20k 40k
Training Steps
Number of Trajectories
(a) Model design.
(b) #Trajectories.
Figure 7: Performance of relational data influence model with different designs (a) and number trajectories (b).
C.8 Rollout Length
In Table TO we investigate the effect of varying the rollout length in Group-MATES, considering values 2, 10, and 20. We observe that 7 = 10 performs slightly better than 2, suggesting that moderate increase in rollout length enables our relational data influence model to better capture long-term effects. However, increasing T to 20 results in a performance decline, likely due to the increased complexity modeling combinatorial effects over longer trajectories, which incurs additional challenges for the relational data influence model. These results indicate the importance
18

---

Table 10: Results DCLM IB-1x setting with different rollout length T.
COMMONSENSE
LANGUAGE UNDERSTANDING
READING COMPREHENSION
SYMBOLIC
WORLD
REASONING
PROBLEM SoLVING
KNOWLEDGE
CORE
T
(3 tasksi
(6 makks
(3 tasks)
(5 tasks)
(5 tosks)
(22 tasks)
0.40461
0.22536
0.201
0.30685
20
0.36549
0.39901
0.21265
0.20501
0,30081
0.30262
10 (OURS)
0.36997
0.39744
0.23922
0.20250
0.30793
0.30747
of selecting an appropriate rollout length that sufficiently reflects group-level data influence while remaining tractable for the relational data influence model learn effectively.
D Limitations
Our current study focuses models ranging from 412M 2.8B parameters, providing initial validation of our proposed methods. However, extending these insights to large-scale, production- level training scenarios remains a promising direction. On one hand, scaling up offers greater flexibility and potential gains for data selection, the larger candidate pool and increased demand for efficiency make sophisticated curation strategies more valuable, and the relative cost of data selection becomes less significant. On the other hand, large-scale pretraining may introduce new stability and optimization challenges that call for dedicated methodological advances. We leave the exploration of these directions to future work.
Future research could further advance group-level data influence theory itself, for example by characterizing the interactions and dependencies among data groups, analyzing the conditions under which group-level influences significantly diverge from individual data influences, and developing new theoretical frameworks that connect influence modeling with generalization and representation learning. Such work may yield deeper insights into the fundamental principles that govern collective data effects and provide stronger foundations for principled data curation strategies.
E
Broader Impacts
Our work paves the way for future where efficient pretraining seamlessly integrates data valuation, curation, and model training into a unified, self-optimizing framework. By advancing group-level data selection, our approach empowers foundation models to utilize data wisely and purposefully, significantly reducing computational costs while enhancing scalability and generalization. This break- through has the potential to lower resource barriers, making high-performance Al more accessible to a wider range of researchers and organizations.
Beyond efficiency, work improves the interpretability of training data influence, shedding light on how different subsets contribute to model learning. As foundation models become increasingly capable of dynamically adapting to evolving data distributions, they will drive progress in various fields, from AI-driven scientific discovery to large-scale real- wal-world applications. Moving forward, our approach lays the groundwork for a new paradigm in pretraining- -one where models autonomously optimize their learning trajectories with minimal human intervention, leading to more efficient, adaptive, and impactful Al development.
19

---

Table 11: Full results on DCLM 400M-4x. The number beside each task denotes the number of few-shot demonstrations used for evaluation. We exclude CommonsenseQQ from the core score calculation due to its instability and limited informativeness. For instance, in the original DCLM paper, the 412M model dramatically outperforms the 1.4B model by 76.6% on this task.
RANDOM
EDU CLASSIFFER
MATES
QUAD
GROUP-MATHE
AGE EVAL__SAA, (3)
0.19565
0.28696
0.20435
0.20000
0.27826
CHALLENGE (10)
0.29522
0.32253
0.29863
0.29181
0.27730
ARC_EASY (10)
0.57912
0.59975
0.57323
0,58460
0,56860
BIGBENCHL ALGORITHMS (10)
0.44697
0.33712
0.39697
0.43258
0,44091
BIGBENCII_DYO BE____DC_________ LANGUAGES (10)
0.19300
0,21600
0.18800
0,20300
0.16500
HIGRENCH, LANGUAGE IDENTIFICATION (10)
0.24690
0.25320
0.25500
0,25310
0,25750
BIGBENCH_ OPERATORS (10)
0.14762
0.18095
0.16190
0.14762
0.20952
BIGBENCIL BA_WIKIDATA (10)
0,52099
0,52492
0.52360
0,50431
0.51557
BIGRENCH REPEAT_ CORRAT_COPY___ SOOPY_LOICCCC (10)
0.00000
0.03125
0.06250
0,00000
0.03125
(10)
0,56881
0,49021
0.59113
0,58899
0.61407
COMMONSENSE_OO (10)
0.37838
0,22195
0.22523
0.31286
0.20393
COPA (0)
0.62000
0.69000
0.66000
0.74000
0,68000
COQA (0)
0.21195
0,21283
0.22836
0.21308
0.21320
HELLASWAG (10)
0.45230
0,45399
0.45519
0.45589
0.45907
HELLASWAG (0)
0.45638
0.45688
0.45828
0,45818
0.46116
JEOPARDY
0.12347
0.14875
0.11854
0.09442
0.14690
LAMBADA, OPENAL (0)
0,50708
0.45624
0.50340
0.50010
0,50049
MMLU FEWSHOT MMLU_FEWSHJOT (5)
0.24948
0.24992
0.22825
0.25419
0.26629
OPENBOOK_OG
0.33400
0,33600
0.34200
0.34200
0.33400
PIQA (10)
0.70403
0.69369
0.70131
0.70022
0.70185
SQUAD (10)
0,23709
0.23936
0.27436
0,23094
0,25232
WINOGRAD (0)
0.69231
0,70330
0.69963
0.68864
0.69231
WINOGRANDE (0)
0.54538
0.55406
0.53354
0.52802
0,54775
CORE
0.21356
0,21821
0.22260
0.22358
0.23362
Table 12: Full results on DCLM IB-Ix. The number beside each task denotes the number few-shot demonstrations used for evaluation. We exclude CommonsenseQA from the core score calculation due to instability and limited informativeness. For instance, in the original DCLM paper, the 412M model dramatically outperforms the 1.4B model by 76.6% on this task.
TASK
RANDOM
EDU CLASSIFIER
MATES
QUAD
GROUP-MATES
AGL LVAL_LSAT_AA (3)
0.19565
0,23913
0.24783
0.26522
0,27826
ARC_ ANC_CILLLNNGE (10)
0.36007
0,37799
0,36092
0.34386
0.35836
ARC FASY (10)
0.65362
0.69360
0.64689
0.64226
0.65909
BIGBENCIL_CO ALGORITIMS (10)
0.44091
0,44015
0.43485
0.44394
0.41667
RIGRENCH DYCK_I
BDCC__DANCTCACCCEOCCCCCCC LANGHAGES (10)
0.224010
0,27300
0.23600
0.17400
0.22600
HIGBENCH_ LANGUAGE, IDENTIFCATION (10)
0.25430
0.24940
0.24370
0.25390
0,25410
BIGBENCIL_
(10)
0.22388
0.22381
0.20476
0.20000
0,23333
BIGBENCH_ LL___Q_____. SA__IIIDAAA (10)
0.60179
0.60066
0,59151
0.59288
0,58531
HIGBENCH REPEAT CEPPATTCOPY__CC COPPPLLLICCCC (10)
0.03125
0.06250
0.06250
0.09375
0.06250
BOOLQ (10)
0.61957
0.51315
0.61988
0.54220
0.62538
COMMONSENSE_CO (10)
0.31368
0,21458
0.27600
0.26536
0.33579
COPA (0)
0.70000
0.67000
0.72000
7.70000
0,72000
COQA (0)
0.30527
0.31003
0.31204
0.31229
0,31079
HELLASWAG (10)
0.57648
0.57170
0.58156
0.57220
0.58604
HELLASWAG (0)
0,58186
0,57518
0.58335
0.57837
0,57807
JEOPARDY (10)
0.24318
0.31211
0.24653
0.26231
0,23064
LAMRADA OPENAI (0)
0.59441
0.55055
0.60120
0.59402
0.60489
MMLU FEWSHOT (5)
KMLU_FEWSHOT
0.25699
0.25345
0.25423
0.25644
0.27533
OPENBOOK
0.38400
0.39200
0.38000
0.37000
0.38600
PIOA (10)
0.73558
0.74102
0.73830
0,74483
0.74429
SQUAD (10)
0.35762
0.41183
0.36471
0.39773
0,39272
WINOGRAD (0)
0.74359
0,75458
0.78755
0.79853
0.77656
WINOGRANDE (0)
0.58800
0,58011
0.57380
0.57853
0.59116
CORE
0.29456
0.29257
0.30288
0.29340
0.30747
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions, such as the motivation of group-level data selection, the parametrization using relational data influence model, and the empirical validation of the proposed method. These claims are consistent with the experimental results and discussions provided in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper includes discussion of its limitations in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides a complete set of assumptions in Section 3 and a complete proof for the theoretical backups in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
The paper contains detailed descriptions the experimental setup, including datasets and models used, training procedures, and hyperparameters in Section 5 The code
is open-sourced at pectts://gitaan httpp////gggia.b..occ/ceeoeooroe http://accbbooooeesrrrcccc///c/a hrapreeeeeeeeeeeeaacccrrrrrrrr-a MATES
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper uses public datasets and open-sources its code that faithfully reproduces the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides wwwpss/SooSSuiiiiiiooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https (ww.eeswwwiiiiioooolllooooooeoae /nips.cc/ppls http//bblccgguuiees/eeeep
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies all training and details, including the datasets used, the models employed, and the training hyperparameters in Section 5 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
23

---

Justification: The paper reports enror bars for small-scale experiments (e.g., Figure 4a). but given the significant computation cost of large-scale pretraining, the main experiments are not repeated multiple times. This is common choice in the field and also the standard practice on the DCLM benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides details on the type compute resources used, including the specific hardware, memory, and the approximate time required for execution of the experiments in T
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////eaarpps.cccuccaii/////
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses its potential societal impacts in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper discusses the measures taken to ensure the responsible release the models, including the use of appropriate licensing and ethical guidelines to prevent misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
25

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper credits the original creators of datasets and models used in the experiments and respects their licenses and terms of use.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, such as the relational data influence model, and provides documentation alongside these assets with sufficient instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
26

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components. LLM is only used for writing polishment.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, non ntanstandard components.
Please refer to our LLM policy octTpe////eerrrescccc/o/ooorrrre for what should or should not be described.
27


Reviews:
[{"review_id": "XLPxodBnEz", "reviewer": "Reviewer_zK2t", "summary": "This paper focuses on trying to improve pretraining data selection by considering the utility of groups of training examples rather than just considering the influence of a training example individually. As a proof of concept, the group data selection is first done in a computationally expensive, brute force manner. The authors then propose a computationally cheaper approximation called Group-MATES for doing group-level data selection. This method is then tested on training on setups from the DCLM competition at 3 different scales. Finally ablations are performed on the various components of Group-MATES.", "questions": "1. The paper says the results in Table 1 are for models pretrained from scratch. At what step in training do you start your data rollouts for training the relational data influence model?\n\n2. Did you try different similarity functions or embedding models for the relational data influence model?", "limitations": "Yes, the authors have adequately discussed the limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nSection 2 and 3 clearly lays out previous work and the motivation for the paper. These introductory sections end with an experiment to demonstrate \"an empirical gap between group and individual data influence oracles.\" Starting with an intermediate checkpoint in the DCLM 400M-4x baseline, the goal is to choose $n$ data points that will best minimize the loss in the subsequent training steps. As a baseline, $n$ data points are chosen based on their individual oracle influences. For group-level influences, optimizing directly over the groups results in too large of a search space so instead the authors use a sequential process where the next data point is chosen greedily. The single point with minimal oracle individual influence is added to the selected data and a gradient step is taken with this point; the process is then repeated at each training step with the already chosen data points excluded. Figure 1 shows this results in very different data selection than the individual selection baseline and a substantial improvement in performance on downstream tasks.\n\nThe problem is this brute-force group selection is prohibitively expensive, as the oracle individual influences must be computed at each step. The paper proposes Group-MATES as a cheaper approximation. It builds on MATES which approximates individual data influence with a parametric model (a learned regression model on the activations of the last hidden layer of the language model). This is then scaled by a term dependent on the sum of some similarity metric between the data point under consideration and those already selected (i.e. the individual data influence is weighted according to its relation to the selected set) with trainable parameters. The parameters of the relational data influence model are trained by sampling data trajectories, computing the oracle individual influence for the selected data, and then minimizing the mean squared error between this and the proposed model. In short, the goal is for the trained model to approximate the oracle individual influences at each step of training for any individual data point so that this does not have to be computed as in the brute-force method. A final approximation and computational savings is provided by doing the procedure on clusters of the data rather than as a whole.\n\nThe method is compared to MATES and random selection in Table 1 and outperforms both these methods.\n\n\n\n**Weaknesses**\n\nWhile Group-MATES does outperform MATES, the gains are modest compared to this baseline and there remains a large gap between Group-MATES and the brute-force group selection (see Figure 4). Thus while the paper does make progress towards approximating this brute-force group selection, it is unclear whether the additional computational work (and overhead to set up the method) is worthwhile for practitioners over simply using MATES. In addition, all the computational comparisons (e.g. Table 5) are done in terms of FLOPS but I would hypothesize a number of the steps for Group-MATES are less efficient in terms of throughput than training steps (i.e. equal FLOPS would not lead to equal wall clock time). However, I would appreciate clarification from the authors on this based on their experiments. Overall, these observations are primarily reflected in my significance score.\n\nI think the right panel of Fig. 1 would be a better summary of your paper if it included the relational results as in Fig. 4b (the figures are identical other than this omission). Given the brute-force group method is computationally infeasible, it would be useful to communicate early on the performance of the proposed approximation. Also I would like to see the MATES baseline included in Figure 3.", "was_revised": true, "final_justification": "The gains from Group-MATES over MATES are modest which is still reflected in my significance score but the revisions from the rebuttal now give the reader a much clearer picture of the computational cost and relative efficiency gains of these methods. I thus recommend accept provided the revisions from the rebuttal are included in the final draft of the paper (wall clock time measurements and the MATES baseline is included in Figure 3) .", "rebuttal": "Thank you for your time and insightful review of our paper! We address your questions/comments below:\n\n**Weakness 1:** The gains over MATES are modest and there remains a large gap between Group-MATES and the brute-force group selection. It is unclear whether the additional computational work (and overhead to set up the method) is worthwhile for practitioners over simply using MATES. \n\n**Response:** In the table below, we demonstrate the actual wall clock time for each part of our method, compared with MATES and brute-force selection. The main difference between Group-MATES and MATES lies in the oracle data influence collection step, where Group-MATES collects 20k rollout trajectories (each containing 10 oracle data influences) and MATES collects 80k locally probed oracle data influences. However, this gap only accounts for **less than 3%** of the total wall clock time relative to pretraining hours. Notably, our data selection can be performed solely on CPUs after the data influence model inference, and is highly parallelizable—e.g., different clusters can utilize separate cores for independent selection. Considering the 2x net efficiency gain achieved by Group-MATES compared to MATES (see next response), this additional wall clock time is negligible.\n\nFurthermore, as you noticed, brute-force group selection is computationally infeasible, and that’s why we only use a 100-step training to showcase its performance. Group-MATES effectively approaches brute-force performance at a pretty low additional cost, and we hope our work inspires future research in this promising direction.\n\n|                                               | Group-MATES (400M-4x) | MATES (400M-4x) | Brute-Force Group Selection (400M-4x) | Group-MATES (1B-1x) | MATES (1B-1x) | Group-MATES (3B-1x) | MATES (3B-1x) |\n| --------------------------------------------- | --------------------- | --------------- | ------------------------------------- | ------------------- | ------------- | ------------------- | ------------- |\n| Model Pretraining (H100 Hours)                | 104                   | 104             | 104                                   | 240                 | 240           | 740                 | 740           |\n| Oracle Data Influence Collection (H100 Hours) | 4.5                   | 1.8             | 22632240                              | 12.1                | 4.8           | 18.7                | 7.5           |\n| Data Influence Model Training (H100 Hours)    | 2.7                   | 2.0             | -                                     | 2.7                 | 2.0           | 2.7                 | 2.0           |\n| Data Influence Model Inference (H100 Hours)   | 14.2                  | 14.2            | -                                     | 12.0                | 12.0          | 23.9                | 23.9          |\n| Data Selection (208-Core CPU Time)            | 10min                 | 1min            | 151min                                | 8min                | 1min          | 17min               | 2min          |\n\n---\n\n**Weakness 2:** MATES baseline in Figure 3.\n\n**Response:** Thank you for your suggestion! Below is the comparison of Group-MATES and MATES performance w.r.t. training tokens. For each method, we report **Core score/total FLOPs (1e19)/total GPU hours** in each cell. “Total” here means we consider all costs from data selection and model pretraining.\n\n*400M-4x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES           | Group-MATES     |\n| -------------------- | ---------------- | -------------- | --------------- | --------------- |\n| 19.7                 | 0.193/4.8/62     | 0.196/5.8/85   | 0.196/5.1/73    | 0.198/5.2/75    |\n| 26.2                 | 0.208/6.4/83     | 0.216/8.4/114  | 0.214/7.0/98    | *0.222/7.1/100* |\n| 32.8                 | 0.214/8.0/104    | 0.224/11.0/142 | *0.223/8.9/122* | 0.234/9.1/125   |\n| 39.4 (for reference) | 0.218/9.6/125    | -              | -               | -               |\n| 45.9 (for reference) | *0.221/11.2/146* | -              | -               | -               |\n\n*1B-1x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES            | Group-MATES      |\n| -------------------- | ---------------- | -------------- | ---------------- | ---------------- |\n| 16.8                 | 0.257/14.4/144   | 0.260/15.3/159 | 0.262/14.8/155   | 0.266/15.0/160   |\n| 22.4                 | 0.283/19.2/192   | 0.288/21.0/212 | 0.291/19.9/207   | 0.298/20.3/213   |\n| 28.0                 | 0.295/24.0/240   | 0.293/26.8/265 | *0.303/25.1/259* | *0.308/25.7/267* |\n| 33.6 (for reference) | *0.304/28.8/288* | -              | -                | -                |\n| 39.2 (for reference) | *0.308/33.6/336* | -              | -                | -                |\n\nWe can see that Group-MATES achieves **31.5%** and **20.5%** net efficiency gains (measured by GPU hours) in 400M-4x and 1B-1x setups, respectively. By contrast, MATES only achieves **16.4%** and **10.1%** net efficiency gains in 400M-4x and 1B-1x setups. Therefore, Group-MATES nearly **doubles** the net efficiency gain of MATES, further validating the advantage of our group-level selection over individual selection. We will include this comparison in Figure 3 and add a plot with GPU hours as the x-axis.\n\n---\n\n**Question 1:** The paper says the results in Table 1 are for models pretrained from scratch. At what step in training do you start your data rollouts for training the relational data influence model?\n\n**Response:** In the first 10B tokens, the model of Group-MATES is trained on randomly selected data. We then collect rollouts twice after training on 10B and 20B tokens for the 400M-4x and 1B-1x settings, and after 10B and 30B tokens for the 3B-1x setting. The model is continuously trained on the selected data, following the model-aware data selection pipeline in MATES (L206-L209). We will make this setup clear in our Section 5.\n\n---\n\n**Question 2:** Did you try different similarity functions or embedding models for the relational data influence model?\n\n**Response:** Yes. We experimented with different similarity functions (cosine, dot product, FFN) and embedding models (BERT vs. BGE) in Appendix C.4. We found that BGE outperforms BERT in approximating oracle data influences due to its optimization for sentence embeddings. Among similarity functions, using a trainable FFN achieves similar performance with our cosine similarity but adds parameters, while using dot product significantly reduces performance. These results validate our choice of cosine similarity that aligns with the original BGE. We will highlight this analysis in our main content and add an overview for Appendix C to better guide the readers.\n\n---\n\nWe sincerely thank you again for your valuable time and thoughtful review. We're delighted that you recognize the motivation of our method and highlight our superior performance. In our revised version, we will compare different methods w.r.t. wall clock time and put all baselines in Figure 3."}, {"review_id": "e64mRleAwX", "reviewer": "Reviewer_aEny", "summary": "The paper proposed Group-MATES as a variant of the MATES method using clustered groups on relational data influence scores and selecting in a parallel way, which enables efficient data selection.", "questions": "1. Have you compared the exact wall clock time of MATES and Group-MATES, including the breakdown of data selection process and the training process? \n2. Why is Figure 3 only containing the random baseline? Is there results from other baselines available?", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and clearly presented.\n2. The proposed method has a significant improvement ovoer random baselines.\n3. Comprehensive ablation studies have been provided.\n\nWeaknesses:\n1. While the speed-quality frontier is studied, it is using the number of tokens/flops. But the parallel nature of group selection should make is more efficient in wall-clock time but there seems to be no discussion or result on that. See Q1.\n2. Figure 3 seems misleading since it only compares with the random baseline. The performance improvement over other baselines seems rather marginal in Table 1, while I imagine the result would be more significant if the proposed method excels at wall-clock time. See Q2.\n3. The experiment is only conducted on one dataset/benchmark which might be limited, but I am not an expert in this domain so correct me if I am wrong.", "was_revised": true, "final_justification": "The rebuttal solved most concerns but I am not an expert in the domain so I raise my score but with low confidence.", "rebuttal": "Thank you for your time and insightful review of our paper! We address your questions/comments below:\n\n**Weakness 1 + Question 1:** Have you compared the exact wall clock time of MATES and Group-MATES, including the breakdown of data selection process and the training process?\n\n**Response:** In the table below, we demonstrate the actual wall clock time for each part of our method, compared with MATES and brute-force selection. The main difference between Group-MATES and MATES lies in the oracle data influence collection step, where Group-MATES collects 20k rollout trajectories (each containing 10 oracle data influences) and MATES collects 80k locally probed oracle data influences. However, this gap only accounts for **less than 3%** of the total wall clock time relative to pretraining hours. Notably, our data selection can be performed solely on CPUs after the data influence model inference, and is highly parallelizable—e.g., different clusters can utilize separate cores for independent selection. Considering the 2x net efficiency gain achieved by Group-MATES compared to MATES (see next response), this additional wall clock time is negligible.\n\n|                                               | Group-MATES (400M-4x) | MATES (400M-4x) | Brute-Force Group Selection (400M-4x) | Group-MATES (1B-1x) | MATES (1B-1x) | Group-MATES (3B-1x) | MATES (3B-1x) |\n| --------------------------------------------- | --------------------- | --------------- | ------------------------------------- | ------------------- | ------------- | ------------------- | ------------- |\n| Model Pretraining (H100 Hours)                | 104                   | 104             | 104                                   | 240                 | 240           | 740                 | 740           |\n| Oracle Data Influence Collection (H100 Hours) | 4.5                   | 1.8             | 22632240                              | 12.1                | 4.8           | 18.7                | 7.5           |\n| Data Influence Model Training (H100 Hours)    | 2.7                   | 2.0             | -                                     | 2.7                 | 2.0           | 2.7                 | 2.0           |\n| Data Influence Model Inference (H100 Hours)   | 14.2                  | 14.2            | -                                     | 12.0                | 12.0          | 23.9                | 23.9          |\n| Data Selection (208-Core CPU Time)            | 10min                 | 1min            | 151min                                | 8min                | 1min          | 17min               | 2min          |\n\n---\n\n**Weakness 2 + Question 2:** Why is Figure 3 only containing the random baseline? Are there results from other baselines available?\n\n**Response:** Yes, all baseline results are available except for WebOrganizer, for which we directly report the final results from their paper, as their code and data are not fully open-sourced yet. Figure 3 intends to show the net efficiency gain of Group-MATES over random. The table below shows the comprehensive result. For each method, we report **Core score/total FLOPs (1e19)/total GPU hours** in each cell. “Total” here means we consider all costs from data selection and model pretraining.\n\n*400M-4x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES           | Group-MATES     |\n| -------------------- | ---------------- | -------------- | --------------- | --------------- |\n| 19.7                 | 0.193/4.8/62     | 0.196/5.8/85   | 0.196/5.1/73    | 0.198/5.2/75    |\n| 26.2                 | 0.208/6.4/83     | 0.216/8.4/114  | 0.214/7.0/98    | *0.222/7.1/100* |\n| 32.8                 | 0.214/8.0/104    | 0.224/11.0/142 | *0.223/8.9/122* | 0.234/9.1/125   |\n| 39.4 (for reference) | 0.218/9.6/125    | -              | -               | -               |\n| 45.9 (for reference) | *0.221/11.2/146* | -              | -               | -               |\n\n*1B-1x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES            | Group-MATES      |\n| -------------------- | ---------------- | -------------- | ---------------- | ---------------- |\n| 16.8                 | 0.257/14.4/144   | 0.260/15.3/159 | 0.262/14.8/155   | 0.266/15.0/160   |\n| 22.4                 | 0.283/19.2/192   | 0.288/21.0/212 | 0.291/19.9/207   | 0.298/20.3/213   |\n| 28.0                 | 0.295/24.0/240   | 0.293/26.8/265 | *0.303/25.1/259* | *0.308/25.7/267* |\n| 33.6 (for reference) | *0.304/28.8/288* | -              | -                | -                |\n| 39.2 (for reference) | *0.308/33.6/336* | -              | -                | -                |\n\nWe can see that Group-MATES achieves **31.5%** and **20.5%** net efficiency gains (measured by GPU hours) in 400M-4x and 1B-1x setups, respectively. By contrast, MATES only achieves **16.4%** and **10.1%** net efficiency gains in 400M-4x and 1B-1x setups. Therefore, Group-MATES nearly **doubles** the net efficiency gain of MATES, further validating the advantage of our group-level selection over individual selection. We will include all baseline comparisons in Figure 3 and add a plot with GPU hours as the x-axis.\n\n---\n\n**Weakness 3:** The experiment is only conducted on one dataset/benchmark which might be limited.\n\n**Response:** The benchmark used in our main experiments, DCLM, is a standardized pretraining data curation benchmark that unifies the data selection pool, training scripts, and **22 Core evaluation tasks** that holistically and robustly evaluate the pretrained models (L211-L225). It has been widely adopted as the primary setup in a line of recent works on data selection [1], data mixing [2], and data synthesis [3], reflecting its status as a commonly recognized and authoritative benchmark for pretraining data curation.\n\nTo demonstrate the generalization ability of our method, we also conduct experiments on the C4 dataset in Appendix C.6, following the identical setup in MATES to select 25B tokens from a 125B-token pool. As shown in Table 7, Group-MATES consistently matches or surpasses the performance of the previous best method, MATES, on **8 out of 9** evaluation tasks. We will add a pointer to this setup in the main text to better guide readers.\n\n---\n\nWe sincerely thank you again for your valuable time and thoughtful review. We're delighted that you find our paper well-written, recognize the effectiveness of our method, and highlight our comprehensive ablation studies. In our revised version, we will compare different methods w.r.t. wall clock time and put all baselines in Figure 3.\n\n[1] Zhang, Jifan, et al. \"GPT-4o as the Gold Standard: A Scalable and General Purpose Approach to Filter Language Model Pretraining Data.\" arXiv preprint arXiv:2410.02755 (2024).\n\n[2] Wettig, Alexander, et al. \"Organize the Web: Constructing Domains Enhances Pre-Training Data Curation.\" ICML 2025.\n\n[3] Nguyen, Thao, et al. \"Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models.\" arXiv preprint arXiv:2506.04689 (2025)."}, {"review_id": "3YQ6hA6iqq", "reviewer": "Reviewer_aQV7", "summary": "This paper studies the data selection problem in the pre-training stage, which is an important issue. The paper differs itself from previous works in that it studies the combinatorial effect of different data group. To avoid tedious effort, the group-level selection is parameterized by a relational data influence model. The evaluation is conducted on DCLM at different scales and shows sufficient improvement.", "questions": "1. It appears that the improvement brought by Group-MATES reduces as the LLM model size increases. Does this mean that eventually all data selection will degrade to random selection, considering the size of SOTA LLMs?\n2. What are the differences between data selection in classic ML settings and LM settings?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper is generally well-written. The majority of its sections are relatively easy to read and understand. \n2. The intuition of the proposed method is clear and the proposed method, especially the parameterization of group-level selection, is backed by theoretical analysis.\n3. The proposed method demonstrates strong empirical results under different evaluation settings, showcasing the superiority of this approach.\n\nCons:\n\n1. Figure 1a is not very understandable, which may harm the readability of the introduction. The same goes for Fig 6b.\n2. The evaluation scope seems limited, only on the DCLM benchmark. Although the related LM benchmark is limited, extending the methods to other NLP tasks could better prove the consistent improvement under different settings.\n3. The content of the paper is heavily chunked due to the page limit. Please consider re-organizing the sections for better readability.\n4. Some of the content lacks further discussion, especially the content in Appendix C.", "was_revised": true, "final_justification": "Final Justification", "rebuttal": "Thank you for your time and insightful review of our paper! We address your questions/comments below:\n\n**Weakness 1:** Figure 1a/6b is not very understandable, which may harm the readability of the introduction.\n\n**Response:** Thank you for your suggestions! Figure 1a serves as our motivation that individual data selection (Eq. 8) often deviates from brute-force group selection (Eq. 9-11) as the dataset size increases. Details about the experimental setup and the definition of overlap are provided in our preliminary section (L141-L155). We will move Figure 1 directly into Section 3 for better readability.\n\nIn Figure 6b, we analyze the distribution of relationship weights $R$ (Eq. 14) between data points within the same cluster (intra-cluster) or across different clusters (inter-cluster). To enable efficient inference, we compute relationship weights only within each cluster in the selection process (L197-L198), and Figure 6b illustrates that this design preserves essential relationship information, as inter-cluster relationship weights are distributed around 0. We will add this clarification to the paper.\n\n---\n\n**Weakness 2:** The evaluation scope seems limited, only on the DCLM benchmark. Extending the methods to other NLP tasks could better prove the consistent improvement under different settings.\n\n**Response:** The benchmark used in our main experiments, DCLM, is a standardized pretraining data curation benchmark that unifies the data selection pool, training scripts, and **22 Core evaluation tasks** that holistically and robustly evaluate the pretrained models (L211-L225). It has been widely adopted as the primary setup in a line of recent works on data selection [1], data mixing [2], and data synthesis [3], reflecting its status as a commonly recognized and authoritative benchmark for pretraining data curation.\n\nTo demonstrate the generalization ability of our method, we also conduct experiments on the C4 dataset in Appendix C.6, following the identical setup in MATES to select 25B tokens from a 125B-token pool. As shown in Table 7, Group-MATES consistently matches or surpasses the performance of the previous best method, MATES, on **8 out of 9** evaluation tasks. We will add a pointer to this setup in the main text to better guide readers.\n\n---\n\n**Weakness 3 & 4:** The content of the paper is heavily chunked due to the page limit. Please consider re-organizing the sections for better readability. Some of the content lacks further discussion, especially the content in Appendix C.\n\n**Response:** Thank you for your suggestions! Appendix C demonstrates the additional results that are not able to be covered in the main content due to the page limit. However, we will re-organize these based on your comments. Specifically, we will:\n\n1. Move “C.1 FLOPs breakdown” and “C.2 Equalization for Compute” to main text to highlight the low additional cost and net efficiency gain of Group-MATES.\n2. Form “C.3 Selection Ratio”, “C.4 Design of Relational Data Influence Model”, and “C.5 Number of Collected Trajectories” into one section, as they are all ablations to validate our design choices and hyperparameters.\n3. Make C.6 “Comparison in MATES Setup” a separate section, as it is a new setup to show the generalization ability of our method.\n4. Add more cases to facilitate insightful discussions in C.7 Case Study.\n\n---\n\n**Question 1:** The improvement brought by Group-MATES reduces as the LLM model size increases. Does this mean that eventually all data selection will degrade to random selection, considering the size of SOTA LLMs?\n\n**Response:** This is an interesting question. It is a common finding [4] that larger models are more robust to data quality variations and consequently require more extensive training to fully manifest the benefits of data selection. To validate this, we further run a set of experiments on 1B-3x setup, where the training tokens are 3 times more than 1B-1x setup. Results are shown below:\n\n| 1B-3x       | Commonsense Reasoning | Language Understanding | Reading Comprehension | Symbolic Problem | World Knowledge | Core        |\n| :---------- | :-------------------- | :--------------------- | :-------------------- | :--------------- | :-------------- | :---------- |\n| Random      | 0.42433               | 0.44592                | 0.27744               | 0.17986          | 0.35294         | 0.33840     |\n| MATES       | 0.40872               | **0.45021**            | 0.29262               | 0.20697          | 0.34452         | 0.34376     |\n| Group-MATES | **0.43300**           | 0.44907                | **0.29663**           | **0.21737**      | **0.36320**     | **0.35391** |\n\nWe can see that the absolute Core score improvement of Group-MATES over random increases from 1.3% (1B-1x) to 1.6% (1B-3x), doubling the gains achieved by MATES. These results demonstrate that Group-MATES consistently maintains its performance advantage even as model size and training data scale up. We will add it in the paper.\n\nData selection is a very common and essential practice in building state-of-the-art LLMs, as highlighted in Gemini [5], Qwen [6], and DeepSeek [7] reports. In general, web-crawled data is often noisy and necessitates extensive filtering and delicate algorithms to select high-quality data. Our work proposes a novel group-level selection framework that maximizes group-level data utility at a low cost, and we believe it remains effective in production-level pretraining data construction, as group-level selection is still an underexplored yet critical topic.\n\n---\n\n**Question 2:** What are the differences between data selection in classic ML settings and LM settings?\n\n**Response:** We summarize three key points that characterize the differences between data selection in classic machine learning and language model pretraining:\n\n1. **Data Scale:** The scale of data used in LLM pretraining is often orders of magnitude larger than that in traditional ML settings. This significant difference poses challenges for many conventional methods (e.g., Shapley value [8]), which are computationally tractable in classic computer vision tasks such as CIFAR-10 and CIFAR-100, but become infeasible for large-scale pretraining data selection.\n2. **Data Quality:** Classic ML settings typically operate on supervised datasets where most data is already of high quality. In contrast, the raw web corpus in LM pretraining is often noisy, providing greater potential gains from identifying and selecting high-quality subsets of data.\n3. **Complexity:** LM data selection is not a single-step process, but a combinational and multi-stage pipeline that includes rule-based filtering, deduplication, and quality assessment. Each stage provides its distinct value. Our work focuses on the later stages of this pipeline, where basic cleaning has been performed and model-oriented selection yields more substantial benefits.\n\n---\n\nWe sincerely thank you again for your valuable time and thoughtful review. We're delighted that you find our paper well-written, recognize the motivation of our method, and highlight our superior performance. In our revised version, we will re-organize the paper for better readability and add 1B-3x results.\n\n[1] Zhang, Jifan, et al. \"GPT-4o as the Gold Standard: A Scalable and General Purpose Approach to Filter Language Model Pretraining Data.\" arXiv preprint arXiv:2410.02755 (2024).\n\n[2] Wettig, Alexander, et al. \"Organize the Web: Constructing Domains Enhances Pre-Training Data Curation.\" ICML 2025.\n\n[3] Nguyen, Thao, et al. \"Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models.\" arXiv preprint arXiv:2506.04689 (2025).\n\n[4] Chang, Ernie, et al. \"Scaling Parameter-Constrained Language Models with Quality Data.\" EMNLP 2024.\n\n[5] Comanici, Gheorghe, et al. \"Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.\" arXiv preprint arXiv:2507.06261 (2025).\n\n[6] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025).\n\n[7] Liu, Aixin, et al. \"Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.\" arXiv preprint arXiv:2405.04434 (2024).\n\n[8] Cai, Huaiguang. \"CHG Shapley: Efficient Data Valuation and Selection towards Trustworthy Machine Learning.\" arXiv preprint arXiv:2406.11730 (2024)."}, {"review_id": "4Nrh1QdOpG", "reviewer": "Reviewer_NRhp", "summary": "This paper introduces a new data selection approach Group-MATES to optimize the speed-quality frontier of LLM pretraining. The relational data influence model is introduced and is used as a heuristic to approximate data influence. Experiments show improvement over SOTA.", "questions": "1. What are the parameters to be optimized in \\Theta_rel? What is w_o? I see some explaination for equation (7) but that's for MATES.\n2. For Appendix A.2 Connection between Relationship Weight and Trajectory-Specific Influence Function, \"relationship weight R in Eq. 14 serves a similar purpose by reweighting the influence of a data point based on its relationships with other points in the training trajectory\" this seems quite a large jump to the heuristics you are using. In Section 3, the discussions are on individual data influences (group-level influences are also approximated by decomposition into individual influences). Can you provide more insights on why you use heuristics that consider interactive effects?\n3. What is the computational overhead of Group-MATES? It seems the computation of (13) grows linearly with t as it considers the interaction of x_t with all previous data?\n4. How do Group-MATES compare with MATES?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The authors have carried out extensive experiments and the experimental setup are detailed and well explained.\n\nWeakness: The algorithm description is unclear and hard to follow. Despite the presence of Figure 2, I still struggle to understand how algorithm works. Notations in 4.1 are introduced without proper explanation (13) - (15).\n\"Theoretical analysis\" in A.2 seems forced and not directly related to (13) - (15). The intuition is not clear from the explanation. (See questions below)", "was_revised": true, "final_justification": "In the responses, authors addressed my concerns in great detail. Therefore, I raise my score.\nHowever, since this is not in my area, I am not confident about my review.", "rebuttal": "Thank you for your time and insightful review of our paper! We address your questions/comments below:\n\n**Question 1:** What are the parameters to be optimized in \\Theta_rel? What is w_o?\n\n**Response:** In Section 4, we reuse the notation from our preliminary section (L133-L136). Therefore, the trainable parameters of $\\Theta_{\\text{rel}}$ still consist of two parts, (1) a pretrained base language model that produces sentence representation $\\textbf{h}\\_{x_i}$ and (2) a regression head $\\textbf{w}_o$ with input_dim = hidden size and output_dim = 1. The initialization of these two parts is described in L237-L238, with further details provided in L502-L511. We will reiterate the definitions of these notations in Section 4 in the revised version.\n\n---\n\n**Question 2:** In Section 3, the discussions are on individual data influences (group-level influences are also approximated by decomposition into individual influences). Can you provide more insights on why you use heuristics that consider interactive effects?\n\n**Response:** As revealed by prior studies [1, 2], group-level influences often deviate from the sum of individual influences since the interactions among data points can sometimes cancel out or amplify individual influences. To capture these interactive dynamics, we build our relational data influence model with relationship weights (Eq. 14) to capture the interactive effects among training data. A case study illustrating such effects is presented in Appendix C.7, and its outcome aligns with intuitive expectations. We will add a reference to this study in the main text.\n\nWe mention trajectory-specific influence function in Appendix A.2, as we find it an interesting theoretical tool that quantifies data influence given the training trajectory. However, computing exact Hessian products for each training data (i.e., Eq. 40) is significantly more expensive than our method that parameterizes data influence with a relational model. Therefore, in their original paper [3], the primary setup is pretraining on 1% of the Pile dataset (~3B tokens), while ours can efficiently scale to 56B tokens (3B-1x) and beyond. We are willing to incorporate direct performance comparisons with this recent work once it is open-sourced.\n\n---\n\n**Question 3:** What is the computational overhead of Group-MATES? The computation of (13) grows linearly with t as it considers the interaction of x_t with all previous data?\n\n**Response:** We have shown the computational overhead measured by FLOPs in Appendix C.1 and by wall clock time in the table below. The findings are consistent: Group-MATES accounts for only a very small portion (e.g., 3.4% in 3B-1x setup) of the total computation, and the relative overhead is generally smaller in larger setups since pretraining FLOPs dominate the computation. We will move this computational analysis to the main text.\n\n|                                               | Group-MATES (400M-4x) | MATES (400M-4x) | Brute-Force Group Selection (400M-4x) | Group-MATES (1B-1x) | MATES (1B-1x) | Group-MATES (3B-1x) | MATES (3B-1x) |\n| --------------------------------------------- | --------------------- | --------------- | ------------------------------------- | ------------------- | ------------- | ------------------- | ------------- |\n| Model Pretraining (H100 Hours)                | 104                   | 104             | 104                                   | 240                 | 240           | 740                 | 740           |\n| Oracle Data Influence Collection (H100 Hours) | 4.5                   | 1.8             | 22632240                              | 12.1                | 4.8           | 18.7                | 7.5           |\n| Data Influence Model Training (H100 Hours)    | 2.7                   | 2.0             | -                                     | 2.7                 | 2.0           | 2.7                 | 2.0           |\n| Data Influence Model Inference (H100 Hours)   | 14.2                  | 14.2            | -                                     | 12.0                | 12.0          | 23.9                | 23.9          |\n| Data Selection (208-Core CPU Time)            | 10min                 | 1min            | 151min                                | 8min                | 1min          | 17min               | 2min          |\n\nThe computation of Eq. 13 indeed grows linearly with $t$ (or our selected size $n$), which motivates our design of cluster-based selection (Section 4.3). By distributing the selected size into separate clusters, we significantly reduce the time complexity by $d$ (the number of clusters). Furthermore, our data selection can be performed solely on CPUs after one-time data influence model inference (to get $\\textbf{h}\\_{x_i}$ and $\\textbf{w}_o\\cdot\\textbf{h}\\_{x_i}$), and is highly parallelizable—e.g., different clusters can utilize separate cores for independent selection. The actual efficiency gain of this cluster-based method is demonstrated in Figure 6a.\n\n---\n\n**Question 4:** How do Group-MATES compare with MATES?\n\n**Response:** From the methodology perspective, MATES approaches individual data selection by modeling individual data influences given a certain model state (Eq. 7), whereas Group-MATES approaches group-level data selection by modeling both individual data influences as well as the relationships among data points given model training trajectories (Eq. 13). As a language model is ultimately trained on a group of data step-by-step, the selection procedure of Group-MATES better aligns with the actual training process.\n\nTo fairly compare Group-MATES with MATES, we report **Core score/total FLOPs (1e19)/total GPU hours** in the table below. “Total” here means we consider all costs from data selection and model pretraining. We can see that Group-MATES only incurs **~3% additional FLOPs/wall clock time** compared to MATES, yet **doubles the net efficiency gain** (compute reduction for reaching a certain performance) of MATES over random selection across different scales. This highlights the benefits of pursuing group-level selection with our relational data influence model.\n\n*400M-4x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES           | Group-MATES     |\n| -------------------- | ---------------- | -------------- | --------------- | --------------- |\n| 19.7                 | 0.193/4.8/62     | 0.196/5.8/85   | 0.196/5.1/73    | 0.198/5.2/75    |\n| 26.2                 | 0.208/6.4/83     | 0.216/8.4/114  | 0.214/7.0/98    | *0.222/7.1/100* (net efficiency gain = 31.5%) |\n| 32.8                 | 0.214/8.0/104    | 0.224/11.0/142 | *0.223/8.9/122* (net efficiency gain = 16.4%) | 0.234/9.1/125   |\n| 39.4 (for reference) | 0.218/9.6/125    | -              | -               | -               |\n| 45.9 (for reference) | *0.221/11.2/146* | -              | -               | -               |\n\n*1B-1x (cells used to calculate net efficiency gain are marked italic):*\n\n| Tokens (B)           | Random           | Quad           | MATES            | Group-MATES      |\n| -------------------- | ---------------- | -------------- | ---------------- | ---------------- |\n| 16.8                 | 0.257/14.4/144   | 0.260/15.3/159 | 0.262/14.8/155   | 0.266/15.0/160   |\n| 22.4                 | 0.283/19.2/192   | 0.288/21.0/212 | 0.291/19.9/207   | 0.298/20.3/213   |\n| 28.0                 | 0.295/24.0/240   | 0.293/26.8/265 | *0.303/25.1/259* (net efficiency gain = 10.1%) | *0.308/25.7/267* (net efficiency gain = 20.5%) |\n| 33.6 (for reference) | *0.304/28.8/288* | -              | -                | -                |\n| 39.2 (for reference) | *0.308/33.6/336* | -              | -                | -                |\n\n---\n\nWe sincerely thank you again for your valuable time and thoughtful review. We're delighted that you find our experiments extensive and well explained. In our revised version, we will clarify notations in our method section, add an overview algorithm for our method, and highlight the computational overhead analysis in the main text.\n\n[1] Hu, Yuzheng, et al. \"Most Influential Subset Selection: Challenges, Promises, and Beyond.\" NeurIPS 2024.\n\n[2] Huang, Jenny Y., et al. \"Approximations to worst-case data dropping: unmasking failure modes.\" arXiv preprint arXiv:2408.09008 (2024).\n\n[3] Wang, Jiachen T., et al. \"Capturing the Temporal Dependence of Training Data Influence.\" ICLR 2025."}]

Author rebuttal:

Meta-review: This paper studies group-level data selection for efficient language model pretraining. The authors introduce Group-MATES, a method that leverages a separately trained relational data influence model for data selection. The authors also propose a cluster-based approach to further accelerate the selection process. Empirical results demonstrate the effectiveness of the proposed approach. After the rebuttal, all reviewers leaned toward acceptance.
Decision: Accept (poster)