Title: Group-Level Data Selection for Efficient Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Pre-training

Data Selection

Data Influence
Abstract: The efficiency and quality of language model pretraining are largely determined by the way pretraining data are selected. In this paper, we introduce *Group-MATES*, an efficient group-level data selection approach to optimize the speed-quality frontier of language model pretraining. Specifically, Group-MATES parameterizes costly group-level selection with a relational data influence model. To train this model, we sample training trajectories of the language model and collect oracle data influences alongside. The relational data influence model approximates the oracle data influence by weighting individual influence with relationships among training data. To enable efficient selection with our relational data influence model, we partition the dataset into small clusters using relationship weights and select data within each cluster independently. Experiments on DCLM 400M-4x, 1B-1x, and 3B-1x show that Group-MATES achieves 3.5\%-9.4\% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improvements achieved by state-of-the-art individual data selection baselines. Furthermore, Group-MATES reduces the number of tokens required to reach a certain downstream performance by up to 1.75x, substantially elevating the speed-quality frontier. Further analyses highlight the critical role of relationship weights in the relational data influence model and the effectiveness of our cluster-based inference. Our code is open-sourced at https://github.com/facebookresearch/Group-MATES.


Peer reviews:
comments: SUMMARY:
This paper introduces a new data selection approach Group-MATES to optimize the speed-quality frontier of LLM pretraining. The relational data influence model is introduced and is used as a heuristic to approximate data influence. Experiments show improvement over SOTA.

STRENGTHS AND WEAKNESSES:
Strengths: The authors have carried out extensive experiments and the experimental setup are detailed and well explained.

Weakness: The algorithm description is unclear and hard to follow. Despite the presence of Figure 2, I still struggle to understand how algorithm works. Notations in 4.1 are introduced without proper explanation (13) - (15).
"Theoretical analysis" in A.2 seems forced and not directly related to (13) - (15). The intuition is not clear from the explanation. (See questions below)

QUESTIONS:
1. What are the parameters to be optimized in \Theta_rel? What is w_o? I see some explaination for equation (7) but that's for MATES.
2. For Appendix A.2 Connection between Relationship Weight and Trajectory-Specific Influence Function, "relationship weight R in Eq. 14 serves a similar purpose by reweighting the influence of a data point based on its relationships with other points in the training trajectory" this seems quite a large jump to the heuristics you are using. In Section 3, the discussions are on individual data influences (group-level influences are also approximated by decomposition into individual influences). Can you provide more insights on why you use heuristics that consider interactive effects?
3. What is the computational overhead of Group-MATES? It seems the computation of (13) grows linearly with t as it considers the interaction of x_t with all previous data?
4. How do Group-MATES compare with MATES?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper studies the data selection problem in the pre-training stage, which is an important issue. The paper differs itself from previous works in that it studies the combinatorial effect of different data group. To avoid tedious effort, the group-level selection is parameterized by a relational data influence model. The evaluation is conducted on DCLM at different scales and shows sufficient improvement.

STRENGTHS AND WEAKNESSES:
Pros:

1. The paper is generally well-written. The majority of its sections are relatively easy to read and understand. 
2. The intuition of the proposed method is clear and the proposed method, especially the parameterization of group-level selection, is backed by theoretical analysis.
3. The proposed method demonstrates strong empirical results under different evaluation settings, showcasing the superiority of this approach.

Cons:

1. Figure 1a is not very understandable, which may harm the readability of the introduction. The same goes for Fig 6b.
2. The evaluation scope seems limited, only on the DCLM benchmark. Although the related LM benchmark is limited, extending the methods to other NLP tasks could better prove the consistent improvement under different settings.
3. The content of the paper is heavily chunked due to the page limit. Please consider re-organizing the sections for better readability.
4. Some of the content lacks further discussion, especially the content in Appendix C.

QUESTIONS:
1. It appears that the improvement brought by Group-MATES reduces as the LLM model size increases. Does this mean that eventually all data selection will degrade to random selection, considering the size of SOTA LLMs?
2. What are the differences between data selection in classic ML settings and LM settings?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
The paper proposed Group-MATES as a variant of the MATES method using clustered groups on relational data influence scores and selecting in a parallel way, which enables efficient data selection.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-written and clearly presented.
2. The proposed method has a significant improvement ovoer random baselines.
3. Comprehensive ablation studies have been provided.

Weaknesses:
1. While the speed-quality frontier is studied, it is using the number of tokens/flops. But the parallel nature of group selection should make is more efficient in wall-clock time but there seems to be no discussion or result on that. See Q1.
2. Figure 3 seems misleading since it only compares with the random baseline. The performance improvement over other baselines seems rather marginal in Table 1, while I imagine the result would be more significant if the proposed method excels at wall-clock time. See Q2.
3. The experiment is only conducted on one dataset/benchmark which might be limited, but I am not an expert in this domain so correct me if I am wrong.

QUESTIONS:
1. Have you compared the exact wall clock time of MATES and Group-MATES, including the breakdown of data selection process and the training process? 
2. Why is Figure 3 only containing the random baseline? Is there results from other baselines available?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on trying to improve pretraining data selection by considering the utility of groups of training examples rather than just considering the influence of a training example individually. As a proof of concept, the group data selection is first done in a computationally expensive, brute force manner. The authors then propose a computationally cheaper approximation called Group-MATES for doing group-level data selection. This method is then tested on training on setups from the DCLM competition at 3 different scales. Finally ablations are performed on the various components of Group-MATES.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nSection 2 and 3 clearly lays out previous work and the motivation for the paper. These introductory sections end with an experiment to demonstrate \"an empirical gap between group and individual data influence oracles.\" Starting with an intermediate checkpoint in the DCLM 400M-4x baseline, the goal is to choose $n$ data points that will best minimize the loss in the subsequent training steps. As a baseline, $n$ data points are chosen based on their individual oracle influences. For group-level influences, optimizing directly over the groups results in too large of a search space so instead the authors use a sequential process where the next data point is chosen greedily. The single point with minimal oracle individual influence is added to the selected data and a gradient step is taken with this point; the process is then repeated at each training step with the already chosen data points excluded. Figure 1 shows this results in very different data selection than the individual selection baseline and a substantial improvement in performance on downstream tasks.\n\nThe problem is this brute-force group selection is prohibitively expensive, as the oracle individual influences must be computed at each step. The paper proposes Group-MATES as a cheaper approximation. It builds on MATES which approximates individual data influence with a parametric model (a learned regression model on the activations of the last hidden layer of the language model). This is then scaled by a term dependent on the sum of some similarity metric between the data point under consideration and those already selected (i.e. the individual data influence is weighted according to its relation to the selected set) with trainable parameters. The parameters of the relational data influence model are trained by sampling data trajectories, computing the oracle individual influence for the selected data, and then minimizing the mean squared error between this and the proposed model. In short, the goal is for the trained model to approximate the oracle individual influences at each step of training for any individual data point so that this does not have to be computed as in the brute-force method. A final approximation and computational savings is provided by doing the procedure on clusters of the data rather than as a whole.\n\nThe method is compared to MATES and random selection in Table 1 and outperforms both these methods.\n\n\n\n**Weaknesses**\n\nWhile Group-MATES does outperform MATES, the gains are modest compared to this baseline and there remains a large gap between Group-MATES and the brute-force group selection (see Figure 4). Thus while the paper does make progress towards approximating this brute-force group selection, it is unclear whether the additional computational work (and overhead to set up the method) is worthwhile for practitioners over simply using MATES. In addition, all the computational comparisons (e.g. Table 5) are done in terms of FLOPS but I would hypothesize a number of the steps for Group-MATES are less efficient in terms of throughput than training steps (i.e. equal FLOPS would not lead to equal wall clock time). However, I would appreciate clarification from the authors on this based on their experiments. Overall, these observations are primarily reflected in my significance score.\n\nI think the right panel of Fig. 1 would be a better summary of your paper if it included the relational results as in Fig. 4b (the figures are identical other than this omission). Given the brute-force group method is computationally infeasible, it would be useful to communicate early on the performance of the proposed approximation. Also I would like to see the MATES baseline included in Figure 3.\n\nQUESTIONS:\n1. The paper says the results in Table 1 are for models pretrained from scratch. At what step in training do you start your data rollouts for training the relational data influence model?\n\n2. Did you try different similarity functions or embedding models for the relational data influence model?\n\nLIMITATIONS:\nYes, the authors have adequately discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-08"}