Title: DataRater: Meta-Learned Dataset Curation
Venue: NeurIPS 2025
Accepted: yes
Keywords: meta-learning

dataset curation

data rating

bilevel optimization
Abstract: The quality of foundation models depends heavily on their training data.
Consequently, great efforts have been put into dataset curation.
Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics.
An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training.
This type of meta-learning could allow more sophisticated, fine-grained, and effective curation.
Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data.
In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel meta-learning approach called DataRater for curating data for training LLMs. The DataRater framework trains a data-selector model in a meta-learning fashion by optimizing through unrolled inner optimizations of an actual LM being trained. The paper conducts experiments across different model scales on three publicly available datasets. Empirical evidence shows that the trained DataRater models can select high-quality training data for LMs, thereby improving compute efficiency of training. Further analysis on the type of training data samples selected by the DataRater are also provided to guide intuitions on how DataRater works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is very well written and provides a great overview of the method, both from an intuition perspective and a technical implementation perspective.\n- The problem outlined by the paper---improving training efficiency via data curation---is an extremely important problem and the paper does a good job of characterizing it.\n- The intuitions provided by the analysis in Section 3 (Figures 7 and 8) are very helpful in understanding what data samples are selected by DataRater. More papers in the data curation literature should adopt this.\n\nWeaknesses:\n- From figure 1 and as discussed in Section 3, it seems like DataRater brings large boosts for relatively under-filtered datasets like Pile and C4/no-clean, whereas for a reasonably well filtered dataset like C4, the gains seem to be a little bit more modest. This raises the question if DataRater is complementary to other LM data curation / filtering methods or not? The benefits of the SoTA filtering pipelines used in current LMs is that they are, for the most part, complementary i.e. you can use multiple fast-text classifiers together, and performance improves (see [Nemotron-CC](https://arxiv.org/abs/2412.02595) paper). Hence, it is an important question to answer if the DataRater method is complementary or orthogonal to more such filters i.e. can DataRater be used as one filter in a stack of many more filters?\n- DataRater assumes access to a test distribution. In the set of experiments conducted, that is just from the same distribution as the training distribution. What happens when your test distribution is quite different from your training distribution? For example, could the authors discuss / ground their findings with respect to the experiments conducted in the [Color-filter](https://proceedings.neurips.cc/paper_files/paper/2024/file/b0f25f0a63cc544d506e4c1374a3c807-Paper-Conference.pdf) did, where they explicitly have a “validation set” that is the target distribution of interest.\n- The paper misses out on a lot of relevant literature that also tackles the same problem of data curation for LMs. Some notable omissions are: [DataComp-LM](https://arxiv.org/abs/2502.10341), [AskLLM](https://arxiv.org/pdf/2402.09668), [Qurating](https://arxiv.org/abs/2402.09739), [Nemotron-CC](https://arxiv.org/abs/2412.02595), [DSIR](https://arxiv.org/abs/2302.03169), [WebOrganizer](https://arxiv.org/abs/2502.10341), [SemDeDup](https://arxiv.org/abs/2303.09540) etc.\n- There is a lack of good baselines. All the experiments conducted only show comparisons to a standard baseline that is trained on uniformly sampled data. However, simple baselines like training a fast-text classifier on the same training data distribution as DataRater as the positives are missing. It is unclear if DataRater would substantially outperform such simple data selection methods, justifying the additional cost of the meta-learning training (given that training a fast-text classifier is extremely cheap in comparison).\n\nQUESTIONS:\n- Doesn’t DataRater require manual tuning of the threshold as well? It seems a bit disingenuous to claim that your method does not need manual tuning (in line )? It requires the same amount of tuning as a fast-text classifier pretty much?\n- Does DataRater also work for >1-epoch training or is it limited to the single-repeat setting? From the description, it is unclear if the experiments are conducted for a single or multi-epoch setting (especially given the different discard fractions used).\n- Why do the inner model parameters need to be reset? Does this not making the DataRater somewhat off-policy, compared to the learning signal gotten from the updated inner models? A more detailed explanation of this would be good to add in the paper.\n- One thing that is not clear to me from this statement in lines 168-172: Is this using offline DataRater or an online version that filters out every batch? Do you assume amortization of running the DataRater model on the full dataset once, or do you still count that towards the total FLOPs needed for training? The claims in the paper around compute efficiency sometimes use number of training steps, and sometimes use FLOPs. If indeed the cost of running inference of DataRater is not accounted for, making claims about compute efficiency purely using number of training steps is not fair in my opinion.\n\nLIMITATIONS:\nYes, the paper provides a good list of limitations, both technical and societal, in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"DataRater: Meta-Learned Dataset Curation\" proposes a method for estimating the value of individual data points for training foundation models, using meta-learning with meta-gradients. The authors frame their contribution as making LLM training more data-efficient, especially since we are running out of human-curated text to train on.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* Topic: The problem of curating training data for foundation models, especially with increasing reliance on synthetic and noisy data, is critical and underexplored at scale.\n\n* Empirical results: Consistent performance and compute savings across model scales (50M to 1B) and diverse datasets (The Pile, C4, C4/noclean). Also, the authors show that a DataRater trained on 400M inner models generalizes to smaller and larger models. This is practically useful and rarely demonstrated in other works. Cute qualitative analysis. \n\n* Theoretical grounding: Meta-learning data value, bilevel optimization, and filtering via data scoring are all well-explored. The novelty here is in their combination. This is a rather incremental novelty, but still a novelty.\n\n\n**Weaknesses:**\n\n* Baseline: There’s no head-to-head comparison with existing learned filtering approaches like SEAL, JST, or PreSelect (only to the unfiltered baseline). There's also no vanilla baseline comparison, as I'd expect in figure 4. \n\n* Downstream utility: The evaluation focuses on proxy tasks (HellaSwag, PIQA), not on a large downstream deployment (e.g., instruction tuning, chat models). Claims about general usability or production utility would benefit from such an example.\n\n* Compute cost: Training the DataRater takes ~58% the compute of training a 1B LLM. The argument about amortizing this cost is valid, but practically speculative without end-to-end downstream deployment scenarios.\n\n* High-quality data: The method is much more effective on noisy data (Pile, C4/noclean); on already filtered datasets like C4, the benefits are small and sometimes ambiguous.\n\n* Technical details: While referencing MixFlow-MG and other tricks for scalability, the paper provides limited detail on hyperparameters, stability, or how these techniques differ from baseline approaches in practical terms. For reproducibility or for researchers hoping to build on this, more clarity would be welcome.\n\nQUESTIONS:\n* How does DataRater compare against recently learned data selection methods?\n\n* Since the DataRater is trained to optimize performance on a held-out subset of the same dataset, how do you ensure that it isn’t overfitting to the meta-validation metric or gaming spurious correlations in that specific split? Could you explore robustness to alternative outer losses/distribution shift between D_train and D_test/generalization to truly different downstream tasks?\n\n* On a similar note, how sensitive is DataRater to the choice of outer loss, or shifts in domain/task?\n\nNitpicking: Please make sure that your figures are in the right place and that you refer to them in the text itself (e.g., the first time you refer to figure 1, which is on page 2, is on page 6; figures 2-4 were never referred to).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DataRater, a meta-learning framework for automated dataset curation that learns to estimate the training value of individual data points for foundation models. The approach addresses the current reliance on manual heuristics and coarse-grained data mixing in large-scale model training.\n\nDataRater uses a transformer-based scorer that assigns scores to data points, trained via bilevel optimization with meta-gradients. The meta-learning objective is to improve training efficiency on held-out validation data. The method employs a continuous relaxation of discrete data selection, using softmax-weighted gradients during training and top-K filtering during actual curation.\n\nThe authors evaluate DataRater across three datasets of varying quality (C4, C4/noclean, The Pile) and four model scales (50M to 1B parameters). A key finding is cross-scale generalization: a DataRater trained with 400M parameter inner models effectively transfers to target models ranging from 50M to 1B parameters, with consistent optimal filtering proportions across scales.\n\nThe method achieves substantial compute savings on lower-quality datasets, with up to 46.6% net compute reduction on The Pile while matching baseline performance. Results show DataRater learns to down-weight data that aligns with human intuitions of poor quality, such as OCR errors, encoding issues, and non-English content. The approach demonstrates improvements across 73 out of 84 experimental configurations spanning multiple datasets, model scales, and evaluation metrics.\n\nThe technical implementation leverages MixFlow-MG for scalable bilevel optimization, making the approach computationally feasible for realistic model and dataset sizes. The framework supports both batch-level filtering for efficiency and individual data point scoring for parallel processing pipelines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Quality**: The technical approach is sound with proper bilevel optimization formulation and scalable implementation using MixFlow-MG. The cross-scale generalization finding (DataRater trained on 400M models works for 50M-1B targets) is empirically robust and theoretically interesting. Experiments span multiple datasets, model scales, and metrics, providing comprehensive evaluation.\n- **Clarity**: The paper is well-written with clear mathematical formulation and algorithmic specification. The meta-learning framework is explained intuitively, and experimental setup is thoroughly described. Figures effectively communicate key results, particularly the compute efficiency gains.\n- **Significance**: Addresses an important scalability challenge in foundation model training. Achieves substantial compute savings (up to 46.6%) on noisy datasets while maintaining performance. The cross-scale generalization property has important practical implications for amortizing curation costs across model sizes.\n\n### Weaknesses\n- **Quality**: gap in baseline comparisons—only compares against unfiltered data, not against established heuristic filtering, perplexity-based methods, or random subsampling. This makes it impossible to assess whether neural approaches justify their computational overhead versus simpler alternatives.\n- **Clarity**: Limited analysis of what DataRater actually learned beyond qualitative examples. No systematic feature analysis or rule extraction that could provide transferable insights to the community.\n**Significance**: Minimal gains on well-curated datasets (C4) where most practitioners operate, limiting immediate practical value. Training cost (~58.4% of target model) isn't justified by small improvements on quality data.\n- **Originality**: While the application is novel, the core bilevel optimization techniques are established. The continuous relaxation approach follows standard practices in differentiable discrete optimization.\n\nQUESTIONS:\n**1. Baseline Comparisons** How does DataRater compare against established curation methods beyond unfiltered data? Please provide comparisons against: (a) heuristic filtering similar to C4's pipeline applied to The Pile, (b) perplexity-based filtering, and (c) random subsampling at equivalent discard rates. **Criterion**: Adding proper baselines would increase Quality if DataRater shows meaningful advantages over ablated heuristics.\n\n**2. Cost-Benefit Analysis and Decision Framework** When is DataRater worth the ~58.4% training cost versus cheaper alternatives? Please provide: (a) clear decision criteria for when to use DataRater vs. heuristics, (b) break-even analysis for different dataset quality levels, and (c) guidelines for practitioners. **Criterion**: Clear practical guidance would increase Significance.\n\n**3. Systematic Analysis of Learned Patterns** Can you extract interpretable rules from DataRater beyond qualitative examples? Please provide: (a) quantitative correlation analysis between scores and measurable text features, (b) comparison of learned patterns vs. existing heuristics, and (c) feature importance analysis that could inform simpler filtering methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with finding most influencial training data for foundational models in order to improve training efficiency, but also possibly predictive performance. The goal of a filtering process is to select a subset of training data that leads to the smallest test error. The DataRater approach is to solve a continuous linear relaxation for discrete subsampling of data, where for each gradient update and minibatch a continuous weight in [0,1] is chosen. The weights are represented by a learned function, the DataRater model, where the outputs are normalized by a softmax preference function Parameter opt. is done via meta-Learning of DataRater model parameteres by backpropagrating through multiple inner loops. The authors build their implementation on prior work MixFlow-MG and a population of inner models for meta-gradient calculation, in order make optimization feasible and stable. The data curation is then performed using top-K filtering, i.e., removing data with low predicted weight.\n\nThe system is evaluated by learning DataRater, then training a LM of sizes up to 1B from random init. on the resulting data. Datasets used are C4 variants and Pile. The Inner LLMs and DataRater model are based on Chinchilla architectures. The final accuracy is measured on HellaSwag, SIQA, PIQA, ARC Easy, Commonsense QA downstream tasks. The results show that DataRater accelerates learning (i.e., FLOP savings) esp. for lower quality datasets. The authors also evaluate the best data drop rates, which correlates with the amount of available noise.\n\nSTRENGTHS AND WEAKNESSES:\n## Pros\n- Clear presentation of methods and results\n- Sensible and original meta-learning approach to learn a data value function\n- Good empirical results for the method, showing efficiency and predictive performance improvements compared to training with all data\n\n## Cons\n- No empirical comparison to competitors and not sufficient conceptual argumentation why new approach is adding value wrt existing work.\n- Missing argumentation for the choice of datasets wrt (task-) complexity\n\nQUESTIONS:\n- The pipeline first trained a DataRater model using multiple inner LLMs, then trains the final LLM using the curated data. Could one also train such a final LLM directly, similar to the inner LLMs? I understand that the DataRater model should be reused multiple times to actually save compute, but could end-to-end optimization also fine the optimal amount of needed data?\n- Would Multi-criteria optimization be possible to also reduce dataset size directly?\n- What are the advantages over competing approach mentioned in related works - e.g., influence functions, SHAP values, JST, ... ? More efficient? Better quality assessments?\n- T=2 inner models updates have been found to be best. Did the performance remain stable with more updates and steadily increase? Can you please give possible explanations for the good fit of T=2?\n- How complex are the chosen datasets? Do the findings generalize to any complexity of text-based downstream task?\n- Is the optimization process sufficiently stable for updating a trained DataRater wrt new data?\n\nLIMITATIONS:\nLimitations become clearer when answering the questions in the above section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}