Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice
Authors: Jiachen T. Wang
Tong Wu
Kaifeng Lyu
James Zou
Dawn Song
Ruoxi Jia
Prateek Mittal
Venue: ICLR 2026 Poster (2026)
Keywords: proxy models
data curation
Primary area: interpretability and explainable AI
TL;DR: We propose using very small learning rates for proxy models to better preserve the relative performance rankings that would be obtained with optimally-tuned large-scale training.

Abstract: Data teams at frontier AI companies routinely train small proxy models to make critical decisions about pretraining data recipes for full-scale training runs. However, the community has a limited understanding of whether and when conclusions drawn from small-scale experiments reliably transfer to full-scale model training. In this work, we uncover a subtle yet critical issue in the standard experimental protocol for data recipe assessment: the use of identical small-scale model training configurations across all data recipes in the name of ``fair" comparison. We show that the experiment conclusions about data quality can flip with even minor adjustments to training hyperparameters, as the optimal training configuration is inherently data-dependent. Moreover, this fixed-configuration protocol diverges from full-scale model development pipelines, where hyperparameter optimization is a standard step. Consequently, we posit that the objective of data recipe assessment should be to identify the recipe that yields the best performance under data-specific tuning. To mitigate the high cost of hyperparameter tuning, we introduce a simple patch to the evaluation protocol: using *reduced* learning rates for proxy model training. We show that this approach yields relative performance that strongly correlates with that of fully tuned large-scale LLM pretraining runs. Theoretically, we prove that for random-feature models, this approach preserves the ordering of datasets according to their optimal achievable loss. Empirically, we validate this approach across 23 data recipes covering four critical dimensions of data curation, demonstrating dramatic improvements in the reliability of small-scale experiments.

Published as a conference paper at ICLR 2026
CAN SMALL TRAINING RUNS RELIABLY GUIDE DATA CURATION? RETHINKING PROXY-MODEL PRACTICE
Jiachen T. Wang Princeton University
Tong Wu Princeton University
Kaifeng Lyu"
Tsinghua University
James Zou Stanford University
Dawn Song
Ruoxi Jia!
Prateck Mittal
UC Berkeley
Virginia Tech
Princeton University
ABSTRACT
Data teams at frontier AI companies routinely train small proxy models make critical decisions about pretraining data recipes for full-scale training runs. How- ever, the community has limited understanding of whether and when conclusions drawn from small-scale experiments reliably transfer to full-scale model training. In this work, we uncover subtle yet critical issue in the standard experimental protocol for data recipe assessment: the use of identical small-scale model train- ing configurations across all data recipes in the name of "fair" comparison. We show that the experiment conclusions about data quality can flip with minor adjustments to training hyperparameters, as the optimal training configuration is in- herently data-dependent. Moreover, this fixed-confiurration protocol diverges from full-scale model development pipelines, where hyperparameter optimization is a standard step. Consequently, we posit that the objective of data recipe assessment should be to identify the recipe that yields the best performance under data-specific tuning. To mitigate the high cost of hyperparameter tuning. we introduce a simple patch to the evaluation protocol: using reduced learning rates for proxy model training. We show that this approach yields relative performance that strongly correlates with that of fully tuned large-scale L.M pretraining runs. Theoretically, we prove that for random-feature models, this approach preserves the ordering datasets according to their optimal achievable loss. Empirically, we validate this approach across 23 data recipes covering four critical dimensions of data curation, demonstrating dramatic improvements in the reliability of small-scale experiments.
INTRODUCTION
High-quality data has emerged as the primary driver of progress in modern AI development Comanici et al. (2025). Constructing the data recipe for training frontier Al models requires series of high- stakes decisions such as how to filter low-quality text, how aggressively to deduplicate, and how to balance diverse data sources. Unfortunately, there is little theoretical guidance human intuition to direct these choices, leaving practitioners to rely on actual model training for data quality assessment.
Proxy-model-based techniques. The most direct approach to selecting data recipe is train full- scale models for each candidate recipe and compare their performance. However, this is prohibitively expensive for large-scale model training, as it would require numerous complete training runs. Researchers and practitioners have widely adopted smaller "proxy models" as surrogates to efficiently estimate each dataset's utility for full-scale training, significantly reducing the computational burden (Coleman et al., 2020). Given the computational efficiency and ease of implementation, small-scale proxy model experiments have guided data decisions for many high-profile models and open-source datasets (Schuhmann et al. 2021 Mazumder et al. 2023 Li et al. 2024 Dubey et al.) 2024).
Rethinking data recipe ablation in practical workflows. Given the critical importance of data quality, modern AI development teams typically employ division of labor: specialized data teams
Work done while at UC Berkeley.
I Equal contribution as senior authors.

---

Published as a conference paper at ICLR 2026
Data preparation phase
Model training phase
Candidate datasets
Small-scale model
Proxy traiinng run evallation on each Dj
Hyperparameter Tuning (eeariing rese, batch size,
Large model
Recommend the
Ui 3x10
D2
Loss
D2
D2 D2
Goal: Small praxy model rankings match the rankings of optimally-tuned large target models (Section 3)
Challenge: Proxy model rankings are highly sensitive to minor variations in hyperparameters like the learning rate (Section 4) approach: proxy models
principled tiny learning rate reliable proxy for optimally-tuned large models (Section 5)
Overview of the industrial practice of data recipe ablation using proxy models. Left: Data teams assess each candidate dataset using small-scale proxy models trained with identical hyperparameters, and recommend the data recipe with the best performance. Right: Model training teams perform extensive hyperparameter tuning to optimize performance on the large target model.
Figure 1:
Train proxy models with stondard (3 x
Train proxy models with LR
AP2-H60T400
Pile-reEurPPart
bC M-ddduup-
MilerrostckkExchrg
RP2-H50T50
Hile-noGiHHib
Rle-nnoEuroard
bX dedup-G
Pile Ary
Webb
bC M-dedup deduppL
30e-0070 50000-00020
P22-H70 707 a
Bile-C77--00 80@
0C70 7070-220A10 S20A10
30
Pile CC70-5304g 530/0
Staa- xchange
D2LM-ckdup--
bOLM decup
Mle-noGitHib
Pile Wiieel
14
DCLM-dedup dedup
Ne Kr
15
DOLM-dedup-
Rie-0C70-S100
22-H100
C
18
RP2-H50
RP2-HDOT10
19
RP2-H66 T40
Fle-CC70--530
M72-H70
50-00300
DCLM-dedup-663 65
22
492-0900:0
DCLM
23
RP2-HI00 TO
Proxy Rarking
Torget Ranking
Proxy (125M) Ronking
Target Ronking (with per-dateset HP tuning)
(with per-dateset turing)
Figure 2: Validation loss rankings of 23 data recipes evaluated on proxy models (GPT2- and target models (Pythia-IB), where target models undergo extensive dataset-specifie hyperparameter tuning. Rankings are determined by the pretrained models' loss on Pile's validation split (Gao) et al. 2020).. Left: When proxy models trained with standard learning (3 10-1), data recipe rankings exhibit severe disagreement between proxy and target scales, with many dramatic reorderings that would lead suboptimal data recipe ablation. Right: When proxy models trained with tiny learning rate (1 x 10 10-5), dataset rankings remain highly consistent across scales.
curate and optimize training data recipe, then recommend high-quality dataset to model training teams who subsequently optimize training configurations (e.g.. hyperparameters) specifically for the dataset (Hoffmann et al.) [202]] Anil et al. 2023] Almazrouei et al. (202)) (Figure T]. However, exist- ing data-centric research evaluates and compares data recipes in ways fundamentally disconnected from this practical workflow. Most experiments in the data-centric literature and benchmarks evaluate all candidate data recipes under fixed training hyperparameters for "fairness", while practical model training uses optimally-tuned hxperparameter configurations specifically adapted to each dataset. We therefore propose refined objective for data-centric Al: find the data recipe that maximizes performance under optimally-tune hyperparameters, reflecting how data is actually used in practice.
Minor variations in proxy training configuration can alter data recipe rankings (Section 4. The refined objective imposes critical requirement for proxy mooy-mooel-Beee badel-based techniques to succeed: the best data recipe identified through small-scale training runs must remain superior after (i) scaling the model to its target size and (ii) the training team optimizes training hyperparameters. However, this goal presents significant challenges given current proxy-model-basee methods, where small proxy models evaluate and rank candidate datasets under fixed, heuristically chosen hyperparameter configuration. Due to the strong interdependence between training hyperparameters and data distribution. each data recipe inherently requires its optimal training configuration (Hutter et al.) 2019 Sivaprasad (et al., 2020). Our experiments (Figure demonstrate that in language model pretraining, even minor variations in training hyperparameters, particularly learning rate, dramatically alter data recipe
2

---

Published as a conference paper at ICLR 2026
rankings and lead to suboptimal recommendations. If dataset ordering collapses under such minor hyperparameter tweaks at the same proxy scale, broader sweeps at larger scales will almost certainly reorder the ranking further.
patch (Section 5 In the long run, the tight interplay between data and training hyperparameters suggests that these two components should be optimize meantime, however, practitioners still need drop-in patch that allows data teams to assess and optimize their data curation pipelines through small-scale experiments alone. We propose a simple yet effective remedy; train the proxy model with a tiny learning rate. Two key empirical observations inspire this approach: (i) within the same model architecture, datasets' performance under a tiny learning rate strongly correlates with their optimal achievable performance after dataset-specific hyperparameter tuning; (ii) data recipe rankings remain stable under tiny learning rates as models scale from small to large sizes. We provide formal proof for these empirical findings for random feature models. Specifically, we prove that, as network width grows, training with a sufficiently small learning rate preserves the ordering of datasets, and the rankings converge to the ordering of their best achievable performance in the infinite- limit. We further characterize this tiny learning rate regime through both theory and practical heuristics.
Training proxy models with tiny learning rates: a theoreticalll-ggouneee
rather than tuned in separate, sequential steps. In the
Experiments. We empirically evaluate the effectiveness of the tiny-learning-ras strategy through comprehensive experiments spanning multiple architectures, scales, and data curation scenarios. We test 23 data recipes covering domain composition, quality filtering, deduplication strategies, and major pretraining corpora across three language model families (GPT2, Pythia, OPT), ranging from 70M IB parameters. Our results demonstrate that proxy models trained with small learning rates achieve significantly improved transferability to hyperparameter-tunee larger target models. Figure 2 illustrates this improvement: the Spearman rank correlation for data recipe rankings between GPT2- and Pythia-1B improved to 0.95 across (23) 253 data recipe pairs when using tiny learning rate (10-5) to train GPT2 instead of commonly used learning rate (3 x 10-1).
2 BACKGROUND: GUIDING DATA CURATION WITH SMALL PROXY MODELS
In this section, we formalize the problem of data recipe ablation for large-scale model training.
Setup & Notations. Consider target model architecture Pgt and a pool of candidate datasets D = [1 D2..... .22....,D., D... where each dataset results from a different data recipe (c.g., different curation algorithms, filter thresholds, or domain mixing ratios). Data recipe ablation aims identify the optimal dataset D.. E 'D that maximizes model performance on validation set D Given a loss function f. let (() == E(;; val) denote the validation loss of model 6. Since model performance depends critically on both training data and hyperparameters (e.g., learning rate, batch size), we write the trained model as 0(D: X) for dataset D and hyperparameter configuration A
Current practice of data recipe ablation with small proxy models. Directly evaluating candidate datasets by training a large target model on each typically computationally prohibitive. common practice to mitigate this issue involves using smaller "proxy models" (IMproxy) < |MMargat) () to predict data quality and determine which data curation recipe to use in large-scale training runs. Small models enable repeated training at substantially reduced computational costs, making them popular for ablation studies of various data curation pipelines (Dubey et al., 2024| et al., 2024). Current practices typically train proxy models on each D, (or its subset) using fixed hyperparameter configuration Ag. then rank the datasets based on small models' performance lval (NaonM DD; Ao)).
3 RETHINKING "HIGH-QUALITY DATASET": A PRACTICAL DEVELOPMENT PERSPECTIVE
Despite the broad adoption of small proxy models for data recipe ablation, the research community has a limited understanding of the conditions under which conclusions from small-scale experiments
Additionally, the training outcome is influenced by stochastic elements such as random initialization and data ordering. We omit the randomness here for clean presentation and treat ((-) as deterministie. We provide additional discussion on the impact of stochasticity on proxy-hased selection in Appendix D.2.1
3

---

Published as a conference paper at ICLR 2026
can be reliably transferred to large-scale production training. Before delving into the question of whether datasets that appear superior for small model training remain optimal for larger models, we must first establish a principled objective for data quality assessment. That is, what constitutes "high-quality datasel" in practical model development? In this section, we discuss a subtle yet critical disconnect between practical workflows and the standard evaluation protocol in data-centric research.
Data recipes need to be assessed under individually optimized training configurations. In the existing literature (e.g., MMgnusson et al. (2025)), the effectiveness of data-centric algorithms is typically assessed by training a large target model on the curated dataset with fixed set of hyperparameters. However, in the actual AI development pipelines. hyperparameter tuning will be performed, and the hyperparameters are tailored to the curated dataset. For instance, GPT-3 (Brown et al. 2020) determines the batch size based on the gradient noise scale (GNS) (McCandlish et al..] 2018). data-specific statistic. Similarly, learning rates (Yang et al.] 2022) and optimization algorithms (Chen al. 2022) are usually adjusted in a data-dependent way. Therefore, we argue that more reasonable goal for the data recipe ablation should optimize the selected dataset's performance under hyperparameters tuned specifically for that dataset. This refined objective acknowledges the strong interaction between data and training hyperparameters in model training (Hutter et al.) 2019), emphasizing that data curation strategies should optimize for the training data's s optimally achievable performance rather than performance under a predefined, potentially suboptimal hyperparameter configuration. Formally, we formalize the data recipe ablation problem as D;- == grgm minxea v (0(D;; X)), where A is predefined feasible space of hyperparameters constrained by factors such the available compute budget.
4 PROXY-MODEL FRAGILITY UNDER HYPERPARAMETER VARIATION
Given our refined objective of identifying the dataset that performs best under its own optimized hy- perparameters, we now examine whether current proxy-model practices align with this goal. Standard practice evaluates each candidate dataset by training proxy model with single, heuristically chosen hyperparameter configuration. Our investigation reveals a concerning consequence: even minor adjustments the learning rate can flip the conclusions drawn from small proxy training runs. Conse- quently, the current practices can fail to identify datasets with the highest potential even for the same proxy model, and even more likely to select suboptimal datasets when scaled larger models with proper hyperparameter tuning. Through sweeps of critical training hyperparameters--nccleeee batch size, weight decay, and token-per-paraun per-parameter ratios-we find that dataset rankings are most sensitive to learning rate. Consequently, we center our main analysis on learning rate and detail the remaining hyperparameter experiments in Appendix D.2.2
Experiments: Learning rate sensitivity can undermine proxy models' reliability. We investigate how minor variations in learning rate values can flip the conclusions drawn from small-scale exper- iments. Figure 3 demonstrates this fragility by comparing DCLM (Li et al. 2024) with one of its variants that underwent more stringent deduplication (detailed configurations in Appendix 1.). We train GPT2-Small (125M) on each dataset using two similar learning rates that reflect typical choices made by practitioners (Karpathy) 2022). The results reveal that dataset rank- ings can be inconsistent with respect to the chosen learning rate. At lower learning rate, DCLM is supe- rior on both the validation loss and the downstream benchmarks. However, a slight increase in learning rate re- verses this performance ranking. This reversal likely stems from DCLM's relatively loose deduplication crite- ria, which tend to favor smaller learn- ing rates, whereas its more stringently deduplicated variant performs better with larger learning rates. The sensi-
Validation
Gmall-5mall.
Benchmaris (dPT2-5mal. 125M)
DCLSA dedup
BC1M
35 07%
CCUU
DCUM-redupp 95
2:85
Accuracy
36 0%%
2.50
2.75
07%
2.70
(a)
3.00-04
3.0-04
(b)
3.0w
Figure 3: Performance comparison of DCLM variants by training GPT2-Small with two similar learning rates in a near-optimal regime. DCLM--eduppGGS is a variant of the DCLM dataset constructed with more stringent deduplication thresholds (see Table [ for details). (a) Validation loss on the Pile dataset. (b) Average accuracy across 5 downstream benchmarks (see Section 6.1]. Star markers indicate the superior recipe in each setting. The shaded regions represent the standard errors calculated across 3 random seeds.
4

---

Published as a conference paper at ICLR 2026
Small learning rate optimelly frtimcllyy-unedd performance
Loss rank correlation between models different sizes
1.
30c.3
34
Rit--070- 50430
DaJM
20c-03
Ale-ccrp C---------SS-
AP2-H400010
088
Optimally rureed Valldarton Loss
3.2
Me-0270-570441
Flc-0C70-53044
RP2 Hedfio
#P2-1mooaa
74e-04
HR2-H00T20
Learning
d
RefinedWib
DCLW SC.W--edii--5
40c-44
Spearman Correlation 0.97
10k0s
46
Abott
475
5.00
5.55
5.50
575
6:00
6.25
650
6.75
200
(a)
Validation Loss (LRx1 10-5)
(b)
Learning
Rote
Figure 4: (a) Correlation between losses of GPT2-Small trained with a small learning rate 1 x 10-6 versus optimally tuned hyperparameters, evaluated on Pile's validation split. Each point represents one dataset, demonstrating that tiny learning rate performance is strongly correlated with the optimal performance for the same model architecture. (b) Heatmap showing Spearman rank correlation between dataset rankings from proxy (GPT2-Smaa (mmall) and target (GPT2-Large) models trained with varying learning rate combinations. High correlation (darker area) in the lower-left indicates that when both models use tiny learning rates, dataset rankings are preserved across model scales.
tivity to learning rate selection high- lights critical limitation of small-scale proxy experiments: conclusions drawn from small-scale training runs with fixed configurations can overfit to those specific settings, potentially leading to suboptimal data curation decisions when scaling to large-scale model training.
High-level intuition: The curse of higher-order effects. To gain intuition about why different learning rates lead to inconsistent data recipe rankings in proxy models, we consider a highly simplified one-step gradient descent setting, where the change in validation loss Lval after one gradient update can be approximated through Taylor expansion:
(0) R (om v2(0) 1 HI
Abval 60
where Hi.. is the Hessian of the validation loss. Consider two datasets D; and D, with corresponding training losses & and Ey. When the learning rate is low, their ranking primarily depends on the first- order gradient alignment terms: (O) (O, versus (vai(0)) (O) However, moderate learning rates, the second-order term becomes significant. For example, two datasets D; and D, that have better first-order gradient alignment can reverse their ordering once the quadratic terms outweigh this difference at moderate values of 1). Overall, the ranking flip can occur as the learning rate moves beyond the tiny regime, where the higher-order curvature term can dominate the change in the loss, override first-order alignment, and ultimately flip the dataset rankings.
5 TRAINING SMALL PROXY MODELS WITH TINY LEARNING RATES IMPROVES TRANSFERABILITY TO TUNED TARGET MODELS
To reduce hyperparameter sensitivity in data recipe ablation and better align with the practical goal of identifying the recipe with the highest performance potential, one straightforward approach is to sweep through extensive hyperparameter configurations for each candidate data recipe and compare their optimal validation loss across all settings. However, this requires numerous training runs per candidate data recipe, which can be prohibitively expensive even with smaller proxy models. Moreover, training randomness can introduce significant variance results, further reducing the reliability of such comparisons. Building insights from one-step gradient update analysis, propose a simple yet effective solution: train proxy models with very small learning rates and rank data recipes by their validation losses under this regime.
5

---

Published as a conference paper at ICLR 2026
5.1 EMPIRICAL FINDINGS & HIGH-LEVEL INTUITION
Notations. We denote the optimal validation loss achievable for a dataset D by Snt-op(((DD) minleA (0(D)) n)) where the minimum is taken over the full hyper-parameter space A. In our experiment, this optimum is approximated via a wide range of hyperparameter sweeps. Our pilot study (see S and Appendix D) shows that dataset rankings are most sensitive to the learning rate, while relatively stable across other major hyperparameters such as batch size. Therefore, throughout this section, we simplify notation and write 0 (D:n) n) to denote model trained with learning rate n) while all other hyperparameters are fixed to standard values from prior literature (details in Appendix
D Given two datasets D; and D,. we denote by ( (0,n) the difference in validation loss when model 0 is trained on these datasets using fixed learning rate 1), and by A (ll opl in their optimal validation losses after full hyperparameter tuning:
(0) the difference
(htl)) val (0, n) = (0(:; n)) - Eval (0() A (ltid)
val-ope (0) opal- - opp opr (0(),)
Our objective is to identify a specific learning rate Yo for proxy models that reliably preserves the ranking of datasets based on their fully-optimized performance on large target models:
sign(
( (1,50) val (0proxy:7m)) = sign( (A ((..))
val (Orgt)
We present two key empirical findings (settings detailed in Appendix D) and the high-level intuition behind them, which motivates our solution of training proxy models with tiny learning rates.
Finding I: For the same model, tiny learning rate performance correlates with optimal hyperparameetr-tunel performance. We first investigate the correlation between the losses for the same model architectures trained by different learning rates. Figure 4 shows the correlation between GPT2-Small's s validation loss on the Pile validation split when trained with tiny learning rate versus its minimum validation loss achieved through extensive hyperparameter sweeps, evaluated across 23 data recipes. The strong correlation demonstrates that for a given model architecture, training with tiny learning rate provides reliable proxy for optimized performance after full hyperparameter tuning. The relative utility of datasets, as measured by tiny learning rate training, is preserved when the same model is optimally configured. Formally, for small 17o- we observe that
sign( A ((65)))) val (Sproxy, 170)) = sign( (d(5)) val. (Oproxy))
for most dataset pairs (D,, Dÿ). Intuition: In one-step gradient update, when the learning rate 1) - 0, model updates are dominated by the first-order gradient alignment term Veva Ve(0). This alignment measures the distributional similarity between training and validation datasets from the neural network's 's perspective, and is a metric widely used in data selection literature (Wang et al., 2024a| Fan al. 2024b a). Therefore, the validation loss at an infinitesimal learning rate effectively soper-bounde the irreducible train- irain-vlidatiin distribution gap, and its ranking across datasets is strongly correlated with the best loss each dataset can achieve after full hyperparameter tuning.
Finding II: Dataset rankings remain consistent across model scales when both use tiny learning rates. Figure [b), we visualize the heatmap of Spearman rank correlation coefficients between proxy and target model dataset rankings across various learning rate combinations. The lower-left region of the heatmap shows that when both proxy and target models are trained with tiny learning rates, the cross-scale transferability of dataset rankings achieves nearly perfect correlation. Formally, this suggests that when both Tjo and 7/1 are small, we have
sign( ((sn) (
val (Oproxy,, mp)) = sign(A (13) (fegt: 71))
for most dataset pairs (D,, D,). Intuition: As we discussed in the one-step gradient update analysis in Section training with tiny learning rates minimizes the higher-order interactions that can confound data recipe comparison when using varying learning rates. While gradient alignment (the first-order term) still neeeedde its relative ranking across different candidate datasets tends to be preserved across model architectures (Sankararaman et al. 2020 Vyas et al./ 2023). When both proxy and target models operate in the tiny-LR regime, their loss rankings are primarily determined by the distributional similarity between training and validation data. As result, the validation- orderings remain relatively stable across different model scales, despite their differing capacities.
Collectively, the empirical observations in ② and 3 together lead to ( when using a small learning rate, motivating our strategy of ranking datasets by their tiny-LR losses on proxy models. Empirical evidence of this high-level intuition can be found in Appendix D.2.4]

---

Published as a conference paper at ICLR 2026
5.2 THEORETICAL ANALYSIS
Section 5.1 uses simplified one-step gradient update analysis to provide high-level intuition for the two empirical findings. To further support our findings, formally prove that training random feature models aligns with the two empirical findings. We choose the random feature model because it is one of the simplest models whose training dynamics at different learning rates and scales can be approximately characterized mathematically. As the training dynamics of deep neural networks are usually non-tractable, series of recent works have used random feature models as proxies to understand the dynamics of neural networks under model scaling (Bordelon et al.) [024] Lin et al.)
2024, Medvedev et al. 2025). The random feature model is also closely related to the Neural Tangent
Kernel (NTK), key concept in the theory of deep neural networks that has often yielded valuable
insights for developing data-centric techniques (Arora et al. 2019 Wang et al.) 2024b).
Theorem 1 (Informal). Given two candidate data distributions DA and DB, if the width of the random feature model is larger than a threshold, then after training on both datasets with learning rates n small enough, with high probability, the relative ordering of (va (D))))))))))))))))))))))))))(() and (0 DB: n)) is the same as the relative ordering of the two validation loss values at the infinite- wiith limit.
See Appendix B for complete theorem statements and proofs.
Interpretation. This theorem establishes that tiny-learningrall training preserves dataset rankings relative to the infinite-width optimum, which serves as theoretical anchor for understanding cross- scale transfcrability. The infinite-width model provides a theoretical reference point for understanding large target models after extensive hyperparameter tuning. In the infinite-wide limit, training random feature models is equivalent to solving a kernel regression problem, and the obtained solution captures the optimal validation loss achievable within the function class. Our theory demonstrates that both optimally-tune dataset rankings dictated by this fffinitewwitth optimum as model width increases. Consequently, dataset rankings from small-scale training with low learning rates are consistent with those from optimally-tune large-scale training with high probability.
large target models and tiny-learning-Tee
proxy models converge to the same
Intuition & Proof sketch. Similar to our intuition from one-step gradient update, our analysis of random feature models decomposes the change in validation loss under SGD into a deterministic drift term that captures the true quality gap between datasets and a variance term from stochastic updates. When a sufficiently wide model is trained with a small learning rate, the drift dominates the variance, so the observed sign of the loss difference matches the sign of the infinite-width optimum gap with high probability. This formalizes the intuition that small learning rates suppress the curse of higher-order effects that otherwise scramble rankings.
5.3 AN EMPIRICAL CHARACTERIZATION OF "TINY" LEARNING RATES
Our analysis suggests the potential benefits of training proxy models with tiny learning rates. A natural question is how small should the learning rate be for reliable proxy training runs? We provide both theoretical guidance and practical recommendations, with extended discussion in Appendix C
Theoretical guidance. Our theoretical analysis (detailed in Appendix C establishes an upper bound for what constitutes a ''tiy"" learning rate in the context of one-step mini-batch SGD. At a high level, we show that Thiny needs to be far smaller than 1/Amax, where Amax is the top eigenvalue of the validation loss Hessian. We stress that one-step analysis has often been shown to give quantitatively useful guidance when coupled with empirical validation (e.g. McCandlish et al. (0188) Smith et al. (2018). Following the established precedent, we treat our derived bound as principled guidance rather than as a rigid constraint. In Appendix C we use model checkpoints that have undergone short warmup training to estimate Miny 's upper bound, and we show that these theoretically- prredicee values regime of learning rates with high transferability.
consistently fall within the impprricaly-identife
Choosing Thiy in practice: simple rule of thumb. While estimating Many from proxy model checkpoints is computationally inexpensive, we find that a simpler heuristic suffices for most LLM pretraining scenarios. In Appendix C. we discuss the relationship between Thiny and the standard learning rate for language model pretraining. Empirically, we find that using learning rates 1-2 orders of magnitude smaller than standard choices of learming rates generally works well. For language model pretraining, this typically translates to learning rates of approximately 10-5 to 10 6 In Section 6.2 we will see that these values work consistently well across various model scales.
7

---

Published as a conference paper at ICLR 2026
Practical implementation also requires consideration of the lower bounds of Thaay due to numerical precision. However, we find that the recommended range of 10-3 to 10-6 lies comfortably above the threshold where numerical errors become significant in standard training. Our experiments confirm that learning rates in this range maintain numerical stability while achieving the desired transferability properties. We provide additional analysis of numerical considerations in Appendix C
6 EXPERIMENTS
6.1 EXPERIMENT SETTINGS
Evaluation protocol. We conduct comprehensive evaluations across multiple model architectures (GPT2, Pythia, OPT) spanning sizes from 70M to IB parameters. We assess transferability us- ing both the validation loss across diverse domains from The Pile corpus (Gao et al.] 2020) and downstream benchmark performance HellaSwag, Winogrande, OpenBookQA, ARC-Easy, and CommonsenseQA. Following standard practices, we primarily set the caan--ee--prrmmete ratio to 20, adhering to Chinchilla's 's compute-optimal ratio (Hoffmann al. (022). Additional ablation studies examining overtraining scenarios (with ratios up to 160) are presented in Appendix D.2.2 All experiments use single-epoch training without sample repetition, following standard LLM pretraining conventions, Crucially, target models undergo dataset-specific hyperparameter optimization to align with our proposed methodology, with tuning budget held constant across all experiments.
Data Recipes. To comprehensively evaluate proxy model transferability across the diverse data curation decisions faced in production scenarios, we construct 23 data recipes spanning four critical dimensions of language data curation. (1) Domain compasition and ablation. We create 10 domain mixture variations from The Pile (Gao et al. 2020 Specifically, 4 recipes maintain a fixed 70% Pile- CC allocation while varying the remaining 30% between StackExchange and ArXiv. Additionally, 6 recipes are created by excluding one domain from the full Pile dataset to assess domain importance. (2) Existing dataset comparison. We evaluate 3 widely-adopted pretraining corpora: C4 (Raffel ct al., 2020), DCLM-baseline (Li et al. 2024). and RefinedWeb (Penedo et al., 2023). They all originate from Common Crawl but vary in curation strategies such as data filtering criteria and deduplication algorithms. (3) Scoring- based data filter. We construct 6 data recipes using different mixing ratios of head-middle versus tail partitions from RedPajama- V2 Weber et al. (2024), ranging from 50:50 to 100:0. These partitions are defined by the perplexity scores from a 5-gram Kneser-Ney model trained on Wikipedia (Wenzek al.) 2020). (4) Deduplication. We create variants of DCLM- MLLM-baseline by modifying the stringency of Massive Web Repetition Filters (Rae et al. 2021). adjusting n-gram and line/paragraph repetition thresholds. The detailed description for the data recipes considered in this work is summarized in Appendix D.1.4 and Table
Evaluation Metries. We report two complementary quantities that capture (i) rank fidelity and (ii) practical decision risk. Spearman rank correlation. This measures the monotone agreement between the proxy-induced ranking and the scrget-scale ranking, where the target ranking obtained by training the target model on each dataset with its own tuned hyperparameters. Top-1: Decision Regret. This addresses the practical question: if data team selects the top-k data recipes ranked by the proxy model for full-scale evaluation, how suboptimal might their final choice be compared with the best data recipe? Formally, we compute the performance difference between the truly optimal dataset and the pess-performing dataset among the proxy's 's top-k selections, with both datasets evaluated using their individually optimized hyperparameters on the target model.
Due to space constraints, leave detailed model training and other settings to Appendix D.I
Remark. A recently released benchmark suite, DataDecide Magnusson et al. 2025), provides up to IB pre-trained models for efficient evaluation of proxy model transferability. However, they use a fixed training configuration across all datasets, which may cause the conclusions overfit to the specific hyperparameter configurations. as we discussed earlier. Therefore, we conduct our own evaluation in which all target models are hyperparameter--un sumee for each data recipe.
Remark (Computational challenges in evaluating proxy model transferabilityy. We consider large target models of size up to 1B parameters, which matches the largest target model size in the established DataDecide benchmark suite (Magnusson et al. 2025) developed by for evaluating proxy model transferabiliity Our experimental protocol requires training target models with extensive dataset- specsse- hyperparameter optimization, which further amplifies computational requirements in
8

---

Published as a conference paper at ICLR 2026
Standard LR Prowy (6PT2-Small 10 +)
Thy Proxy (SPT2-SnalL 10 +)
10-
Speammn Carvelation
06
02-
0..
Gitfuls Publled Centrol
ArXiy Baratori NDH ExParter
Publlte Abstracts Dat Hethematics Starkf echange
USPTO Buokgrounds sundg
HellsSweg
HackerNed
Gurenbarg (PG-19)
PhilPppers Earen Emails
Fread.n
ARC. Eusy
Wilepedla (on)
Commordenaah
OpenBeoiCQA
Winegrande
Pile-GC
Average rank correlation between proxy (GPT2-Small and target (GPT2-Large) for the loss computed over a variety of validation domains (from Pile) and downstream benchmarks. Compared to the standard learning rate used in the literature (Karpathy 2022). tiny learning rate consistently improves the proxy model's 's rank transferability, often by a significant margin.
Figure 5:
Corneaatinn Serhaae LR and presed laree Sagge Large, 77400
between cptindl denseer and it presy select ana fueed sarget radek (Fee vi kee)
Gap betheoon apritmal dehareh top 4 adecctons on funed tange? modele (wug soroes bexteoord) Srendind pracy 10112 Small o200g)
Corrdatre
0.25
(P112 (00000)
0.20
Accuracy Exp
66
07
C10
000
Nob (790
(221)
000
00
(0)
Learning Rats: (pnany)
(b)
(c)
Tap-k
Figure 6: Dataset ranking consistency between proxy and target models across different learning rates, evaluated on validation loss and downstream benchmarks. (a) Spearman rank correlation between dataset rankings from proxy models (GPT2-Small Pythia-70M, OPT-)25M) and larger target model (GPT2-Large) as function of proxy model learning rate, evaluated on aggregated Pile validation loss. (b) Top-k gap between optimal dataset and proxy selections on hyperparameter- optimized target models, measured using Pile validation loss. (c) Top-k gap measured using the averaged accuracies across the 5 benchmarks zero indicate that proxy models reliably identify datasets that perform optimally on larger models. Shaded areas show 95% bootstrap CIs computed by resampling seeds (3 runs per dataset). Additional results are available in Appendix D.2.1
considered (see Section 6.1 Gaps approaching
evaluation. However, this is essential for properly evaluating each dataset's 's optimal performance potential as advocated in our refined objective. Throughout this project, we completed > 20, 000 model training runs (including both small proxy models and large target models). We leave the exploration of efficient evaluation protocols for proxy-to proxy-to-tarye to-taee transferabilityy as a future direction.
6.2 RESULTS
As shown in Figure (a), training proxy models with standard learning rates (3 x 10-4 for GPT2-
Small Karpathy 2022) and OPT-125M (Zhang al. 2022). 10 (0-3 for Pythia-70M (Biderman et al. 2023 yields only modest ranking agreement, with Spearman rank correlation < 0.75. This poor rank consistency can lead suboptimal data selection decisions and performance degradation in large-scale model training. However, when the learning rate drops below 1 x 10-4, we observe substantial improvements in ranking agreement. Under these smaller learning rates, the proxy models achieve performance ranking correlations above 0.92 across all three architectures. Figure S extends these results beyond aggregated validation loss to examine how small learning rates affect the reliability of proxy models across individual Pile domains and downstream benchmarks. The results demonstrate that training proxy models with reduced learning rates improves data recipe ranking correlations across nearly all evaluation metries. In contrast, standard learning rate proxies exhibit catastrophic failure across most metrics, yielding near-zero or negative correlations in many domains.
Figure 6 (b) and (c) demonstrate the practical implications of these ranking correlations through the top-k decision regret metric. When data teams select the top-k data recipes for large-scale model training based small-scale experiments, proxies trained with tiny learning rates consistently

---

Published as a conference paper at ICLR 2026
minimize the performance gap relative to the ground -ouund-truthh optimal data recipes for the target model. In contrast, standard learning rate proxies can incur > 0.25 validation loss degradation when is small. This stark difference demonstrates that conducting small-scale ablation studies with smaller learning rates can significantly enhance the reliability of data decisions in large-scale model training.
7 CONCLUSION & FUTURE WORKS
We demonstrate that standard data recipe ablations are brittle in small-scale settings; even slight adjustments the learning rate can yield inconsistent findings. To address this, we proposed simple yet effective solution: using tiny learning rates during proxy model training. In the following, we discuss the scope of this work and outline several potential future directions.
Extending proxy model techniques to multi-epoch and curriculum learning. This work focuses on single-epoch LLM pretraining. Designing effective proxy training runs for multi-epoch scenarios presents significant challenges. A key question is how to subsample the full dataset while properly accounting for the repetition patterns that emerge across epochs. Similar technical difficulties arise in curriculum learning. Given the growing data scarcity, developing principled small-scale proxy training protocols for multi-epoch and curriculum-bart learning is important future direction.
Joint optimization of data and training configurations. As we mentioned in the Introduction, our approach serves as a simple remedy to the proxy-model-assed approach. However, because challenge stems from the strong coupling between training configurations and datasets, we argue that, in the long term, datasets and tunable training configurations should be jointly optimized. Recent advances in gradient-basce hyperparameter optimization (Lorraine et al. (2020) and algorithm unrolling (Chen et al. 2022) may provide inspiration and building blocks for this direction.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is supported in part by the Apple PhD Fellowship. Ruoxi Jia and the ReDS lab acknowledge support through grants from the National Science Foundation under grants I1S-2312794, IIS-2313130, and OAC-2239622.
We thank Lin Chen, Mohammadhosseen Bateni, Jennifer Brennan, Clayton Sanford, and Vahab Mirrokni at Google Research for their helpful feedback on the preliminary version of this work.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Howett. Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.089003 2024.
Ebtesam Almazrouei. Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Mérouane Debbah, Êtienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint 0rXiv:2311.16800 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint arXiv:2305. 2023.
David Anugraha, Genta Indra Winata, Chenyue Li, Patrick Amadeus Irawan, and En-Shiun Annie Lee. Proxylm: Predicting language model performance on multilingual tasks via proxy models. arXiv preprint arXiv:2406.09:3 2024.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Fric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning, pp. 4345-4382. PMLR, 2024.
David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 1311 2020.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36:36000-36040, 2023.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark. Journal of Machine Learning Research, 23(189);1-59, 2022.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations, 2020.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv22507.06261, 2025.
11

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXhv:240..21783, 2024.
Simin Fan, David Grangier, and Pierre Ablin. Dynamic gradient alignment for online data mixing. arXiv preprini arXiv:2410.0244:0 2024a.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization
estimation. In Forty-first International Conference on Machine Learning 2024b.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00000 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL https: http:///eennoddooreeeeeeeeeeeeee /ttp///corrd//2600000000 12608602
Xinran Gu, Kaifeng Lyu, Jiazheng Li, and Jingzhao Zhang. Data mixing can induce phase transitions in knowledge acquisition. In / Can't i Believe It's Not Better: Challenges in Applied Deep Learning, 2025.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimil large language models. arXiv preprint arXiv:
2022.
Junyuan Hong, Jiachen T Wang, Chenhui Zhang, LI Zhangheng, Bo Li, and Zhangyang Wang. prompt engineer. In The Twelfth International Conference on Learning Representations, 2024.
Dp-opt: Make large language model your
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint urXiv:2404.06395 2024.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
Stanislaw jastrzkebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arX1v:1711.04623, 2017.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Automatic prediction of compute-optimal data composition for training llss. arXiv preprint arXiv:2407.20177, 2024.
Andrej Karpathy. nanogpt:
The simplest, fastest repository for training/finetuning medium-sizze sized gpts. 2022.
Alaa Khaddaj. Logan Engstrom, and Aleksander Madry. Small-to-large generalization: Training data influences models consistently across scale. In The Thirteenth International Conference on Learning Representations, 2025.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus- Robert Müller. Efficient backprop. Neural networks; Tricks of the trade, pp. 9-50. Springer, 2002.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pp. 148-156. Elsevier, 1994.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran. Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuni tuning. arXiv preprint arXiv:2002.11770 2020.
12

---

Published as a conference paper at ICLR 2026
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing 142822 2024.
Systems,
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. Advances in Neural Information Processing Systems, 37:60556-6006 2024.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. In First Conference on Language Modeling, 2024a.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang. and Min Lin. Regmix: Data mixture regression for language model pre-training, arXiv preprint wxXiv22477.01499 2024b.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference artificial intelligence and statistics, pp. 1550-1552. PMLR, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0110,, 2017.
lan Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, et al. Datadecide: How to predict best pretraining data with small experiments. arXiv preprini arXiv:2504. 11393, 2025.
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, et al. Dataperf: Benchmarks for data-centric ai development. Advances in Neural Information Processing Systems, 36:5320- 5347, 2023.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint 17X/ 2018.
Marko Medvedev, Kaifeng I Lyu, Dingli Yu, Sanjeev Arora, Zhiyuan Li, and Nathan Srebro. Weak-to- strong generalization even in random feature networks, provably. arXiv preprint arXiv:2503.02877, 2025.
Dheeraj Mckala, Alex Nguyen, and Jingbo Shang. Smaller language models are capable of selecting instruction-tuns -unniig training data for larger language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 10456-10470, 2024.
Shiwen Ni, Haihong Wu, Di Yang, Qiang Qu, Hamid Hinejad-Rokny and Min Yang. Small language model as data prospector for large language model. arXiv preprint 00X9:22412.09990, 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2336.01 2023.
Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability, pp. 1679-1706, 1994.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 212 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-66 167,, 2020.
13

---

Published as a conference paper at ICLR 2026
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh, Ayan Chakrabarti, Wittawat Jitkrittum, Vladimir Feinberg, Seungyeon Kim, Hrayr Harutyunyan, Nikunj Saunshi, Zachary Nado, et al. A little help goes a long way: Efficient Ilm training by leveraging small lms. arXiv preprint arXiv:2410. 187799 2024.
Karthik Abinav Sankararaman, Soham De, Zheng Xu, W Ronny Huang, and Tom Goldstein. The impact of neural network overparameteriiztioon gradient confusion and stochastic gradient descent. In International conference on machine learning, pp. 8469-8479. PMLR, 2020.
Schuhmann, R Vencu, R Beaumont, R Kaczmarczyk, C Mullis, A Katta, T Coombes, Jitsev, and A Laion Komatsuzaki. 400m: Open dataset of clip-filtered 400 million image-text pairs. arxiv. arXiv preprint arXiv:2111.021 2021.
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. Optimizer benchmarking needs account for hyperparameter tuning. In International conference machine learning, pp. 9046-9045. PMLR, 2020.
Samuel L Smith, Picter-Jan Kindermans, Chris Ying, and Quoe Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
Mandt Stephan, Matthew D Hoffman, David M Blei, al. Stochastic gradient descent approximate bayesian inference. Journal of Machine Learning Research, 18(134):1-3 2017.
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks are consistent across widths at realistic scales, Advances in Neural Information Processing Systems, 1036--00000, 2023.
Jiachen Tianhao Wang, Tong Wu, Dawn Song, Prateck Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. Advances in Neural Information Processing Systems, 37:131197-13122 2024a.
Jingtan Wang. Xiaoqiang Lin, Rui Qiao, Chuan -henn Foo, and Bryan Kian Hsiang Low. Helpful or harmful data? fine-tuning free shapley attribution for explaining language model predictions. In Forty-first International Conference on Machine Learning, 2024b.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: open dataset for training large language models. Advances in neural information processing systems, 37: 116462-116492. 2024.
Ziting Wen, Oscar Pizarro, and Stefan B Williams. Feature alignment: Rethinking efficient active learning via proxy in the context of pre-trained models. Transactions on Machine Learning Research, 2024.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, 2020.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798 0086668111 19818, 2023.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint ar X2x::220330044:0 2022.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403. 16952,, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language models. arXiv preprint arXiv:2205. 20806,,20 2022.
14

---

Published as a conference paper at ICLR 2026
A EXTENDED BACKGROUND & RELATED WORKS
A.1 TAXONOMY OF
METHODOLOGY
In the literature, proxy- mooy--ooeel-asse approaches for data quality optimization typically fall into three main categories, each of which uses different strategies to reduce computational costs.
(I) Model-scale proxies maintain the full training schedule but use models with fewer parameters (e.g., reduced layer count or hidden dimensions) for the proxy training run. This approach is relatively common in early literature where the dataset sizes are relatively small (Coleman et al.. 2020). We
found one example in large language model pretraining (Xie et al., (023).
(2) Data-scale proxies preserve the target model architecture while drastically reducing the number of training iterations or tokens processed (Kang et al.) (024). This strategy is commonly used for optimizing training data in continual pretraining scenarios and is particularly prevalent in developing compact models such as Phi-4 (Abdin et al.) 2024). The approach operates under the fundamental assumption that early training dynamics provide sufficient predictive signals to forecast long-term convergence properties and final model performance accurately.
(3) Ratio-prest matio-preserving proxies simultaneously reduce both model and data scale, generally maintaining consistent scaling relationships between them (c.g.. (Li et al. 2024; Magnusson et al. 2025 ). In language model training, practitioners typically preserve the parameter-to-toker ratio prescribed by Chinchilla scaling laws (Hoffmann al.) 2022). This balanced approach during downscaling ensures that proxy models operate in appropriate regimes-neither underfitting nor overfitting relative to their capacity-which better mimies the learning dynamics of full-scale target models.
Scope of this work: Our work primarily focuses on (3) Ratio-preserving proxies, representing the most widely adopted approach in large-scale language model development. A systematic study examining the other two proxy training methodologies and their comparative effectiveness remains an important direction for future research.
Remark (Architecture and scale considerations.). We emphasize that this work focuses specifically on optimizing training hyperparameters for proxy models, particularly the learning rate, rather than addressing questions about proxy model architecture scale selection. While our experiments demonstrate the effectiveness of tiny learning rates across three different model families (GPT-2, Pythia, and OPT) with varying model sizes, we do not delve into how architectural choices or the proxy-to-taree size ratio affect the reliability of dataset selection. The question of how large a proxy model should be relative the target model, and whether certain architectural families provide better transferability than others, is an important future work. Our theoretical analysis in Section 5.2 for random feature models provides initial insights into width requirements, but extending this understanding to modern transformer architectures and establishing principled guidelines for proxy model sizing remains an open challenge.
A.2 EXTENDED RELATED WORKS
Proxy models in data-centric machine learning. Proxy models have emerged as standard efficiency solution across the literature of data-centric machine learning algorithms that require model retraining. This approach originated from early works demonstrating that feature representations from small, computationally efficient proxy models could effectively substitute for representations from larger, more accurate target models in the task of coreset selection and active learning (Lewis & Catlett 1994 Coleman et al., 2020). The research community has leveraged this technique to accelerate various dataset optimization algorithms, including data mixture optimization (c.g.. (Xie
al. 2023, Chen et al. (023)), ) data point filtering (e.g.. (Mekala et al. [2244] Ni et al.) 2024 active learning (e.g.. (Wen et al. 20244) ) and prompt engineering (e-g., (Liu et al., 2024a| Hong et all (024)).
Recent works have even extended this paradigm to knowledge distillation (Rawat et al. 2024 where
small proxy models guide the early training stages of larger models.
Data-dependent scaling laws. A parallel research direction explores using proxy model performance
to directly predict target model performance by fitting scaling laws. For instance, (Anugraha et al.
2024) fits regression models
proxy performance metrics to forecast target model outcomes. Brandfonbrener et al. (2024) shows that test losses between two models trained on separate datasets can follow predictable scaling patterns, though their work specifically examines paired model and
15

---

Published as a conference paper at ICLR 2026
dataset size configurations. The line of works that most relevant to this paper (Luu et al. 2024b Ye et al. 2024) use small proxy models to fit data mixture-dependerr scaling laws that predict target model performance across different mixture configurations. They then use these predictions to optimize the composition of training data mixtures. Another work by (Kang et al. 2024) explores an alternative proxy approach, where models with identical architecture are trained for fewer iterations, and the resulting performance curves are extrapolated through scaling laws to predict outcomes at extended training durations. However, these approaches share a limitation: all of the proxy training runs use identical, fixed hyperparameters across all data mixtures. As discussed in our paper, this approach can inadvertently favor datasets that align well with the chosen training configuration, potentially distorting the resulting data mixture recommendations. Revisiting these data mixture optimization methods with hyperparameter-tunel proxy models is an interesting and promising way to extend our findings.
Understanding proxy-to-target transferability. Despite the widespread use of proxy models, relatively few studies have systematically investigated the reliability and conditions under which proxy model conclusions transfer effectively to large target models. (Khaddaj et al. 2025) empirically showed that proxy model losses generally correlate strongly with those of large-scale models across various training datasets, though the relationship depends somewhat on specific test domains or downstream tasks. Their work demonstrates that data attribution and selection tasks can be reliably performed using smaller proxy models. More recently, (Magnusson et al. (025) released DataDecide, a comprehensive suite containing over 30,000 checkpoints spanning 25 corpora. Their analysis shows that scaling law-based approaches do not consistently outperform single-scale proxy models for performance prediction. Theoretical analysis in this area remains notably sparse, with (Gu al.) 2025) providing a rare exception by proving that the optimal ratio between factual and web- web-scraped data can shift significantly when scaling from small proxy models to larger target models. However, these studies neither identified the specific conditions that lead to high transferability nor examined how training hyperparameters affect proxy model reliability. Our work significantly extends this line by demonstrating that proxy training runs with tiny learning rates lead to strong transferability across model scales, offering a computationally efficient yet more reliable enhancement to existing proxy-model pipelines.
Connections to hyperparameter transferability, The proxy- moox-moodel-based dataset selection problem bears striking similarities to well-documented challenges in hyperparameter optimization, where configurations optimized for small-scale models often transfer poorly to larger architectures (Li et al. 2020: Yang et al., 2022). While solutions like j-parameterizztion have been proposed address hyperparameter transferability across model scales, their application to dataset transferability remains unexplored. p-paraaeterrization might theoretically improve data transferability when small and large models train for identical iteration counts, but proxy-based techniques typically train on substantially reduced time. An interesting future direction is explore whether analogous principles can be developed specifically for training data's 's transferability. On the other hand, our work demonstrates that simply training proxy models with tiny learning rates, without modifying any other training configurations, significantly improves cross-scale data transferability.
16

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSIS
In this section, we analyze two-layer random feature networks and prove that the transferability to datasets' minimum achievable performance holds when the proxy models are trained with tiny learning rates.
B.1 SETUP
Random Feature Networks. We analyze two-layer random feature networks, also known as random feature models. Let a R - R be a Lipschitz function. The weights in the first layer U = (u1...., um) Rdxm are drawn independently from a bounded distribution on I. For all in compact input space X C RRd define the feature map
(x)) (uu )));
(x) ( (1)(()))
(a)
R
The second layer is parameterized by a weight vector 0 E m Given the first-layer weights and the second-layer weights 0, the two-layer network outputs
(1)
(U)) (m)).
This function computes a linear combination of randomly generated nonlinear features of the input æ.
Data Distribution. A data distribution D is probability distribution over X' x R, where the marginal distribution on inputs æ, denoted as Dx, is assumed to admít continuous density function over X. Each sample (x, y) D consists of an input and label y. We say that D has label noise if there exists target function fD X + IR such that y = (z) almost surely for (z.y) D.
Training. We define Pm, (n; B, I") as the distribution of (0, U) of a width-m network trained data distribution with the following procedure. Define the MSE loss as:
e(f; æ; 4) - ([(x))))) -
(U L)( = B(ay)>D (e.5 2z
(6)
We randomly sample the first-layer weights U LTTH described above and freeze them. We then optimize the second-layer weights 0 using one-pass mini-batch SGD with batch size B and learning rate 17, starting from 0g 0. The update rule is given by:
2 00
(((5 (...))))))))))
0.+1 0.
B
b=1
0=0:
where (xr,s, Vt,8) is the 6-th training input and label in the t-th batch, sampled independently from D. (p(m) (D; B, T) is the distribution of (0r, U) obtained by running the above procedure for T steps.
Validation. We fix data distribution Dral validation, called the validation distribution. The validation loss is given by
Lval(f) E(m.y)-D.n xf)
(8)
Given a training distribution D with no label noise, we define the expected validation loss a width-m network trained on D with batch size and learning rate v for 7'' steps
(mm) (D:n, B,T) = (D.
[Cva)(
(9)
Further, we define the best achievable validation loss of a width-m network trained on D with batch size B for I steps (after tuning the learning rate 17) as
vall-opt (D:BIT) min ((m) (m) n>0
I(m)
Di B,
(10)
(m) Nopt
B,
argmin
I (m) vaal
(D;
(11)
タ>0
Understanding the Infinite- Limit. Before stating our assumptions on the data distributions, we first provide some intuition on the infinite-width limit of the random feature model. random fea- ture model with width naturally induces kernel function ((U)( x²) (o(1)))) (()) (x)))
17

---

Published as a conference paper at ICLR 2026
on the input space X. As - oo, the randommess in the features (x)) averages out due to the law of large numbers and the kernel ((U)(,, (x, x') converges to deterministic kemnel K(x, x'):
(xx x²) = Eu-u (u, x) (u,,
(12)
In this limit, the kernel captures the expressive power of the aafinit-wwidth model: learning in this regime becomes equivalent to kernel regression using K. The function class defined by the model becomes a Reproducing Kernel Hilbert Space (RKHS) associated with K, denoted as Hx. The properties of this the target function-directte influence the learnability of the problem. With this in mind, we now formalize the assumptions we make about the data distributions in our theoretical analysis.
whether it is positive definite and how well it aligns with
Behaved Distributions. We say that a distribution D with no label noise is well-behaved if the following conditions are satisfied:
1. Full Support on the Input Space. The probability density of Dx is strictly positive on the input space X. This ensures that D& has full support on X.
2. Positive-Definite Kernel. The kernel K K(x, x') has strictly positive minimum eigenvalue on L (Dx), the square-interrable function space under Dx. This prevents the kernel function from being degenerate.
3. Realizability at Infinite Width. The target function I' lies in the RKHS Hx associ- ated with the kernel K. More specifically, there exists v supp(L)) - R such that (1) < 3o and it holds for all æ X' that
(x) Euner ((u,, x))((u)
(13)
This condition ensures that fū can be learned by kernel regression with kernel K and the random feature model with infinite width.
4. Compatible with the Validation Distribution. The distribution 1) satisfies the following:
lim lim ((. (O; U)] -0 T-000
(14)
where Pt, (D; = (D; Topt (D; B, T)) is the distribution of (0, U)) obtained by running SGD when the learning rate is optimally tuned for minimizing the validation loss. Intuitively, this condition means that asymptotically, the best way minimize the validation loss is to fit S exactly. In other words, although there is a distribution shift between /) and Dval. when the network is sufficiently wide and the training is sufficiently long, tuning the learning rate 77 for minimizing the validation loss is essentially equivalent to tuning it for fitting f better on D. This is consistent with the typical phenomenon of validation loss, which usually decreases as training progresses. It is easy to see that this condition implies the following:
lim lim I(m) m m=000 T-000 val- (D: (fb))
(15)
where call (fz( the best achievable validation loss of the training distribution D.
Main Result. Now we are ready to state our main result. Suppose we have two training distributions DA and DB. We want to know training on which distribution leads smaller validation loss. Mathematically, we want to know which one has a smaller (m) Zut B, T). The following theorem
shows that comparing Z(m) (-;n, B,T) with a sufficiently small learning rate reliably preserves this ordering.
Theorem 2 (Main Theorem). Let DA and DB be two well-behaved training distributions with no label noise. Assume that the best achievable validation loss of DA is different that of DB, i.e.,
DAB = Cvallfb)) Lval (fbb) ) # O.
(16)
Then, there exist constants mo To, Co, Mmax > O such that for all m 2 ma, T> To B 1, if lies in the range 4 S n S min [a Mmax)), then
sign (I(m) (DA; B,T) (m) (Dn; B.T)) (Dn:n,B,T))
sign(AAn)
(17)
18

---

Published as a conference paper at ICLR 2026
Interpretation. The theorem establishes that the sign of the difference in the expected validation loss of two datasets, sign val (DA: A: 7); B, T)
val (DB: -DB: n,, B, T)), when trained with a small, fixed learning rate 1). is identical to the sign of the difference in their best achievable validation losses in the infinite-width limit, sign(A, (A AB). For random feature models, this infinite-width limit, ())( represents the theoretical optimum that model achieve after full hyperparameter tuning and sufficient scaling. Theorem observed in the tiny-leamnigg rate regime accurately predicts the ranking according to the datasets' true potential. Importantly, as model size increases, the dataset rankings for optimally-tuned larger target models also converge this infinite-width limit, thereby providing the necessary theoretical link to justify using small models with tiny learning rates for reliable data selection.
2 shows that for a given set of candidate data recipes, the ranking
B.2 PROOFS
To prove the main theorem, it suffices to show the following lemma.
Lemma 3. Let D be a training distribution that has no label noise and is compatible with Dval- Given > O, there exist constants mo, To, CD. a, Anax > O such that for all m 2 mg, 2 To.
B 2 1, i lies in the range T I S min {aBm, Tmmxx) ) then
I (Din, B,T) (fbi S..
(18)
Proof of Theorem Applying Lemma
3 to D DA and D = DB with A 3.
have constants m/ To co: a, A
0 and m T.P aB a. 1/ > such that for all
minfcō "!] T
m 2 max {mg, m }. T 2 max (TØ TP). J. 2 if 1) lies in the range
S S
min {a Bm, (oBBm, a Bm, Tmaxx Vmax then
(m)) val (Dain, B,T) - Lvall (Sba))
AAB
( (DB; B,T)- Lval(fDa)
A AB
S
(19)
3
Therefore, sign (I(M)) (DAiT) - (mm)
(DB:n. B,T)) sign(AAB) as desired.
In the rest of this section, we prove Lemma 3 for fixed training distribution D. In the following, we define the following quantities depending on the first-layer weights U:
H. = E (aWN)D (2)) (2)
(x) |u]
(20)
B = E E
uo (z) ul.
(x))
(21)
Vm
μ = H B E argmin (D.U,)
(22)
TER
Σ = - W Blesy (1-))())))
(10) (1))
U
(23)
B.2.1 LOSS DECOMPOSITION
First, we present the following lemma that decomposes the validation loss of any random feature model into the best achievable validation loss plus some error terms.
Lemma 4, For all U E dmxm and 0 E R.
Ve> 0: Cumi (10 S (3
(24)
Ve E 1):
(ral(111))) ) 2 (1 c) - (--1-)))
-
(25)
where 3 supmex Dx(x)
(w)
19

---

Published as a conference paper at ICLR 2026
Proof. For the first inequality, since (a + b)² = (1 c)a + (1 + }) for all > 0, have
1 2 ((a))) Deal (10 (x) y)²
(vailSo
1 - Llay)>Daa (V f3 (x)))²
1+1 E Bla
(z)
- + Caal (fb) + (1 ():
where the last inequality follows from the fact that D(a (z) fb
=
l - (ad)) S (x - D
For the second inequality, since (a bb² 2 (1 - (1=c)a² - -(2-1)b² and >>> for all c E (0, 1), we have
1 2 (a. (x) -)²
1-1
2 1- E (ma)) Dina [VV fū (z))²
_/- E (aa (z)
2 (1 Gral (ft) 3. (- 1)LD(0,U).
which completes the proof.
Taking the expectation over U and the training process, we have the following corollary.
Corollary 5. For all m 2 1, > 0, B 2 and T 2 1,
V (0,
(m)) [D: B,T) - L (ra) S cLval(F)) + B + 1 (m) (n, B,T),
where _ (n, B,T)= = (( (D:,,, (CD(0:U))
Proof. Taking the expectation over (0, p(m) (D; B,T), we have
I'm) (D:n, B,T) < (1 Cvai (fb) 3 (0 L) (n, B, B,T), (D:n, B, 2 (1 - coa (fb) - 3 - 1)L (y. B,T).
(26)
I/mi
(27)
Rearranging the incqualities proves the result.
We further decompose (/m²) (1,, B,T) as follows:
I/ (m) (n. B,T) = Eu-ee [LD(M: U)] + 1aa.
(28)
approximation error
optimization error
Now analyze the approximation error and the optimization error separately.
B.2.2 APPROXIMATION ERROR ANALYSIS
The following lemma gives an upper bound on the approximation error. Lemma 6. For random feature models with width m, it holds that
B[CD(H; C2
(29)
where C := supxex SuPussupp(()) flo((u, x)) (p())).
Proof. Since the distribution D is well-behaved, there exists function supp(U)) + R such that sup SuPuesupp(()) [u())] < oo and it holds for all æ E X' that
(x) Eu x))
(30)
20

---

Published as a conference paper at ICLR 2026
Let R be the vector whose i-th entry is vm (2t)) Then, we have
(1-1 (
B[CD(H:U) ((C((0 ( =
E Elay)
a(u, a))u(u;)
(31)
1
( Varuwul (ou, x))(())
(32)
m
Ci
(33)
m
where the last inequality follows from the fact that Varwwat [o((u, x))x(()) S Ct.
B.2.3 OPTIMIZATION ERROR ANALYSIS
Now we analyze the optimization error. We first present a series of lemmas that upper bound a few quantities that related to the optimization error.
Lemma 7. There exists constant R > 0 such that the following conditions hold almost surely over the draw of U:
1. (0)) (x)) S for all X;
2. (x))2 & for all a E X;
3. H < III;
4. tr(2) S R2 (p;
Proof. This directly follows from the following facts: (1) each u, in U is drawn from bounded distribution U; (2) the input space X' is bounded; (3) the activation function is Lipschitz.
Lemma 8. If m is sufficiently large, there exists constant No 0 such that Pr Dmmin H)) 2 Ao 2
1 1- C3e O2m over the draw of U.
Proof. Let T()) L2()2) + L2(xx) be the following integral operator:
f)()) = EE
a(u,, ()) x))) x²))s
(34)
Taking the expectation over U, we have
- where (0)))) f(x) = BR'nD, [K (xx x (x)))
(35)
By applying matrix concentration bounds Pinelis 1994) to ((U) - (x0) we have with probability least 1 Cze Cam -Cm
(17(0) - (ox) lop S Amin (T())) )min(((2)))
(36)
Combining this with Weyl's inequality and setting Ào 1 Amin (T(x)) completes the proof.
Lemma 9, There exists a constant p > O such that the following conditions hold:
1. C(O; U) < bp;;
2. ||:3||2 S Rop
Rp 3. ||l||2 S Aumaa (H)
Proof. Let p² E(Z [ Then L(O 202 Since (1)(1 I VmR for all
E X, we have [|B]|2 < Rp by Cauchy-Schwarz inequality. By the definition of μ., we have ||la||2 Rp Am)
21

---

Published as a conference paper at ICLR 2026
Now we ready to analyze the optimization error. Let 5, : 0, ja. The following lemma provides a bound on dG.
Lemma 10. Given U E IRdxm, in S
Amin 1
then [[[[)r] 1
U can be bounded as follows:
E [1 2
U] (1 7mmin (H))* 1100112 7
tr(E)
(37)
Ami (H)
Proof. Let A, = Bm Σb ((0r)) and & = Bym Db (J/1))( (2)..) - vc,))
Then, we can rewrite the update rule as
B
Oc+1 = 0: 11 Bym
(38)
0,
B
0: 4 B B/m m
1
(0: u. ((x())))) + (110
)1.o)) 0 o(211))
(39)
Vm
0: 4 17 A,(40 n1) TE
(40)
Therefore, have
8141 (I (I- -1):)8; -nE
(41)
Expanding the recursion, we have
6, = (1 (1-nA)-)) ... (. 70 Σ (I -A-1) .......--- nA++1)..
(42)
==0
Since & is mean zero and independent of A, and &. for all < L, we have
111 U I1(I 7) (I A7)
(43)
+ n Σ E II - . 77 11++4)) U]
(44)
Note that for any vector v independent of A., we have BII[(I - ,,,1001)1 Elu
24 + 172 A3 yo]
||0||2 2np Ho + 17 R² I (1 21/2min ( (H) + 17 RT*)
1/01
1
(1 7mmin H )) ||| 112. Then
U < (1 n Amin H)) 1100 + 11 Σ (1 Munin (H)) (r()) (r(E)
1-
El| 8. 1
(45)
B
==
< (1 n Amin H)) 1100112
+
"
tr(E)
(46)
B
Amin
H
which completes the proof.
Lemma 11. There exists constant C4, Co, C. > 0 such that for all S #*.
E(CD(O:U) - Co(u; C. n + Caul E U)] Oge Cam
-Cm-
(47)
B
where the expectation is taken over (0.U) ~ p( B,T).
Proof. With probability at least 1 - Cze - C2m, we have Aumin (H) 2 Ag By Lemma 10 we have
tr (Σ)
E 8r 1 U = (1 - n7ta)' 1100012
-
(48)
B Xa
(1 - nンo)²"
Rp
1
1
R² LD((;U)
(49)
Ao
B
Xo
22

---

Published as a conference paper at ICLR 2026
Let C4 R² (Ro) and C3 R²2 n²., Then have
E(Cn(O:U) - -pllu U R |1 UJ
(50)
C4 (1 - n1>20) + Csn B
LD(w;U).
(51)
If Amiu H) < λo then we still have (D( U)= I Cs for some constant C 0 since 1j S Amaax (H)
and descent lemma holds. Let Cg C3 Putting all the pieces together proves the result.
B.2.4 PUTTING LEMMAS TOGETHER
Proof of Lemma
3 By Corollary
5 it holds for all (0,1] that
I I (D:n. B,T) Lval(f)) S clv
(1 20mp
(n, B,T).
(52)
By Lemmas and
we can bound _ (n) B, T) by
L Im (n, B,T) E[CD(H:U) C₁(1 n/o)" + Can B(Co(: + Coe Cam
(53)
C9CsM Bm
+ +c((1 - (1)o== + Ct Com
(54)
m
Let be parameter. Let 7max large that C₂(1 Aomax I 4. then if r small that Cl < and is not too small to make C1(1 - m/o)" larger than 6. we have
da R
If m is so large that Si + Cam
Cam < 4 and if T is so
T. (n. B,T) S 5.
(55)
Setting small enough 8, have (m) val
(D:n. B,T) Lval(fh) S cCval(ft) B- (1+ 2)
(56)
$ 38 + 2 (f
(57)
where we set A, which completes the proof.
86
Finally, we can make so small that the right hand side is smaller than
23

---

Published as a conference paper at ICLR 2026
C AN EMPIRICAL CHARACTERIZATION OF "TINY" LEARNING RATES (EXTENDED)
Our analysis suggests the potential benefits of training proxy models with tiny learning rates. A natural question is how small should the learning rate be to ensure reliable transferability? We provide both theoretical guidance and practical recommendations here.
Theoretical guidance. While our main analysis focuses on the full-batch gradient descent case, real-world training uses SGD. Here, extend our analysis establish precise bounds what constitutes "tiny" learning rate when mini-batch SGD introduces gradient noise. We stress that this one-step analysis is standard in the literature (e.g. McCandlish et al. (2018): [Smith al. (2018) and has often been shown to give quantitatively useful guidance when coupled with empirical validation. For single SGD update, the parameter change is:
0 = 0 - (VE(() + ()
where & is zero-me meen gradient noise with covariance Σ/B, B is the batch size. We analyze the expected validation loss change through second-order Taylor expansion:
Aaval (0) ~ (0)) (Ve(0) + + 7² 2 ((V
where H := H is the validation loss Hessian. We introduce shorthand notation a = Cval (0) (e(0) for the gradient alignment and b Ve(O)THHH for the curvature term. Expanding the second-order Taylor approximation, we have
1/2 2
Aval (0) ~ -1a evai (0). E (0+25" HVE(0) + +
Taking expectation over mini-batch noise
112 (b+ ((="H)HEE) HE))
[[ -1a +
2
For the quadratic term, have IE|5" H [E(ur(HEET)) tr(HHBlE"") tr(HE)/B yielding:
112 b+ tr( tr(HE)
B[ -na +
B
For dataset quality assessment to remain consistent, the first-order term must dominate the gradient dynamics. Consider datasets D, and D, with corresponding values a, by E, and aj. bj. E,. The datasets will maintain their relative ordering based on the first-order terms when:
2 b, by + [rr(H))) - tr(HE,)]
In<a; - a3))
>
2
Rearranging to solve for ry:
2/a, agl
<
b. by + [rr( HΣ,) tr( HE,)]]
To find a uniform bound valid for all dataset pairs, we define:
Aa = min (a, anl ig
(smallest alignment gap between two datasets)
G2 max V (0)]
(maximum squared gradient norm)
= max Ex) (maximum gradient trace variance)
b
For the curvature term difference. have:
b; by (L, -V "
Amax (1 + [()(
2A G2
24

---

Published as a conference paper at ICLR 2026
where Amax
|| H is the spectral norm of the Hessian. Similarly, for the noise term difference:
Amax
2 2Amax g
B (n(HH,)) tH(HH))] &
(tn(E,) + t
B
B
Combining these bounds and applying the triangle inequality:
1 b; by 1 (I((H)))) tr tr(HE,))))
o G²
2Amax
B
This yields our final upper bound on the learning rate:
Aa
Thiny
G2
+ 0²
(58)
Amax
Remark: Scope of the analysis. Our bound in Equation (58) provides local guidance by controlling the expected loss change for the next SGD step. However, practical training involves multiple steps where parameters evolve continuously, and higher-order terms along with ill-conditioning effects can become significant. Following established practices in analyzing critical batch size (CBS) (McCandlish et al. 2018). we treat this bound as theoretical guidance rather than strict constraint. Empirical validation. Figure and 8 demonstrate that our theoretically predicted values of Qjiny align well with empirically observed regimes of high cross-scale transferability. We estimate Viin using model checkpoints obtained after short warmup training. Specifically, we train each candidate dataset for 500 warmup steps using standard learning rate of 3 x 10 Karpathy (2022), with complete training configurations detailed in Appendix D. We then compute the upper bound from Equation (58) using statistics gathered from 20 batches sampled during this warmup phase. Our bound estimation requires computing Hessian eigenvalues and gradient variances, which we estimate efficiently through power iteration combined with Pearlmutter's 's Hessian-vectes products and a covariance trace Monte Carlo estimator. Importantly, this estimation process requires only minutes of computation time, making it practically viable for real-world applications. As shown in both Figure [ the predicted values consistently fall within the learning rate regime that exhibits very high cross-scale transferability, thereby validating both the theoretical soundness and practical utility of our analysis.
Choosing Thay in practice: simple rule of thumb. While estimating Tuny from proxy model checkpoints is computationally inexpensive, we find that simpler heuristic suffices for most LLM pretraining scenarios. In typical LLM pretraining, gradient clipping is used to control gradient magnitudes, with the batch gradient norm capped at some constant C. We can therefore conservatively
set G = C. Substituting these values into our bound yields Muny < Amaa CT
Au
Since optimal learning
rates for neural network training typically scale as 1/Amax (LeCun et al. 2002). Thiny should be orders of magnitude smaller than the optimal learning rate, with the specitic factor depending on the minimum gradient alignment difference between any dataset pair. Empirically, we find that using learning rates 1-2 orders of magnitude smaller than optimal generally works well. For language model pretraining, this typically translates to learning rates of approximately 10-5 to 10 Radford et al, (2019): Karpathy (2022) As demonstrated in Figure 7 and 8 these predicted values consistently achieve high cross-scale transferability across various model architectures and scales. Importantly, while such small learning rates would be impractical for complete model training due to slow convergence, they prove highly effective in assessing and comparing different data mixtures.
Lower limit of Thiy.. We further remark that in practice, Thiny can also not be too small due to factors such floating point errors. While extremely small learning rates benefit transferability, there exists a practical lower bound determined by finite floating point precision. In 32-bit every number is represented with 23-bit mantissa, giving around 7 decimal digits of precision. The gap between 1.0 and the next larger representable value is therefore 2-23 any increment smaller than this is rounded away. Consequently, for parameters whose magnitude is O(1), the parameter update magnitude (i.e., learning-rate sawting-rate-iim---raaaeee times-gradient product) below 10-7 will be flushed zero, imposing practical lower bound on usable Tany.
format,
23 Rs 1.1910-7 where
25

---

Published as a conference paper at ICLR 2026
Correlation between proxies
v5 woopiimally-eee tuned large target models (GP TP-T2-ooo Large, 774M))
10
Spearmnn Correlation
0.6
Predicted upper bound tiny regime
6212-5mal (125M)
0.0
Learning Rate (proxy)
Figure 7: The same GPT2-Small curve in Figure 6 (a), but highlighting the theoretical bounds of tiny learning rate regimes. The shaded region indicates the theoretically predicted range for Muny with the upper bound given by Equation [88
Correlation
prokies
vs spotimally large target models (GP Le-Lag,, 774M)
10:
Spearman Cornelation
0.6
Predicted upper bound tiny LR regine
7yMhio-70M
0.0
10
Learning Rate (proxy)
Figure 8: The same Pythia-70M curve in Figure 6 (a) but highlight the theoretical bounds of tiny learning rate regimes. The shaded region indicates the theoretically predicted range for Tuny with the upper bound given by Equation (58).
26

---

Published as a conference paper at ICLR 2026
D EXPERIMENT SETTINGS & ADDITIONAL EXPERIMENTS
D.1 ADDITIONAL EXPERIMENT SETTINGS
D.1.1 DETAILS FOR EVALUATION METRICS
Top-k Decision Regret. To quantify the practical risk of proxy-model-assed dataset selection, define the top-k decision regret as the performance gap between the truly optimal dataset and the best-performing dataset among the proxy's top-k recommendations. Let Sş denote the set of & datasets with the highest small proxy model performance. The top-k decision regret is then expressed
A (A)
min DEEA
min DjED,AEA
(D;;:()
where the first term represents the best achievable performance (in terms of validation loss) among the proxy's 's top-k selections when each is tuned on the large target model, and the second term represents the globally optimal performance across all candidate datasets. This metric captures the performance degradation that could result from relying on proxy model rankings, providing an estimate of the practical risk in dataset selection workflows.
D.1.2 MODEL TRAINING AND HYPERPARAMENT TUNING
Model architectures. We evaluate on 3 classes of different model architectures: GPT-2 (Radford al.) (011)), Pythia (Biderman et al. 2023), and OPT (Zhang al., (022). For target models, we use GPT2-Large (774M) and Pythia- IB. For small proxy models, we use GPT2-Small (125M), Pythia-70M, and OPT-125M.
Training configurations and hyperparameter tuning, For all training runs, we set token- parameter (TPP) ratio be 20, i.c.. 1x Chinchilla's compute-optimal ratio (Hoffmann al (2022) with the exception of the ablation study in Appendix D.2.2 We use AdamW optimizer (Loshchiloy Hutter 2017) across all training runs. For hyperparameter tuning on large target models, we use simple grid search across the three most important hyperparameters in LLM pretraining: learning rate, batch size, and weight decay. For learning rate, we sweep from 8 x 10-5,
10- 5 x 10-4, 7 x 10-4, 1 x 10-3, For batch size, we sweep from 128, 256, 512, 1024. For weight decay, we sweep from 0.001, 0.01, 0.1. For small proxy models, we set batch size as 128 and weight decay 0.1, with additional ablation studies
10
other choices of these two hyperparameters in Appendix [.2.2] For both small proxy and large target models, we set other hyperparameters the AdamW optimizer to be 31 0.9, 822- = 0.999, and 1e 8. We fix the context length at 1024 tokens across all training runs. We follow the Warmup-Stable Staul--ttabe-Deeer Decay (WSD) learning rate scheduler proposed in MiniCPM Hu al. (2024), allocating 2000 training steps for warmup, 66% for stable training at peak learning rate, and the remaining for linear decay. Throughout this work, "learning rate" refers to the peak learning rate achieved during the stable phase of the WSD schedule. We use full precision training for small proxy models and bfloat16 precision for large target models. We set gradient clipping with a maximum norm of 1. to stabilize training dynamics across all training runs. All model training is conducted on 32 NVIDIA H100 GPUs with 80 GB of memory.
1.1.3 DOWNSTREAM BENCHMARK EVALUATION
We use the Language Model Evaluation Harness Gao al. 2024) for benchmark evaluation. For each benchmark, the framework computes the length-normalee log- log-likelihood of the model on each candidate answer and marks response as correct if the correct answer receives the highest logelikei score.
1.1.4 DETAILED DATASET RECIPE CONFIGURATIONS
We evaluate tiny learning rate approach across 23 carefully designed data recipes that span wide spectrum of practical data curation decisions faced by LLM pretraining practitioners. The detailed description for the data recipes considered in this work is summarized in Table
Category 1: Domain compusition and ablation. We create 10 domain mixture variations from The Pile (Gao et al. 2020). Specifically, 4 recipes maintain a fixed 70% Pile-CC allocation while
27

---

Published as a conference paper at ICLR 2026
Table 1: Summary of 23 dataset recipes used in experiments. Recipes grouped by category
represent different data curation decisions faced in practice.
Category
Recipe ID
Description
Pile-CC70-S0A30
70% Pile-CC, 0% StackExchange. 30% ArXiv
Pile-0070-S10A20
70% Pile-CC, 10% StackExchange, 20% ArXiv
Pile-CC70-920110
70% Pile-CC, 20% StackExchange, 10% ArXiv
Pile-Cc70-S30A0
70% Pile-CC, 30% StackExchange, 0% ArXiv
Domain Composition
Pile-noArXiv
Full Pile excluding ArXiv
Pile-noGitäüb
Full Pile excluding GitHub
Pile-noBuroParl
Full Pile excluding EuroParl
Bile-noStackExchange
Full Pile excluding StackExchange
Full Pile excluding PubMed Central
Pile-noRLkipedia.
Full Pile excluding Wikipedia
C4
C4 (Raffel et al
2020
Established
Datasets
DCIM
DCLM-baselme
2024
RefinedWeb
RefinedWeb (Penedo el a. 2023
RP2-H50T50
50% head-middle, 50% tail
RP2-H60T40
60% head-middle, 40%
Quality
RF2-170т30
70% head-middle, 30% tail
Filter
RF2-H80T20
80% head-middle, 20% tail 90% dead-middle, 10% tail 100% head-middle, 0% tail
RF2-H90T10
RP2-H100TO
DCIM-dedup-L
Striet kin/ppragraph thresholds Very strict line/paragraph thresholds Striet 2,3,4-gram thresholds Strict 5,6,7,8,9, 9,11,,,,0--ramm thresholds
Deduplication strength
DCIM-dedup-LS
DCI dcl.M-dedup-G
DCC MGddddedd
varying the remaining 30% between StackExchange and ArXiv. Additionally, 6 recipes created by excluding one domain from the full Pile dataset to assess domain importance.
Category 2: Established datasets. We evaluate 3 widely-adopted pretraining corpora: C4 (Raffel
jet al., 2020), DCLM-baseline (Li et af.] (024), and RefinedWeh (Penedo et al., 2023). They all originate from the Common Crawl but vary in their curation strategies, including data filtering criteria and deduplication algorithms.
Category 3: Scoring-based data filter. We construct 6 data recipes using different mixing ratios of head-middle versus tail partitions from RedPajjma-V22 Webber 100:0. These partitions are defined by the perplexity scores from 5 5-gram Kneser-Ney model trained on Wikipedia (Wenzek et al., 2020). Specifically, RedPajama-V V2 (Weber et al.) 2024) partitions its 113B documents into quality buckets based on the perplexity scores from a 5 5-gram Kneser-Ney model trained on Wikipedia corpus. Documents are categorized as "head" (low perplexity). "middle" (medium perplexity), or "tail" (high perplexity). Our six mixing ratios between the head-middle and tail partitions represent different macision in score-based data filtering faced by LLM pretraining practitioners.
Weber et al. (2024), ranging from 50:50 to
Category 4: Deduplication. To understand how deduplication decisions transfer across scales, create variants of DCLM-baseline by modifying the Massive Web repetition filter Rac et al. (2021) used in its curation pipeline. The Massive Web repetition filter removes documents with excessive n-gram repetitions at various granularities. We systematically adjust thresholds for line/paragraph repetitions and n-gram frequencies Massive Web repetition filter are summarized in Table 2 We further summarize the changes in thresholds for each of the newly created 4 data recipes Tables 3 4 and 6 At a high level, the default thresholds are relatively conservative, and the newly created 4 data recipes are more aggressive in removing documents with repeated lines (DCLM-dedup- or short n-grams (DCIM-dedup-Gar DCI (CLM-decup-GS).
create new data recipes. The default thresholds for the
DCLM- DCLM- dedup-LS)
28

---

Published as a conference paper at ICLR 2026
Table 2: Default Massive Web repetition thresholds (used in DCLM). document is filtered if any measurement exceeds its threshold; lower is stricter.
Measurement
Threshold
Duplicate line fraction
0.30
Duplicate paragraph fraction
0.30
Duplicate line character fraction
0.20
Duplicate paragraph character fraction
0.20
Top 2-gram character fraction
0,20
Top 3-gram character fraction
0.18
Top 4-gram character fraction
0.16
Duplicate 5-gram character fraction
0.15
Duplicate 6-gram character fraction
0.14
Duplicate 7-gram character fraction Duplicate 8-gram character fraction
0.13
0.12
Duplicate 9-gram character fraction
0.11
Duplicate 10-gram character fraction
0.10
Table 3: Massive Web repetition thresholds for DCLM-dedup- Changes vs. baseline in bold.
Measurement
Threshold
Duplicate line fraction
0.20
Duplicate paragraph fraction
0.20
Duplicate line character fraction
0.15
Duplicate paragraph character fraction
0.15
Top 2-gram character fraction
0.20
Top 3-gram character fraction
0.18
Top 4-gram character fraction
0.16
Duplicate 5-gram character fraction
0.15
Duplicate 6-gram character fraction
0.14
Duplicate 7-gram character fraction Duplicate 8-gram character fraction
0.12
Duplicate 9-gram character fraction
0.11
Duplicate 10-gram character fraction
0.10
Table 4: Massive Web repetition thresholds for DCLM-dedup-LS. -LS Changes vs. baseline in bold.
Measurement
Threshold
Duplicate line fraction
0.15
Duplicate paragraph fraction
0.15
Duplicate line character fraction
0.10
Duplicate paragraph character fraction
0.10
Top 2-gram character fraction
0.20
Top 3-gram character fraction
0.18
Top 4-gram character fraction
0.16
Duplicate 5-gram character fraction
0.15
Duplicate 6-gram character fraction
0.14
Duplicate 7-gram character fraction
0.13
Duplicate 8-gram character fraction
0.12
Duplicate 9-gram character fraction
0.11
Duplicate (0-gram character fraction
0.10
29

---

Published as a conference paper at ICLR 2026
Table 5: Massive Web repetition thresholds for DCIM-dedup-G (tightening 2-4-gram tests).
Changes vs. baseline in bold.
Measurement
Threshold
Duplicate line fraction
0.30
Duplicate paragraph fraction
0.30
Duplicate line character fraction
0.20
Duplicate paragraph character fraction
0.20
Top 2-gram character fraction
0.14
Top 3-gram character fraction
0.12
Top 4-gram character fraction
0.10
Duplicate 5-gram character fraction
0.15
Duplicate 6-gram character fraction
0.14
Duplicate 7-gram character fraction
Duplicate 8-gram character fraction
0.12
Duplicate 9-gram character fraction
0.11
Duplicate 10- gram character fraction
0.10
Table 6: Massive Web repetition thresholds for DCIM-dedup-GS (tightening 5- 5-10-gram tests).
Changes vs. bascline in bold.
Measurement
Threshold
Duplicate line fraction
0.30
Duplicate paragraph fraction
0,30
Duplicate line character fraction
0.20
Duplicate paragraph character fraction
0.20
Top 2-gram character fraction
0.20
Top 3-gram character fraction
Top 4-gram character fraction
0.16
Duplicate 5-gram character fraction
0.08
Duplicate 6-gram character fraction
0.07
Duplicate 7-gram character fraction
0.06
Duplicate 8-gram character fraction
0.05
Duplicate 9-gram character fraction
0.04
Duplicate -0-gram character fraction
0.03
30

---

Published as a conference paper at ICLR 2026
D.2
ADDITIONAL RESULTS & DISCUSSION
D.2.1 ADDITIONAL RESULTS FOR SECTION 6.2
This section presents supplementary results that extend the findings reported in Section 6.2 Figure Figure [5| demonstrate the top-k decision regret performance when using alternative proxy architectures (Pythia-70M and OPT- (PT-125M) when GPT2-Large is the target model. These results corroborate our main findings by showing that the tiny learning rate approach consistently outperforms standard learning rates across diverse model families. In Figure I1 12 and 13 we present the top-k decision regret decomposed across specific evaluation tasks, including HellaSwag, Winogrande, Open ARC-Easy, and CommonsenseQA,
9
well the aggregate performance across all benchmarks. Figure 16 to 22 replicates the complete experimental protocol using Pythia- B as the target architecture. The consistency of improvement' across different evaluation metrics and proxy/target model architectures strengthens the practical applicability of our method in real-world deployment scenarios.
Proxy training runs with tiny learning rates exhibit significantly smaller stochasticity. In addition achieving superior rank correlation, proxy models trained with tiny learning rates demonstrate greatly reduced variance in transferubility. As we can see in Figures (a) and 16] the variation in rank correlation due to training randomness substantially narrower for learning rates below 10-4 This phenomenon can be attributed to the fact that tiny learning rates produce smaller parameter updates, making the training process less sensitive to stochastic factors. More specifically, a very small learning rate produces proportionally smaller parameter diffusions, reducing sensitivity to stochastic factors such as random initialization and gradient noise (jastrzkebski et al. 2017] Stephan et al. (017)). This stability advantage of tiny learning rates provides additional practical benefit: practitioners can achieve consistent dataset rankings without requiring multiple training runs to account for stochastic variation.
6ap between aptimal dateset and top-k prexy selections on tuned torget models (Pile vol loss)
Tiny- proxy (Pythio-7OM, 05) Standard proxy (Pythio-70M. 0.001)
0.15
Loss Gap
0.10
0.05
0.00
Top-k
Figure 9: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (Pythia-70M) selections on tyyerparameter-optiiiii target models (GPTe-Large). measured using Pile validation loss. Shaded areas show 95% bootstrap CIs computed by resampling seeds (3 runs per dataset).
31

---

Published as a conference paper at ICLR 2026
6ap between optimol dotesst and top-k proxy selections on tuned torget models (Pile val loss)
0.25
Tiny-LR presy (OPT-125M, 1e-0E)
Standard proxy (OPT-125M, 0.0003)
0.20
Loss Gop
0.25
0.10
0.05
0.00
7
9
Top-k
Figure 10: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (OPT-125M) selections on byypeparamette--opiiiiii target models (GPT2-Large), measured using Pile validation loss. Shaded areas show 95% bootstrap CIs computed by resampling sceds (3 runs per dataset).
Gap between optimal citusse and top-k prexy selections funed target models (Hill)Swgg)
Gap between optinal dateset and top proxy selections tuned harget models (OponBcoo(QAA
15
0.14
Tey--R pracy (UPTZ) Smadl be:5;
Tiny-u prazy (P7T2 Small 3x-5)
Snandord (k prayy (SP72 Seall 0.0003)
14-
Standord pracy (8PT2- -er2 00000))
Accuracy Sap (%)
0.12
Accurany Sap (%)
12-
0.00
10-
0.08
3.5
0.6
0.04
0.4
0.02
02-
0.00
0.0
5
Top-k
Top-k
6ap between optimal cotosst and top- k praxy so selections on funed torget models (ARC-Eany)
6ap between optinal dataset and top-k proxy selections on funed forget modele (Winegrande)
0.7
--- Shandore LB pracy (5PT2 Seall (00000)
Thy LR pracy (6PT2 Sexdl le 5)
0.
--- Standard- pracy (SPT2 decll (i0000)
Tay LR pracy (6FT2 Small le 5)
Accuracy Gap (3)
0.6
Accurscy Gap
05
0.4
0.3
0.5
0.2
0.2
0.1
01-
00
0.0
Top-k
Top.
Eep between optinal dataset and top- k prexy selections on funed hor harget node's (Commonsenooca)
Gap between optinal dateset and top prazty selections on tured tanget models (Avg acress all barchmorks)
1.2
Thy-LR pracy (GPTZ Seall 1o-;;
Tey pracy (PT72 Small Je 5) Standord Ui praty (8P72 deel. 0.0000
Snandord LK paa (5FT2 Seal. 0.000)
0.-
1.-
Accuracy Sap (%)
Accuracy Gap (%)
0.8
0.6
0.4
04-
02
3.
Top-k
Top-k
Figure 11: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (GPT2-Small) selections on hyperparancter-oppmiize target models (GPT2-Large) measured using downstream benchmarks.
32

---

Published as a conference paper at ICLR 2026
6ap between distset and provy selections funed torget (HelloSwwo)
6ap between optindl dateset and top selectiors fured torget models (Open8tok(2A)
15-
0.64
Try pracy (SPT2 Seall, 5)
(iT12 Small pracy (SPT2 Secl. (.0000)
(8 pracy (SPT2 0.0003)
Accuracy Gop (%)
Accuracy Gap (%)
12-
100
0.08
0.5
0.06
0.6
0.04
0.4
0.02
0.-
0.00
00
Top-K
Top -k
Gep ophinal top-k prozy selections target models (ARC-Essy)
Gop betasen optinal datssitt top-k pracy selections forget models (Winegrarde)
0.7
Tay-LR pracy (GPTD-Sex), 36-5)
Tiyy&A prezy Standard LR pracy (SPT2 Secl 00000)
3a-5)
Standard LK prery (SPT2 Small, 0.009)
07
066
Accuracy Gop (25)
0.6
0.5
Accurocy Sop
0.1
044
0.3
0.5
02-
011
011
00
00
Top-
Top
6ap between ophinal cotoset top k prowy selections on torget models i (CommsncerooQA))
Gap between optinal and top proxy selections tured target models (Avg ocrees banchmarks)
1.2
Tim--U prazy (GP12 Small, Se
Try-Li prazy (1-12 Small 5e Standorc pracy 18P72- dech b.ccm
Snandard (8 pray (SP72- Seall o 80000)
0.8
1.-
Accuracy Gap (%)
Accuracy Gup (%)
0.6
0.4
04-
02
0.0
5
7
Top-k
Top-k
Figure 12: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (Pythia-70M) selections on hyperparameter-optiinzee target models (GPT2-Large), measured using downstream benchmarks.
33

---

Published as a conference paper at ICLR 2026
Sep between ophimal detoset and top selections funed horget modele (Hellitiwog)
6op between optindl deteset and top selections tuned models (OpenBook()A)
Ligrrey (OPT
16-
(OPT
pracy 00003)
pracy (OPT 777122
14-
Accuracy 6op (%)
0.20
12
Accuracy Gop
10-
08-
0.4
02
00
Top
Top-k
Sap betwvenn ophimal detoset and top selections tuned torget modals (ARC-Bosy)
6ap between optindi deneset selections
models (Winogranda)
0.12
Liger pracy (DFT-1250 0003)
0.30
Standars-RR prayy 0:C003)
pracy (OPT
Accuracy Gop (%)
0.10
Accuracy Gup (3)
0.20
0.06
0.15
0.04
0.08
0.00
Top kk
Top
Sap between optital detoset and top- selections funed models (ComnonsereeQAA)
6cp between optindl detoset and top pronyy selections tuned torget models (Avg benchmarks)
(OPT:425M..
06-
Tryy prany (0PT:125, praxy (2P7-125M, D.c03))
pracy 0003)
05-
Accuracy Gap (%)
Accurscy Gap (8)
04
0:4
03-
03-
02
01
01
00
00
Top-k
Top-k
Figure 13: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (OPT-125M) selections on hyperparameter-opinnizeee target models (GPT2-Large), measured using downstream benchmarks.
Standard LR Proxy @yythcc70M, 1 10-3)
Tiny LR Proxy (Pythie-70M, 1 10-5)
10-
0.8
Spaammn Correlation
06
0.0
-02
Hackerblars
Einster
Grtucb StackExchungg
RubMed Abstracts
Ackiv NEW ExPorter OpenStokQA
DM Mathensrics
Publited Cermal
Guteeberg g (PG- 19)
USPto Backgrounds
MulPopers
Helladiesg
Enron Emails
Pile-CC
Wongrrnde ondgr
Wikipedia (en)
ARc-Ewe
CommonsonsCCA
Freetas
Figure 14: Average rank correlation between proxy (Pythia-70M) and target (GPT2-L Large) for the loss computed over a variety of validation domains (from Pile) and downstream benchmarks.
34

---

Published as a conference paper at ICLR 2026
Standard Prozy (CPT-125M. 10
(OPT-125M 10
Spuarmen Corvaation
0.6
0.4
00
Pubsed Contral
DM Mathematics
SnackE wcchngg
FreeLow
OpanBookOA
Enran Enails
Gribub
HellsSwin
Commordennsn
Winggrande
BarsPorl
ARC-Eesy
ARC-
Philfopers
USPTO Backgrounds
Arkie
Wikppedia (en)
Publtte Abstracte
NHH ExPerter
Gurenberg (P6-19) HockerNinud
Rie-CC
Figure 15: Average rank correlation between proxy (OPT- 125M) and target (GPTe-Large) for the loss computed over a variety validation domains (from Pile) and downstream benchmarks.
Correlation between fixed LR small proxies
optimally- tuned lorge torget models (Pythio-18)
Conrelation
0.8
0.4
GPT2-5mal (125M)
PyM
1PTM (125W)
10
10-4
10
Learning Rate (proxy)
Figure 16: Spearman rank correlation between dataset rankings from proxy models (GPT2-Small, Pythia-70M, 70M, OPT-125M) and larger target model (Pythia- IB) as a function of proxy model learning rate, evaluated aggregated Pile validation loss. Shaded areas show 95% bootstrap Cls computed by resampling seeds (3 runs per dataset).
35

---

Published as a conference paper at ICLR 2026
Gap between aptinal dotrset and top-k selections on funed torget models (HellnSeng)
Gap between ipti dotoset and top-k selections funed torget modeds (OpanBcokQ4)
Try (6FT2 Seat,
Thy (677 Sexil le-5)
2,5
Stendard-UR pracy (PP12 Small, 0:0003)
Standard (6P12 Seall 0.000))
Accuracy Gop (%)
2.-
Accuracy Gep (3)
15-
:5
10-
10-
0.5
0.0
0.-
Top- -k
Top
between op timal detsset prasy selections target models (ARC
Gep betwees dataset and top-k selections fured forget medels (Winegrande)
Tay presy Steedard pracy (PPT2 Small, 0.0003)
25-
Thy-U pracy Se-5)
Standord LK piryy (SPT) 0.009)
Accuracy Sop (7)
1.5
Accuracy
1O-
0
00
Top k
Top.
Gop between op timel datspet top-k k prazy selections tuned torget models (CommonsersecAA)
Sep betweer opt dataset and top-k srony selectiond tuned target models (Aig acress banchmarks)
10
-
Steedard pracy (6P72 Small 0.0003)
praay (6F Ie-b)
2.5
Tey-Ll pracy (jrrg-Smed) te:3) Snandod prayy (SP72 Seal. 0.0000)
Accuracy Gap (%6
0.8
Accuracy Gap (%)
20-
0.6
02
05-
0.
00
Top-k kk
Top-k
Figure 17: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (GPT2-Small) selections on hyperparameter-oppiinze target models (Pythia-1B), measured using downstream benchmarks.
36

---

Published as a conference paper at ICLR 2026
6ap between cososet and provy selections funed torget (HelloSwog)
6ap between optinal datspet and top selectiors fured torget models (Opon8tok(24A
0.14
proee 2)
Try day.k prasy (Pythis D.000)
0.12
prowy Pyytee Nowio70w.coo
0.5
Accurncy Gop (%)
0.10
Accuracy Gap (%)
0.4
0.08
0.06
0.2
0.00
0.0
Top-K
Top -k
Gep ophinal top-k prozy selections targe? models (ARC-Essy)
Gep betwsen optinsl datoseet top-k pracy selections torget models (Winegrarda)
TieyLR prazy (Pyths-70M, (e-5)
0.6
Tiy- pracy doymke-7 16-3) Stor dand Ui prany (Pythi 70M, 0.000
Standard LR pricy Pyrtic C.000
Accuracy Gop (75)
0.5
0.3
Accurocy Sop
0.5
0.4
0.-
0.3
02-
01
01
00
0.0
Top-k
Top
Sap between ophinal dotaset top prowy selections on forget models s (Comm:neeroeQA)
Gap between optinal and top prozy selections tured tanget models (Avg ocrees banchmarks)
0.5
Tiny grany (Pythss 70M, 5)
Trp Li preve (Py 1e Stor dand-LA grazy (Pythis 70M, 0.000)
Standord LB praay Pyrtio c 001)
0.4
0.4
Accuracy Gap (%)
Accurany Gup (%)
0.3
0.5
0.
01
0.1
00
30
5
Top-k
Top-k
Figure 18: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (Pythia-70M) selections on hyperparameter--opiiizeee target models (Pythia-IB), IB), measured using downstream benchmarks.
37

---

Published as a conference paper at ICLR 2026
6ap between ophimal detoset and selections funed horget (HellsSweog)
6ap between optindl dateset and top selections fured models (OpenBrok(4A)
0.35
(OPT:125M,
(SFT
0.30
pracy
precy
0.8
Accuracy Gap (%)
0.25
0.20
Accuracy Gap
2.6
0.15
0.4
0.2
0.0
Top-k
Top
provy selections between funed target models (MRC Ersy)
Gap between optinal datsset selectiond tuned torget models (Winegranda)
0.40
Timp prezy Se:5;
praay 3e:5)
Stosoard procy 125M 0.0003)
Snunderd pricy (OPT:327M), 0.0000)
Accurney Gop (%)
0.50
Accuracy Gap (%)
0.25
0.6
0.4
0.15
0.00
00
Top-K
Top -k
Gep between optimal distoset top-k prosy selections torget mode's (Commoneensic(a))
Gap between optinal top-k procy selections
target models (Avg banchmarks)
Tiry preay Se-1;
praay (OFT-128M 1e Shanderd prccy (255-1154) 0.0000)
0.8
Steacord pracy (OFT 125M 0.0025)
Accuracy 6op (75)
0.6
Accurocy Gop
0.4
0.4
0.3
0.-
011
00
0.0
Top-k
Top-k
Figure 19: Top-k decision regret between the optimal dataset and the best dataset among top-k proxy (GPT2-Small) selections on hypeerprameteeroppiinzee target models (Pythia-1B), measured using downstream benchmarks.
Stardard LR Proxy (Pythic 7DM, 1 1000
Tiny LR Proxy (Pythia 70M, x 10"")
0.8
Sparmen Corveiation
0.4
0.0
-02
OpenBookOA
Freeter
HelleSwng
Poblted Central
Dill Hathonatics
StackE echange Frro&& Emaile
Grttub
EirePart
Wilipedia (en)
ARC ARc-Easy
Commonsenct OA
USPTO SO Backgrounds
Pubteed Abstracte NIH ExPortes
ArXiv
Winegrond onde
Getenberg (RG (9G-19) 19)
HeckerNert
PhilPspers
File-CC
Figure 20: Average rank correlation between proxy (Pythia- 70M) and target (Pythia- IB) for the loss computed over a variety of validation domains (from Pile) and downstream benchmarks.
38

---

Published as a conference paper at ICLR 2026
Freny Small, 10"4)
Tiny Proxy Small, 1x
Spearmur Correlation
0.8
0.6
0.2
Pubsed Control
StackExchangg
GrbuD Rublleed Abstracts
KIH ExPorter Baukgrounds
USPTO Basker groudds
Ark
EuroPori bi Merhematics
Hellasseug
HackerNean
Phil appers
Enron Emails
Guteeberg (P6-19)
CommunsenseCL
Pile-C
Wi Wibipedia (en)
Freatow
OpenBookaA
Minegrondh
ARC-Eooy
Figure 21: Average rank correlation between proxy (GPT2-Small) and target (Pythia- IB) for the loss computed over a variety of validation domains (from Pile) and downstream benchmarks.
Standard LR Proxy (OPT- T-125M 3 10-*)
Tiny LR Proxy (OPT-125M 1 10
0.8
Sparrmnn Carventte
0.6
0.
RubMed Contral
DMt Nathematics Frad.ow Fre..ow
Stack Exchange
OpanBook(A
A
Enron Ensils
Grttsb
SellsSwog Commonsennon
Winsgg Winsgrande onde
BaroPori
Alc--Ess
Phifospers
USPTO Backerounds
ArXiv
Wikipedia (en)
Pubdied Abstracts
NIH ExPorter
Gurenberg (96-19) HockerNized
Pie-CC
Figure 22: Average rank correlation between proxy (OPT-125M) and target (Pythia- IB) for the loss computed over variety of validation domains (from Pile) and downstream benchmarks.
Correlation between fixed LR aptimally tuned performance (Gemma3-270M
1.00
Spearman Correlation
0.95
0.90
0.85
0.80
GPT2-Small (125M)
P0th
0.75
OPT-125M
10-6
10-5
10-4
10-3
Learning Rate (proxy)
Figure 23: Spearman rank correlation between dataset rankings from proxy models (GPT2-Small, Pythia-70M, OPT-125M) and larger target model (Gemma3-270M) as function proxy model learning rate, evaluated aggregated Pile validation loss. Shaded areas show 95% bootstrap Cls computed by resampling seeds (3 runs per data recipe).
39

---

Published as a conference paper at ICLR 2026
D.2.2 ABLATION STUDIES OTHER HYPERPARAMETERS
Figure 24 25
and 26 conduct ablation studies to examine the transferability of small proxy models trained with varying batch sizes, weight decay coefficients, and token-per-parameter ratios (TPP), again with respect to the optimally-tun -aune large target models. All experiments maintain our single- epoch training protocol with no sample repetition.
Batch size and weight decay, As illustrated in Figure and 25 we systematically evaluated batch sizes spanning [32, 64, 128, 256} and weight decay coefficients {0.001, 0.01, 0.1, 1.0) (covering the typical range used in L.MM pretraining). The results demonstrate that dataset rankings maintain Spearman correlations above 0.90 across all configurations when small learning rates are used. This stands in sharp contrast to the dramatic ranking reversals observed with learning rate variations (correlation drops below 0.75 at standard learning rates). These findings justify our focus on learning rate as the primary factor affecting proxy model transferabilityy
fakken-per-parameet ratio (TPP). Due to computational constraints, the TPP ablation study employed Pythia-410M as the target model and was limited to 14 data recipes, as each recipe required dataset-specifie hyperparameter optimization. Figure 26 examines TPP ratios ranging from 20 (Chinchilla optimal) to 160 (8x Chinchilla optimal). Across this range, dataset rankings maintain high correlation with the target model's optimal rankings (Spearman p > 0.85), demonstrating robustness to overtraining. We attribute this stability to the one-epoch convention of LLM pretraining, which eliminates sample repetition and thus avoids the overfitting dynamics that could otherwise confound dataset comparisons.
Our comprehensive ablation studies across batch size, weight decay, and TPP ratios demonstrate that dataset rankings remain stable across these dimensions within practical training regimes, provided small learning rates are used. This validates learning rate as the critical hyperparameter for proxy model transferability and confirms that our proposed tiny learning rate approach provides robust dataset selection across the diverse training configurations encountered in practice.
Correlation between fixed LR small proxies optimally large target models T2 T2-gar 774M)
10
Spearmah Correlation
0.8
0.4
Batch Size Prozy Training
0.2
Batch, Size 54 for Proxy Training
Batch
Training
Batch Size 256 for Proxy Training
10
10 Learning Rate (proxy)
10
Figure 24: Spearman rank correlation between dataset rankings from proxy models (GPT2-Small) trained with different batch sizes and larger target model (GPT2-Large) as function of proxy model learning rate, evaluated on aggregated Pile validation loss.
40

---

Published as a conference paper at ICLR 2026
Correlation befween fixed small proxies
optimally-Tuned lorge target (GPT2-Large, 774M)
10
Spearmon Cornelation
0.6
Weight becay : Praxy Traknin
0.2
Weight Decey 0.01 for Proxy Troining
Wright Dessy
Prozy Traiiing
Weight Dessy 1 for Praxy Training
10
10 Learning Rate (proxy)
10a
Figure 25: Spearman rank correlation between dataset rankings from proxy models (GPT2-Small) trained with different weight decay values and larger target model (GPT2-Large) as function of proxy model learning rate, evaluated on aggregated Pile validation loss.
Correlation between fixed LR small proxies osoppinolly-tuned lorge toryet models (Pythic-410M)
1.0
Spearmon Correlation
0.8
0.6
1PPx1 Chirchila for Prozy Training
0.2
TPP-2x Chinchills for Prazy Training
Ctirchila Praxy Traising
TPP:Bx Otinchilla for Praxy Training
10
10 Learning Rate (proxy)
10
Figure 26: Spearman rank correlation between dataset rankings from proxy models (GPT2-Small) trained with different token-pet perp-eereeeeeee ratios (TPP) and larger target model (Pythia-410M) as function of proxy model learning rate, evaluated on aggregated Pile validation loss.
41

---

Published as a conference paper at ICLR 2026
D.2.3 VISUALIZATION OF LEARNING RATE VS MODEL PERFORMANCE
To further illustrate how different data recipes exhibit varying optimal learning rates, we provide detailed learning rate vs validation loss curves for the six data recipes in "scoring-baser data filter" category. As shown Figure 27 the data recipes exhibit non-trivial gaps in optimally tuned performance (> 0.2 in validation loss between best and worst). Furthermore, their optimal learning rates are not fixed. We can see that the data recipes with higher head middle percentages require higher learning rates. In particular, when the learning rate is 3 x 10-3 the 70:30 mixture appears best, but optimally tuned performance > 0.05 worse than that of the 50:50 mixture. In contrast, smaller learning rates (< 1 10-3) correctly rank these data recipes. Note that the 50:50 mixture (including substantial "tail" data) achieves optimal performance, illustrating that quality signals like perplexity are imperfect proxies (Wikipedia represents only small portion of the Pile dataset). This underscores the critical role of proper ablation experiment protocols.
Volidation Loss of Pythio-18 at different learning rates
4.0
RF2--60-TO0
8F2-H80-T20 -20
RP2-1100-440 RP2-490-T10 T10
AP2-H70-T30
AP2-HICO-TO
Validation Loss
3.7
3.6
3.3
10
10
10
Learning Rate
Figure 27: Validation loss curves as a function of learning rate for six RedPajama- V2 data recipes mixing gead_middle and tail partitions. Each curve represents a different mixing ratio, with RP2- H50-T50 denoting 50% headlmiddle and 50% tail, up RP2-H100-TH (100% head middlee. Black circles mark the optimal learning rate for each recipe. All models are Pythia- IB trained on Pile.
42

---

Published as a conference paper at ICLR 2026
D.2.4 EMPIRICAL VERIFICATION OF GRADIENT ALIGNMENT HYPOTHESIS
In this section, empirically validate the mechanistic hypothesis proposed in Section that tiny learning rates succeed by isolating the stable first-order gradient-alignment signal in the loss reduction.
We directly measure the gradient alignment term. Specifically, we define the accumulated first-order gradient alignment score Jalign = Er Veva( (8:) Vd(0;; .Br) where Be is the training batch at step and Eval is the loss on the validation set. This is the first-order approximation of the reduction in validation loss for vanilla SGD. In the experiment, we compute this score over the first T = 2000 steps for both proxy (GPT2-Smal) and target (GPT2-Large) models across the 23 data recipes.
Scoss-scale stability of first-order alignment. Figure 28 (a) shows the correlation of raw gradient alignment scores between proxy and target models. We observe near-perfect correlation (p 0.98) across small learning rates, supporting our claim that first-order interactions are structurally stable across model scales. At standard learning rates, correlation decreases but remains reasonably high (p > 0.8). This decrease is expected because model parameters diverge more substantially when trained with larger learning rates, and since gradients depend on parameter values, this divergence naturally reduces cross-scale correlation.
Gradient alignment actual performance ranking: Figure 28 (b) shows the relationship between gradient alignment and actual performance rankings. The orange line shows the Spearman correlation between the proxy model's accumulated gradient alignment and the validation loss of the optimally- tuned large target model. Crucially, this correlation remains high (papprorr..6) higher learning rates, whereas the correlation based on the proxy's actual training loss (blue line) drops significantly below 0.4. We note that while highly predictive, this metric theoretically represents the update direction for vanilla SGD; since our actual training employs AdamW, which adapts updates based on moment estimation, the raw gradient alignment serves robust but slightly approximate proxy for the realized loss reduction.
and stable even
Correlation order grodient alignment between target and proxy models
Correlation between fixed small proxies vs optimally tuned lorge target models Te-Lge,,,, 774M)
1.0
Spearnon Correlation
0.9
Spearman Correlation
0.9
0.8
0.8
0.7
0.6
0.6
0.
TargetLRrtee
0.5
Praxy Traning
Torget LB-le-4
First order gradient aligmeet
0.4
10
10-1
10
10t4
(a)
Learning Rate (proxy)
(b)
Learning Rate (proxy)
Figure 28: Empirical verification of the gradient alignment hypothesis. (a) Spearman correlation of accumulated gradient alignment scores between proxy (GPT2-Small) and target (GPT2-Large) models across varying learning rates. High correlation at small learning rates confirms that first- order gradient dynamics remain structurally stable across model scales. (b) Comparison of ranking transferability to optimally-tun
large target models: proxy model's actual validation loss (blue) versus accumulated first-order gradient alignment (orange). The gradient alignment signal maintains strong correlation even at learning rates where actual loss rankings collapse, supporting our hypothesis that first-order terms provide a stable foundation for cross-scale dataset comparison.
43

---

Published as a conference paper at ICLR 2026
D.2.5 EFFECT OF TRAINING DURATION ON TRANSFERABILITY
We investigate whether early stopping can reliably determine dataset rankings when combined with tiny learning rates, motivated by the common practice of training for only a short period during data-ablation experiments. We measure the Spearman rank correlation between model performance at intermediate checkpoints and the optimally- -tune target model performance at the final step, using Pythia-70M. with the same training configurations as our main experiments (1 total training steps) in Section 6
The results strongly support using early stopping when the learning rate small. Figure 29 a) shows that when proxy models are trained with tiny learning rates (e.g.. 10 10 10-6), high rank correlation is maintained even with substantially fewer training steps. For instance, at 3k steps (approximately 27% of full training), tiny learning rate proxies achieve rank correlations above 0.95. This suggests that under the tiny learning rate regime, dataset rankings emerge early and remain stable, potentially enabling significant computational savings. In contrast, with standard learning rates, rank correlation generally improves as training progresses toward convergence.
The transferability of early checkpoints depends the learning rate schedule. We also observe that transferability at Ik steps mnid-warmu)) is notably worse than at later checkpoints across all learning rates. We hypothesize this is because the warmup phase involves rapidly changing learning rates and potentially unstable optimization dynamics. To investigate this, we conducted pilot experiment reducing the warmup period from 2,000 200 steps, shown in Figure 2).b). With shorter warmup (i.e., peak learning rate reached earlier), the lk-step checkpoint achieves comparable transferability later checkpoints. While preliminary. this suggests that the transferability of early checkpoints depends on the learning rate schedule. We leave systematic investigation of the interplay between early stopping and scheduler design to future work.
Correlation aetween fixed LR small proxies ouned lerge target madels (2000 wormup steps)
Cornelation between fixed LR small prokies optimpmlly tuned large torget models (200 wormup stops)
10-
1.0
Speorman Correlation
Spearman Correlction
0.9
0.8
0.7
0.7
Lost stop [1ik sheps)
Last step (11k stepe)
0.6
sTapa
skess
staps
5k steps
stapal
0.5
3K steps
stops
stops
0.4
0.4
10
10
10-3
10
10
10
(a)
Learning Rate (proxy)
(b)
Learning Rate (proxy)
Figure 29: Effect of training duration on proxy-to-target transferability. We measure Spearman rank correlation between dataset rankings from proxy models (Pythia-70M) at intermediate checkpoints and optimally-tuned target models (GrTe-Large) at the final step. (a) With 2,000 warmup steps: tiny learning rates (S 10-5) achieve high rank correlation early in training, while standard learning rates require more training steps for reliable transferability. (b) With 200 warmup steps: reducing the warmup period improves the transfcrability of carly checkpoints.
44


Reviews:
[{"review_id": "EDTwCcAAyv", "reviewer": "Reviewer_yFWw", "summary": "This paper explores the reliability of small-scale experiments for predicting larger-scale training outcomes, which especially focuses on the data selection problem in the proxy model approach. A key finding is that learning rates can influence dataset rankings. To improve the accuracy of proxy model-based predictions, the paper suggests the use of sufficiently small learning rates. The paper provides both theoretical understanding and experimental evidence to support their findings.", "questions": "The major questions and suggestions are in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper offers valuable insights into the proxy model practice. The proposed idea (using small enough learning rates) is notably simple, which is a great advantage for real-world applications. \n- The paper provides rigorous experimental results and detailed findings, which contribute to the reliability of the study.\n- The paper includes various theoretical analyses that deepens the understanding of the underlying reasons of findings.", "weaknesses": "- While the Appendix includes some ablation studies on other hyperparameters, more explicit and detailed explanations regarding the impacts of these hyperparameters would greatly enhance the clarity and utility of the paper. It would allow readers to better understand the sensitivity and robustness of the model to different configurations.\n- Further discussion on how the observed findings might change when applied to different training strategies, such as reinforcement learning vs supervised fine-tuning, could significantly broaden the paper’s scope and strengthen its overall conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Crk8mddGSX", "reviewer": "Reviewer_Uruq", "summary": "This paper investigates the reliability of using small proxy models that use fix sets of hyperparameters to to guide data curation decisions for large-scale LLM training. The authors identify a critical fragility in current practices: dataset rankings from proxy models that rely on fixed set of hyperparameters to train on each datasets may not reveal correctly the utility of each dataset. Instead, the hyperparameters used to evaluate the utility of each dataset should be optimized for each dataset. The paper also identify that learning rate is the most important hyperparameter affecting the proxy models' dataset selection. They propose training proxy models with \"tiny\" learning rates (10⁻⁵ to 10⁻⁶) to improve cross-scale transferability, supported by theoretical analysis on random feature models and comprehensive experiments across 23 data recipes.", "questions": "Also, how are other factors such as dataset sizes, models type, and type of proxy model versus the actual model affect this ranking decision. I dont expect you to run too experiments but further discussion is appreciated.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written. Addresses a real challenge faced by AI labs where data teams must make expensive curation decisions based on small-scale experiments. The demonstration that minor learning rate variations can completely flip dataset rankings (Figure 3) is compelling and well-presented. The tiny learning rate solution is simple to implement without requiring architectural changes or complex procedures.", "weaknesses": "Only single-epoch training (multi-epoch is increasingly important as data becomes scarce). I am not sure if in pratice, people only use single-epoch training to evaluate the utility of datasets. Also, of course, hyperparameter tuning on each dataset is the best to evaluate the utility of a dataset, however, it can be very costly to do in practice. Furthermore, smaller learning rate may make it slower to reach optimum point, which may affect the correct ranking of datasets if trained on single epoch. I feel this point is not addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WgvdXL1kzD", "reviewer": "Reviewer_MjAE", "summary": "The paper addresses how small \"proxy\" models can reliably select pretraining data for large-scale models. It identifies a key flaw in standard practice: proxies are trained with fixed hyperparameters, while large models are tuned for the selected data. The authors demonstrate this makes proxy-based dataset rankings highly fragile, even reversing with minor learning rate adjustments. The primary contribution is a new objective—to find the dataset with the best optimally-tuned performance—and a simple solution: train proxy models with a \"tiny\" learning rate. This \"patch\" is shown to be effective, achieving >0.95 rank correlation with tuned, large target models across 23 data recipes. The authors provide theoretical intuition based on first-order alignment and a proof for random feature models.", "questions": "1. On the Theory Gap: The theory in Section 5.2 and Theorem 2 appears to only prove a within-scale result (that a tiny-LR RFM's ranking matches an infinite-width RFM's ranking). This supports Figure 4(a), but not the paper's main cross-scale claim (Figure 5a). Can you comment on this gap? Do you have any theoretical argument that actually bridges the proxy-to-target scale gap, or is the RFM analysis purely for intuition about the tiny-LR $\\rightarrow$ optimal-loss link?\n\n2. On Gradient Alignment: Your intuition relies heavily on \"first-order gradient alignment\" being preserved across model scales. Do you have more direct evidence for this? For instance, did you try to measure the gradient alignment term ($\\nabla l_{val}(\\theta)\\cdot\\nabla l_{i}(\\theta)$) for different datasets at the 125M scale and the 1B scale (at initialization or after short training) and check if their ranking is also preserved? This seems like a more direct test of your hypothesis.\n\n3. On Multi-Epoch Training: Your work is restricted to a single epoch. How do you speculate these results would change in a multi-epoch regime? One could imagine that over multiple epochs, the \"higher-order\" effects that you claim scramble rankings would have more time to accumulate, even with a tiny LR. Does the \"patch\" still work when models are trained for 2, 4, or 10 epochs? This seems like a critical question for real-world adoption.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear Problem, Simple Solution: The paper identifies a relevant flaw in a common workflow (using fixed-HP proxies). The proposed solution (\"use a tiny LR\") is straightforward and easy to implement.\n\n2. Helpful Empirical Validation: The paper provides a valuable set of experiments across 23 data recipes and 3 model families . Crucially, the authors use the correct (and expensive) ground truth: large models with per-dataset hyperparameter tuning. The results in Figure 2 and 5 clearly show the method's effectiveness .\n\n3. Helpful Conceptual Reframing: The paper's reframing of the data selection objective (Section 3) is a useful perspective for the community. It correctly points out that we should optimize for a dataset's potential after tuning, not its performance in a fixed setting.\n\n4. Supporting Ablation Studies: The appendices show that the tiny-LR method appears robust to other hyperparameter choices like batch size, weight decay, and token-per-parameter ratios, which helps isolate the learning rate as a uniquely sensitive factor .", "weaknesses": "1. Gap Between Theory and Claim: The paper calls its solution \"theoretically-grounded\", but the provided theory (Section 5.2) does not fully support its main cross-scale claim. The Random Feature Model analysis (Theorem 2) supports a within-scale claim (a proxy's tiny-LR loss predicts its own optimal loss) , but does not prove the paper's central thesis (that a proxy's loss predicts a target model's optimal loss). This disconnect undermines the claim that the solution is well-understood from first principles and makes it feel more like an (albeit effective) empirical heuristic.\n\n2. Significant Scope Limitation (Single-Epoch): The paper's findings are restricted to single-epoch training. While this is a clean setting for a study, it is a major departure from many modern (especially Chinchilla-optimal) multi-epoch training regimes. The authors acknowledge this as future work , but it remains a significant open question whether the \"tiny-LR\" advantage holds when long-term optimization dynamics, data ordering, and sample repetition are in play. This limits the currently demonstrated applicability of the patch.\n\n3. Lack of Mechanistic Evidence: The paper proposes \"gradient alignment\" as the intuition for why this works , but this hypothesis is not empirically tested. The paper would be much stronger if it provided direct measurements of the first-order (alignment) and higher-order terms across scales to show the former is stable while the latter is not. Without this, the \"why\" remains speculative, and the contribution is primarily an empirical observation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gkUa5bizkX", "reviewer": "Reviewer_Vv1t", "summary": "In the context of dataset curation, this work examines the situation where small proxy models are used to evaluate different datasets, in order to train a larger target model downstream for the best possible performance. The work finds that proxy models that are not hyperparameter-optimized are poor predictors of downstream performance. The work proposes a simple solution - train with smaller learning rates of 1-2 orders of magnitude less than standard LRs, as this makes the dataset rankings of the proxy models consistent with changes to the target model's performance given each dataset. The intuition and theory behind this solution is that small learning rates cause training to be dominated by the first order component of the gradient which optimizes for alignment of the model between the train and validation data. Using these small learning rates, experiments show that the dataset rankings given by small GPT proxy models are consistent with a) rankings of the proxy models with tuned hyperparameters, and b) rankings of a larger target LLM.", "questions": "How many replicates are in each of the experiments? A concern I have is whether some of the results (e.g. figure 3) are due to random chance from single runs. For some experiments this is clearly not happening, e.g. when computing correlation over multiple models or data recipes, but nevertheless it would be helpful for the authors to clarify when this may be an issue.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem of dataset curation is highly relevant, and the main result (that dataset ranking is sensitive to hyperparameters) is quite intriguing. The proposed method is simple but backed up by a theoretical arguments and experimental evidence. The paper is clearly presented and the intuition is made easy to follow.", "weaknesses": "The main question for me is whether the results are specific to large language transformers, or more troubling, only the GPT and Pythia models in the experiments. Given that the work already covers a lot of ground, I think instead of trying to generalize to other domains (vision especially) it would be best if the authors qualify their language and restrict their findings to the language domain specifically. As for generalizing from the specific models considered, although training is costly, it might be more convincing to have another target model that is somewhat different than the ones considered in the experiments. A weaker but cheaper experiment would be to show correlation between smaller-scale models of different types.\n\nThere is another question more for relevance: how large are the performance differences between different datasets? If they are quite small then the significance of the findings is not only diminished, but the findings themselves may again not generalize to real-world situations, e.g. if evaluating datasets with much larger differences in quality. Here I suggest a potential ablation - the authors could add some artificially bad datasets (e.g. by purposefully duplicating data) to demonstrate that their findings continue to hold outside of the narrow performance range of the datasets in this work.\n\nMinor suggestion: figure 5 and 6 could be swapped as it seems more natural that way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)