Rethinking Data Curation in LLM Training: Online Reweighting  Offers Better Generalization than Offline Methods
Authors: Wanru Zhao
Yihong Chen
Yuzhi Tang
Wentao Ma
Shengchao Hu
Shell Xu Hu
Alex Iacob
Abhinav Mehrotra
Nicholas D. Lane
Venue: ICLR 2026 Poster (2026)
Keywords: Data Selection
Data Mixing
Data Curation
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Data curation is a critical yet under-explored area in large language model (LLM) training. Existing methods, such as data selection and mixing, operate in an offline paradigm, detaching themselves from training. This separation introduces engineering overhead and makes the curation brittle: the entire pipeline must be re-run under model/task shifts. Moreover, offline methods alter data size through hard filtering or resampling, often sacrificing data diversity and harming generalization. We propose to rethink data curation as an online reweighting problem, where sample importance is dynamically adjusted during training via loss weighting rather than static pre-processing. Specifically, we introduce ADAPT (Adaptive Data reweighting for Pretraining and FineTuning), a dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity-based quality signals, without changing the number of training samples. Unlike offline methods that enforce a static data distribution, ADAPT acts as an implicit curriculum learner, progressively shifting focus from coarse-grained patterns to fine-grained semantic distinctions as the model evolves. Experiments on both instruction tuning and large-scale pretraining show that ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross-benchmark generalization under equal FLOPs.

Published as a conference paper at ICLR 2026
RETHINKING DATA CURATION IN LLM TRAINING: ONLINE REWEIGHTING OFFERS BETTER GENERALIZATION THAN OFFLINE METHODS
Wanru Zhao", Yihong Chen', Yuzhi Tang"t, Wentao Ma Shhnnghao Hu
Shell Xu Hu Alex Iacob', Abbinay Mehrotra', Nicholas D. Lane
1 University of Cambridge, OATML, University of Oxford,
3 University of Toronto, "Shanghai Jiao Tong University, *Samsung Al Center
ABSTRACT
Data curation a critical yet underexplored area in large language model (LLM) training. Existing methods, such data selection and mixing, operate in offline paradigm, detaching themselves from training. This separation introduces engi- neering overhead and makes the curation brittle: the entire pipeline must be re-run under model/task shifts. Moreover, offline methods alter data size through hard fil- tering or resampling, often sacrificing data diversity and harming generalization. We propose to rethink data curation as an online reweighting problem, where sample importance is dynamically adjusted during training via loss weighting rather than static pre-processing. Specifically, we introduce ADAPT (Adaptive Data reweighting for Pretraining and Fine' Fine Tuning). dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity- based quality signals, without changing the number of training samples. Unlike offline methods that enforce a static data distribution, ADAPT acts as an implicit curriculum learner, progressively shifting focus from coarse-graines pat- terns to fine-graines semantic distinctions as the model evolves. Experiments on both instruction tuning and large-scal pretraining show that ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross- cross- ben.hmarkchaaakeee generalization under equal FLOPs.
https://ggituub..om////aa///////
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks (Brown al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a), yet their generalization usu- ally hinges on the quality of the training data, including its diversity and mixing (Hoffmann et al., 2022; Kaplan et al., 2020a). While some recent efforts seek to circumvent the data quality issue and improve generalization through architectural interventions (Peyrard et al., 2022; Chen et al., 2023c; Bengio et al., 2025), most pragmatic practices still rely on curating the "right" dataset for the model and the task. Data curation deciding what data to keep and how to weight it - has thus become cornerstone of large language model development. On the other hand, the process of data curation has largely remained secret sauce with pipelines relying on opaque heuristics, ad-hoc engineering choices, or costly trial-and-error, rather than principled frameworks.
Current approaches data curation largely follow offime paradigm, falling into camps: data selection, which keeps a subset of valuable examples, and data mixing, which adjusts sampling fre- quencies to rebalance distributions. Despite their differences,, both approaches typically follow multi-stage pipeline: (I) train or extract features with proxy model, (2) compute quality signals as selection or weighting criteria on a validation set, and (3) retrain the main model on the curated data. While effective in some cases, this paradigm suffers from fundamental drawbacks (Sec. 4) due to its static and subtractive nature. First, it ignores training dynamics: the value of a data sample is not
"Corresponding to: Wanru Zhao (w2341 @ cak.ac.uk) Equal contribution.

---

Published as a conference paper at ICLR 2026
static but changes the model learns. Offline selection freezes this value based on a proxy model, creating a misalignment with the training model's evolving needs. Second, it compromises diver- sity: by relying on hard filtering to select a fixed subset, offline methods risk discarding the broad data distribution essential for robust generalization. Consequently, the resulting subsets often overfit to specific benchmarks and are brittle, which requires the entire pipeline be re-run whenever the model architecture or corpus changes.
In contrast, we propose a paradigm shift to online reweighting, which fundamentally differs from offline selection not just in when the signal is computed, but also in how the data is used. Instead of hard-filtering a subset based on static scores, online reweighting retains the full dataset to preserve diversity, but dynamically modulates each sample's 's contribution via loss weighting, This preserves data diversity compared to offline's "hard cuts" while intelligently controlling each sample's "step size" in parameter updates to adapt to evolving model states.
To solve the limitation of offline data curation, we propose a unified online reweighting framework, ADAPT (Adaptive Data reweighting for Pretraining and Fine' Tuning), which aligns training sam- ples with a validation set via similarity-baset weighting. Our method incurs nearly zero additional overhead yet achieves superior cross-benchmark generalization. Under a unified evaluation proto- col, it consistently outperforms both offline sclection/mixing strategies and prior online reweighting methods, across large-scal scale pretraining and instruction tuning setups.
Our contributions. This work makes the following contributions:
1. ? Formalization and unification of dataset curation pipelines. We present unified view of data curation, showing that data selection, mixing, and balancing can all be expressed as special cases of data reweighting. This perspective shifts the focus from offline pre-processing dy- namic, in-training adjustment of data weights. Which offers a fair protocol we propose to view and evaluate them through a unified lens of online data reweighting.
2. d Online sample-level reweighting method. We introduce a new online global reweighting algorithm that is effective under this unified formalization and protocol. It employs an adaptive per-sample learning rate guided by sample quality signals, without explicitly altering the dataset size, while dynamically adjusting sample weights during training.
E. Better performance and generalization across tasks. On both instruction tuning and large- scale pretraining experiments, our framework achieves up to up to 7.2% higher accuracy per FLOP than offline baselines including LESS (Xia et al., 2024), DoReMi (Xie et al., 2023b), and Reg- Mix (Liu et al., 2024), and other online reweighting method (Sow et al., 2025), and demonstrates 6. %% stronger out-of-domain generalization across benchmarks.
2 BACKGROUND AND RELATED WORK
2.1 DATA CURATION FOR LARGE LANGUAGE MODELS
Researchers and engineers usually rely on web crawls to gather large datasets for training large lan- guage models (Brown, 2020; Computer, 2023; Penedo et al., 2024; Tang et al., 2024; Bai et al., 2023; Kandpal et al., 2025). However, these crawls often include a substantial amount of low-quality or irrelevant content, which makes data curation necessary to build high-quality training sets. Most data curation efforts focus on methods for improving model performance (Raffel et al., 2020; Brown, 2020; Rae et al., 2021; Penedo et al., 2023; Soldaini et al., 2024), including filtering by language (Raf- fel et al., 2020; Xue et al., 2020), heuristic-basee filtering (Gao et al., 2020; Rae et al., 2021; Penedo et al., 2023; Soldaini et al., 2024), quality filtering (Du et al., 2022; Xie et al., 2023c), data dedupli- cation (Lee et al., 2021) and mixing (Xie et al., 2023a; Soboleva et al., 2023; Albalak et al., 2023). However, current approaches curating such datasets generally ad-hoc. We aim to develop principled and automated method for data curation that can also unify different processing stages.
Pretraining data curation. Several recent studies (Xie et al., 2023a; Chen et al., 2023b; Fan et al., 2023: Thakkar et al., 2023) have explored various reweighting techniques to enhance the generaliza- tion and efficiency of language models pretraining. For instance, Xie et al. (2023a) and Fan et al. (2023) are aiming finding the optimal mixture of pretraining corpora to enhance performance
2

---

Published as a conference paper at ICLR 2026
across domains. Chen al. (2023b) propose ordered skill learning method for data selection measuring how effectively it teaches interdependent skills for continual pretraining and ane-tuning. Although effective, these works are aiming at the group level, whereas work explores reweight- ing at the sample level, offering fine-grained control during model training dynamics. Instance-level reweighting has been used in pretraining settings of LLMss (Chen al., 2024; Jiang al., 2024), where each sample per mini-batch is weighted over how individual samples are treated based on their loss values. In contrast, our work studies the effects of various adaptive learning rate with dif- ferent quality signals considering the model states to enhance both performance and generalization for LLMs pretraining and fine-tuning regimes.
Instruction data curation. Research has demonstrated that prioritizing data quality and diver- sity over quantity more helpful for instruction-followiii capabilities (Cao et al., 2023; Chen et al., 2023a; Bukharin & Zhao, 2023; Du al., 2023; Liu al., 2023; Li et al., 2023). Instruction tun- ing data includes base- datasets curated from traditional NLP tasks (Wang et al., 2022; Sanh et al., 2022; Wei al., 2022a; Longpre al., 2023), and open-ended datasets (Taori et al., 2023; Conover et al., 2023; Köpf et al., 2023; Xu et al., 2023; Mukherjee et al., 2023; Zhou et al., 2023; Ding et al., 2023). In work,
focusing adaptively adjust the data weight during instruc- tion tuning without explicitly reduce the number of instruction data samples and obtain model with generalization.
2.2 COMMONLY USED QUALITY SIGNAL FOR DATA CURATION
Lexical Similarity (BM25) (Silva & Barbosa, 2024) quantifies term term-based overlap between train- ing and validation data through sparse retrieval scoring:
(x)
Σ
BM25(x,
(1)
DDvll
PED
This metric captures surface-level textual similarity without semantic understanding.
Semantic Similarity (Embedding) (Rubin et al., 2021) measures dense representation alignment using pretrained encoders:
SEmbed (x)
Σ
cos (6(x). (())).
vall
Deai
where (f-) denotes a frozen embedding model. This approach captures semantic proximity beyond lexical overlap.
Distributional Alignment (Perplexity) (Antonello et al., 2020) evaluates likelihood under ref-
erence language model fo:
(x llog Pa (x).
Lower perplexity indicates stronger distributional alignment with the reference corpus.
Gradient- Basdd Influence (Xia al., 2024) estimates training utility through first-order approx- imation of validation loss reduction:
SGrad (x) (V (x; 0p), V Fa((Dvat:o)))
where An denotes proxy model. This metric directly quantifies how training on x influences performance the validation set Dval-
3 UNIFIED FORMALIZATION OF DATA CURATION
Setup. Given two datasets: the train set Durin and the validation set Dval- Usually, the size of the validation set much smaller than the train set. When Dval sampled from the distribution of downstream test data, the validation dataset Dral is considered as in-domain. The train dataset D samples. This the case when web
on the other hand consists of both in-domain and crawling is used to collect training data from the whole internet. Our goal is to subsample weight the train set Dirain under the guidance the validation set D so that the model training is less affected by the out-of-domain samples.
3

---

Published as a conference paper at ICLR 2026
3.1 DESIGN SPACE DATA QUALITY SIGNAL
In Sec. 2, we present a unified framework for data quality assessment where scoring function takes the form s(x) s(x; 0, Dval) that assigns a quality signal for each data example x, which optionally depends on (proxy) model 0 and the validation set Dval.
3.2 OFFLINE DATA CURATION
The scores are employed once before training (e.g., with proxy model fo and fixed validation set val) The resulting weights [w(x))xe are fixed and used make decision pass which training examples to the real training stage.
Data Selection (sample-level binary weights) Data selection (or filtering) removes part of the training corpus before pretraining begins. It keeps data examples with quality signal above certain threshold
D
(x Drain s(x) T).
From a data reweighting perspective, filtering data points is equivalent to assigning binary weights of the form w(x) = [[s(x) 2 r] E 1}.
Data Mixing (domain-level fractional weights) Data mixing can be seen coarse- coarse-grained data reweighting method that operates at the domain level. other words, data points within the same domain receive the same quality score. Pulling all domain scores {sa} Edoomains and normalize them to obtain data mixing probability for each domain
g(sa)
W/ Σd 8 (sa))
where g transform the score to non-negative with common choice being g: s exp(s). As such, wa determine how much of total training budget B is allocated to each domain: Ba = w B.B. Alternatively, training can be implemented by sampling domains with probability wa (probability mixing) by assigning fixed quota Ba (quota mixing). Since Wa is decided before training, mixing is an offline operation that alters the effective number examples each domain contributes.
3.3
ONLINE DATA CURATION
In this setting, the scoring function depends on the evolving model state 0, and, in some cases, on a dynamic validation set. The resulting weights (x)) x evolve throughout training, adjusting the gradient contributions of examples while preserving the full volume of the training set D
Data Reweighting (sample-level fractional weights) Data reweighting assigns normalized weights examples while keeping the training set size unchanged. Similar to Sec. 3.2, the qual- ity score for example x is transformed to a non-negative weight: w(x) = g(s(x),, which scales their contribution the loss:
L' Σ w(x) L(0:x).
Z=
Σ w(x)
Equivalently, from a stochastic gradient descent perspective, the stochastic gradient w.r.t. x scaled w(x)
by 2 which plays a role of per-sample learning rate in addition to the global learning rate ry:
0,+1 0, - w(x) V vol(O,;x).
Since the total number of training examples remains unchanged, only their relative contribution to parameter updates is modified, data reweighting is inherently an online method.
3.4 UNIFTED EVALUATION OF TOTAL FLOPS OFFLINE vs. ONLINE)
The unification enables compare different methods under the same FLOPS calculation frame- work. We evaluate curation methods under a cost-aware metric that accounts for all computation spent obtain and use the curated data. Let B denote training budget (tokens update steps).
4

---

Published as a conference paper at ICLR 2026
Total FLOPs for offline data curation Offline curation modifies the amount of data that partici- pates in training before training starts. Its total compute is
Foll
prep fD:
F Faiin Frain( D', B*)
data scoring preprecessing
model training on the sampled subset
where (i) Foll
prep includes any corpus-wide scoring, proxy-model passes, retrieval or filtering necessary construct D': (ii) D' S is the retained subset (for selection), or an effective subset induced for data mixing: (iii) B' is the effective training budget after curation. By construction, offline selection uses binary example weights (keep/drop), and offline domain mixing changes per-domain sampling rates, thereby reducing or reallocating the volume of data seen during training.
Total FLOPs for online data curation Online curation does not change the amount of training data; instead, modulates each example's 's contribution during training. Its total compute is
Fon
F Firain D. B)
FoR metrics
(6)
total
madel training the full corpus
an-the fly features
where Fon
merrics denotes lightweight, in-training computations needed to obtain scores (e.g., using cur- rent logits, per-example loss, or gradient norms). Because scoring is amortized inside the training loop and no data removed, Fon metrics is typically modest relative to Furain-
Conversely, online reweighting maintains the full training signal while adding only minimal in-loop metrics computation, providing clearer accuracy-compute trade-off when measured by rotal FLOPs.
4 REVISITING OF OFFLINE DATA SELECTION
In this section, based on the evaluation protocol derived from the unification above, we revisit the accuracy-FL.C FL.PP trade-off of existing offline data selection methods.
Average MMLU vs FLOPs
Average BBH vs FLOPs
210
Method
416
15.
oTher DACS
brtentre
45.
o7i Randar
435
-445
455
33.3
41.
419
31.2
475
FLOPk (lg socie)
RLOPs (log scalel
(a) In-domain evaluation: MMLU validation set
(b) Out-of-domaii evaluation: MMLU validation set BBH benchmark.
MMLU benchmark.
Figure Efficiency-accuracy tradeoff of different data selection methods and full finetune method under our proposed total FLOPs metric. Selection methods (e.g.. LESS) appear competitive under in-domain evaluation, but their advantage vanishes or even reverses in out-of-domain settings. In contrast, (full sft) remains consistently strong across domains.
Observation. Figure illustrates the limitation of offline data selection methods: Overfitting to the validation benchmark. Model trained on offline selected data with MMLU (Hendrycks et al., 2021) as validation set performs well on the MMLU benchmark (Figure la), but the same model generalize poorly BBH benchmark (Suzgun et al., 2022) (Figure lb). This issue especially severe when using LESS (Xia et al., 2024). This reveals that offline methods often overfil to the chosen validation task, lacking a true generalization. However, we observed that vanilla full dataset training demonstrate more stable performance in both benchmark. One possible reason is that official data curation which we directly change the number of data involved in the training by repetition would cause the model to replace generalization ability with memorization (Hernandez et al., 2022).
In summary, the offline paradigm has severe limitations, being neither cost-effective nor generalizable. This motivates to explore online alternative, which may alleviate memorization while retaining more data to enhance generalization. In the next section, we leverage these insights to design an online reweighting method.
5

---

Published as a conference paper at ICLR 2026
5 ADAPTIVE LEARNING RATE FOR ONLINE DATA REWEIGHTING
The limitations of offline data selection suggest that different paradigm is needed: that is gen- eralizable, and adaptive to the evolving state of the model. To this end, we propose an online data reweighting framework that is also suitable for data selection, data mixing with unified formal- ization. Unlike offline methods that commit to a fixed subset before training begins, our framework dynamically adjusts data weights the model learns, it naturally adapts new model states without re-running the data preprocessing and training pipeline from scratch.
5.1 SER-LE LEARNING RATE UPDATE
cast data selection and mixing as online data reweighting. At training step given a minibatch B, D, we update model parameters
8 = 8, -1 Σ w. (,(oVo(((o(t)).i
where wr( 2 O denotes dynamic weight assigned to sample i. Here, (i) directly scales the gradient contribution of sample i, and thus acts per-sample learning rate multiplier: larger weights amplify the effective step size on informative examples, while smaller weights downweight less useful ones.
In our framework, weights are derived from scoring functions s, (v. i) that compare a training example i with validation/query points v E Vall- These scores are aggregated across v, normalized within the current batch pool, and transformed by smooth gating function to produce the final weights w, (i). To ensure stability, weights are clipped to prevent excessively large effective learning rates and avoid gradient explosion. We now describe our score functions.
5.2 MODEL-AGNOSTIC PER-SAMPLE LEARNING RATE UPDATE
ADAPT-BM25 quantifies term-hased overlap between training and validation data through sparse retrieval scoring:
SBM25 (x)
Σ
BM25(x, v).
(8)
Dvall
This metric captures surface-level textual similarity without semantic understanding, We use the stan- dard BM25 score SBM25 (v i) over sparse token matches between query and example Aggregated scores are normalized to obtain target weights.
5.3 PER-SAMPLE LEARNING RATE UPDATE WITH MODEL STATES
ADAPT measures alignment using the model's 's own dense representations rather than a frozen en- coder (Ivison et al., 2025). For input.x with last-layer hidden states {hi) compute a position- weighted mean pooling:
(xx) = Σ whh
W EJ
(9)
where later tokens receive higher weights to counteract the causal mask bias decoder-only models. We then define the similarity score:
Σ cos (Ø(x), (vv)..
SADAPT(X)
(10)
Dvall
PE Deal
Equivalently, we instantiate a representationn-based scorer (A, cos (d((), d(i), where ((-) is the weighted hidden representation.
We use sigmoid function with temperature scaling to produce absolute weights that are independent of the batch composition:
w, (i) = SADAPT((x)
(11)
max(T, I (xpp-SSsDAr(Et))max(t..r

---

Published as a conference paper at ICLR 2026
where a--) denotes the sigmoid function, 7 > O is a temperature hyperparameter (default 1.0), and > 0 is a small constant (e.g., c = 10 used for numerical stability. The temperature parameter controls the steepness of the sigmoid: larger values produce flatter weight distributions, while smaller values create sharper distinctions between high and low similarity samples.
Global Weighting via Sigmoid: The sigmoid transformation maps similarity scores to absolute weights in the interval [0, IJ without requiring normalization across samples in the batch. This de- sign choice ensures that the weight assigned to each sample x; depends solely on its similarity score SADAPT (x,) relative to the anchor set, rather than its rank within the current mini-batch. Conse- quently, sample with a given similarity score receives the same weight regardless of whether appears in a high-quality or low-quality batch, making the weighting mechanism robust to batch- level variations in data quality. This contrasts with normalized weighting schemes (e.g., softmax normalization) where weights are relative to other samples in the same batch, potentially amplifying or dampening effects based on batch composition.
We apply L2 normalization to embeddings (xx) and d(v) before com- puting cosine similarities to ensure scale-invariant similarity measurements. Specifically, for each embedding vector, we compute (xx) c ((x) ((x)/mx(( max((16(xx1122+ e) where prevents division by zero. This normalization step is distinct from weight normalization and serves to standardize the embed- ding space for similarity computation.
Embedding Normalization:
Online Validation Embedding Updates: To maintain alignment with the evolving model represen- tations, anchor embeddings vv are refreshed every R training steps via forward passes on the current model parameters 0,. This online update mechanism ensures that similarity scores reflect the model's current representation space rather than outdated embeddings from earlier train- ing stages. The refresh interval R is a hyperparameter that balances computational efficiency with representation freshness. The complete algorithm is summarized in Algorithm
While gradient-base based methods (c.g.. LESS) directly measure influence, gradients significantly fluc- tuate during early training and are computationally expensive to compute frequently. In contrast, semantic embeddings provide a smoother, more stable signal of relevance that evolves consistently with the model's 's representation space, offering a robust guide for reweighting without the noise of instantaneous gradient estimates.
6 EVALUATION
6.1 EXPERIMENTAL DESIGN
Instruction Tuning Models and Data. use LoRA (Hu et al., 2021) to fine-tune the base model LLAMA-2-7B (Touvron al., 2023c). Following the experimental setup in Wang et al. (2023), we use the instruction tuiing datasets including FLAN V2 (Longpre et al., 2023), CoT (Wei et al., 2022b), DOLLY (Conover et al., 2023) and OPEN ASSISTANT (Köpf et al., 2023). The datasets do not contain any obvious in-domain data for the target queries. We evaluate our method on MMLU (Hendrycks et al., 2020), multiple-choice dataset spanning 57 tasks and BBH (Suzgun et al., 2023) from BIG- Bench selected to evaluate reasoning capabilities. For MMLU, we report 5-shot accuracy; for BBH, we report 3-ohot exact match score. Appendix G contains more details on the training hyperparameter, Appendix D and E contain more dataset details.
Instruction Tuning Baselines. We introduce instruction finetuning baselines in Sec. 2.2.
Pretraining Models and Data. We adopt Tinyllama architecture (Zhang et al., 2024) with 120M parameters, with FlashAttention (Dao et al., 2022) and Lit-GPT (Lightning 2023). More details about the training settings can be found at Appendix F. Following prior works (Touvron et al., 2023b; Zhang et al., 2024; Wettig et al., 2024; Xie al., 2023b), we employ SlimPajama (Touvron et al., 2023b; Computer, 2023) as the text corpus, which is specifically curated for pre-training LLMs. All selections are performed on about 590M training files of SlimPajama, processed with Llama tok- enizer (Touvron et al., 2023b). We evaluate our method on diverse set of 15 downstream bench- marks, following common practice in prior work such as RegMix (Liu et al., 2024). These tasks span a wide range of realistic settings, including: ARC-E (Clark et al., 2018), ARC-C (Clark et al., 2018), COPA (Sarlin et al., 2020), HellaSwag Zellers et al. (2019), Lambada-S (Paperno et al., 2016), Lambada-O O (Radford et al., 2019), LogiQA (Liu et al., 2020), MultiRC (Khashabi et al., 2018), Open-
7

---

Published as a conference paper at ICLR 2026
BookQA (Mihaylov al., 2018), PIQA (Bisk al., 2020), QQP (Wang, 2018). RACE (Lai et al., 2017), SeiQ (Welbl et al., 2017), Social IQA (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021). The reported aceuracy in table 2 is measured in the 0-shot setting scored using the Im-eval-harnes evaluation framework (Gao et al., 2024).
Pretraining Baselines. We compare ADAPT with Uniform selection and existing file selection methods for LLM pre-training, including Doremi (Xie et al., 2023b) and RegMix (Liu et al., 2024). ADAPT requires an anchor set typically consists of examples in the evaluation distribution. To con- struct this set, sample 50 validation examples from each of eight evaluation benchmarks: ARC-C, COPA, Lambada, MultiRC, PiQA, RACE, SeiQ, and Social IQA. For Doremi and Regmix we use the domain weights in (Lu al., 2023) as the selection ratio of text samples in different domains in our experiment. For Uniform we use the same ratio to sample from each domain. We also com- pare LinUpper (Sow et al., 2025): an online sample reweighting strategy where the sample weight is proportional to the normalized loss but is capped at a predefined a value, ensuring that outliers do not dominate the training process. Due to the large cost of pretraining, for each method we sam- ple 9B unique tokens from SlimPajama, and train for a total budget of 50B tokens, i.e., train for approximately 5.6 epochs.
6.2
RESULTS FOR INSTRUCTION TUNING
Average MMLU vs FLOPs
Average BBH vs FLOPS
518
Method
45.9
095
44.5
435
ottine Fronnrt
44.9
46.5
oftine PMt.
ottine FSG
450
456
Randen
479
455
ROAPT GM22
SDAPT
Ful SFT
410
6855
415
45.
300
416
36.5
415
325
455
378
19
19
FLOP% (log scale)
FLOPs (log
(a) In-domain evaluation: MMLU validation set
(b) Out-of-domain evaluation: MMLU validation set
MMLU benchmark.
BBH benchmark.
Figure 2. accuurcyyyy tradeoff. (a) While offline BM25 is efficient low FLOPs, it saturates early. ADAPT dominates the high- perffrrneoreeeee regime (higher FLOPs), avoiding the early plateauing of heuristics. (b) Cru- cially, LESS overfits to the in-domain task, showing regression on OOD tasks (BBH), whereas ADAPT consistently improves generalization on the Pareto fronticr.
As shown in Figure our proposed online data reweighting method consistently outperforms existing approaches under comparable computational budgets:
Table I. Generalization results under different validation-test vxiidation-eesscooiiuuaaannnnn
Method
MML.U/al) MMI.U(test)
MMI.U/val) BBH(test)
BM25
48.7 0.9
42.3 0.8
Embedding
47.0 :: 0.6
40.1 0.5
LESS
50.2 + 0,5
38.7 + 1.5
PPL
46.2 1.1
40.9 0.9
Random
43.5 0.3
38.4 1.0
Full Dataset SFT
49.7 0.2
44.4 0.3
ADAPT-BM25
50.9 0.6
43.7 + 1.2
ADAPT
50.7 = 0.7
44.8 1.3
Comparison with offline data selection methods. Across all baselines in both Figure 2a and Fig- ure 2b, our approach consistently lies on the Pareto frontier, demonstrating the best trade-off between computational efficiency and final accuracy. At the same FLOPs, method achieves substantially higher accuracy than the offline data selection SoTA method LESS, To reach the same performance level, our approach requires significantly fewer FL.OPs than LESS. Moreover, after reaching this
8

---

Published as a conference paper at ICLR 2026
Table 2. Benchmark performance of TinyLlama-120M trained 50B tokens using Uniform, LinUpper, DoReMi, RegMix, and ADAPT.
Tasks
Uniform
LinUpper
DoReMi
RegMix
ADAPT-BM25
ADAPT
ARC-C
+7.75
18.77
18. 18.00+a.8t
18.60.0.06
18.3
+9.11
ARC-E
39.60+0.10
18.85
40. 70+0.87
41.04.0.21
39 39.44.0.00
39.06,0.54
COPA
63.0020.00
60.00w1.72
66.00
61 1..5
63. 23.20z0.40
64.00+1.32
HellaSwag
28.3940.00
27.90
28. 28.03
20.63
27.96.0.07
28.11
Lambada-O
24.68
23.6420028
22.38
24.82
16.19
24.63
Lambada-S
16.98
16.7910.91
16.30
18.30
18.43
18.07
LogiQA
20.4310.54
21.35
21.51
20.89
20.5220.69
21.6610.69
MultiCC
56.6810.00
10.74
48.99
56.70
56.5310.44
55.67
OoenBooKQA
15.80
14.00 10.18
15.20
18.20
15.40
14.20
PIQA
60.55+0.29
60.83+0.45
59.90 59.90+0.39
58.98+0.29
61.00-0.19
61.48+0.23
QQP
36.88+0.00
37.85+1.21
36. 46.84
36.83+0.00
36.82-n.00
36.81
RACE
27.85+0.14
26.41+0.18
26.41+0.01
27.85+0.00
27.39+0.35
26.60+a.49
SciQ
71.00
70.70
72.90
71.20
72.48
72.5021.37
SocialIQA
36.9020.03
36.8520.16
36.95
37.41
36.5220.47
37.05
WinoGrande
20.75
50.8320.09
49.6420.79
50.12
50.61
50.9920.00
Average (All) (1)
37.03 20.11
37.3220.15
37.97
38.05
38.00
Average (Unseen) (1)
31.98+0.09
30.5620.28
31.95
32.46.0.39
33.49
33.73
FLOPs overhead ())
4.92 1019 3.072 x 1018
1.0 10
<< 1.1 x 10
performance, further training with LESS leads overfitting and accuracy degradation, whereas on- line data reweighting continues to exhibit stcady performance gains. Notably, our online reweighting method using BM25 as quality signal (ADAPT-BM25) has better generalization than its offline selec- tion method (BM25) with the other configuration keep the same, which demonstrates the advantage of the online data curation.
Comparison with full-data fine-tuning. In Figure 2a and Table 1, our method also outperforms full- data fine-tuning under equal FLOPs, with the performance gap widening as training progresses. In Fig. 2b and Table. 1, our reweighting strategy that incorporates model state information (ADAPT) out- performs our fixed-state online reweighting (ADAPT- -BM5), highlighting the importance of adapt- ing to the evolving model state to better generalisation.
Generalization cross benchmark. As shown in Figure 2b, we further validate the generalization capability of models trained with ADAPT, where validation set to select the data, and evaluate the model trained on selected data on the BBH (Suzgun al., 2022) task. The results, also reported in Table 1, show that ADAPT achieves a better gen- eralization performance between benchmark compared to other offline data selection baselines and demonstrates comparable generalization with vanilla full dataset instruction tuning. This highlights a key strength of ADAPT: when new downstream tasks or benchmarks emerge, there is no need to repeat the entire data selection and model training pipeline. Over the long term, this substantially reduces FLOPs costs, making ADAPT highly practical for real-world applications.
MMLU (Hendrycks et al., 2021) as the
42505
05.00
20505
4.50
2001
Aoder Barly
(a) Distribution of effective selected examples
(b) Similarity distribution for different epochs
Figure 3. A summary the distribution of effective selected samples similarity scores. Both differences and changes show the proposed method's ability on capturing data features and adaptive learning.

---

Published as a conference paper at ICLR 2026
Data mixing and curriculum. We present the distribution of the effective selected examples for different methods in Figure 3a. Different approaches end up selecting very different amounts of data from each training subset. It demonstrats that online reweighting solution can also help with deciding proper data mixture. We also calculate the effective proportion of used data, by summing up weights of all data together, with the result of 0.501 in our experimental setting. In practice, this number would automatically adapt according to the overall quality of the training corpus we are selecting from.
We also presented Figure 3b. The shift in similarity distributions from epoch to epoch 2 reflects transition from collapsed, overly homogeneous representations toward more diverse and ane-grained embeddings. This diversification enhances generalisation by improving the model's ability to capture subtle distinctions and reducing over-reliance on coarse features. Notably, this process can be viewed as an instance of implicit curriculum learning: the model first clusters samples based on coarse, "eas- ier" patterns and progressively moves toward harder, fine fine-grained discrimination, thereby mirroring the principles of curriculum learning without explicit scheduling.
6.3 PRETRAINING RESULTS
As shown in Table 2, ADAPT outperforms all baselines in terms of average perfor- mance, achieving a 0.19% improvement over Uniform, 0.68% over DoReMi, and 0.97% over LinUpper. RegMix reaches comparable but slightly lower accuracy on average. Interestingly, LinUpper performs worse than Uniform sampling, suggesting that naive sample-level adjustments may not generalize well. Compared to Lin- Upper, which conducts normalization and calculates sample-level weights for each batch, our global sample-level reweight- ing solution demonstrates superior per- formance in handling unknown ratios mixed-quality corpora.
Table Benchmark performance of Tim-Llam--20M trained 100B tokens using Uniform and ADAPT.
Tasks
Uniform
ADAPT
ARC-C
17.75
18.86
ARC-E
40.66
40.45
CОРА
63.00
61.00
HellaSwag
28.35
28.75
Lambada-O
25.13
25.91
Lambada-S
18.47
20.14
LogiQA
20.28
21.66
MultiRC
55.98
56.44
OpenBookQA
16.00
15.00
PIQA
61.48
61.97
QQP
36.81
36.82
To examine the effect of larger training bud- gets, extend training to 100B tokens. As shown Table 3, ADAPT achieves higher benchmark performance than Uni- form on 11 out of 15 downstream tasks, with average improvement of 0.38%. Table 6 further shows that ADAPT con- sistently yields lower validation perplexity than Uniform at both 50B and 100B train- ing budgets. These findings demonstrate that ADAPT improves both validation perplexity and down- stream task performance consistently across compute scales.
RACE
27.37
27.46
SciQ
71.50
72.40
SocialIQA
35.77
37.05
WinoGrande
51.14
51.54
Average (All) (1)
37.98
38.36
Average (Unseen) (1)
32.51
32.67
CONCLUSION AND DISCUSSION
Our work unifies data selection, mixing, and reweighting under FLOPs-aware framework. We show that offline pipelines often incur significant cost via data preprocessing and faces issues with generalization. In contrast, online reweighting integrates seamlessly into training, adapts to model state, and improves efficiency without affecting effective data size.
Empirically, we demonstrate that ADAPT consistently outperforms both offline and online baselines across instruction tuning and pretraining setups. instruction tuning, ADAPT not only achieves higher accuracy under equal FLOPs but also exhibits stronger cross-benchmark generalization (e.g.. from MMLU to BBH), mitigating the brittleness of offline methods. In pretraining, ADAPT im- proves both downstream task accuracy and validation perplexity under 50B and 100B token budgets, underscoring its robustness across compute scales. We further discuss the implications and outline future directions in Sec. A.
10

---

Published as a conference paper at ICLR 2026
ETHICS AND REPRODUCIBILITY STATEMENTS
We adhere to the ICLR Code of Ethics. This paper focuses on comparing methodologies for data curation. To ensure fairness, evaluate online data reweighting and offline data mixing methods under a FLOPs-cquivalen, setting, We also rigorously test their performance in both instruction fine- tuning and pretraining scenarios to provide comprehensive and balanced comparison. During the preparation of this manuscript, we utilized large language models (LLMs) to assist with grammar correction and refinement of the writing. We provide all necessary details to ensure reproducibility of our work. The theoretical justification for our FL.OPs-quuvvleent data curation formalization is presented in Section with detailed FLOOP calculations in Appendix C. Implementation details and training protocols provided in Section 6.1, Appendix B, Appendix F, and Appendix G, while descriptions of the training data are given in Appendix D.
LCKNOWLEDGEMENT
We would like thank Colin Raffel, Zhihao Zhan, Ziqing Fan, Jonathan Hayase, Guilherme Penedo, Hynek Kydlíček and Loubna Ben allal for many stimulating and helpful discussions. This research was supported by the following entities: The Royal Academy Engineering via DANTE (a RAEng Chair); the European Research Council, specifically the REDIAL project; SPRIND under the com- posite learning challenge; Google through a Google Academic Research Award. Yihong acknowl- edges funding from the Horizon Europe grant No. 101213369 (DVPS).
REFERENCES
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models Workshop, 2023.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Man- sheej Paul. Perplexed by perplexity: Perplexity- based data pruning with small reference models, 2024. URL httph::t///// / //wpiv///vvv/v//////////////aav org/ /2d/ 20541.
Richard Antonello, Nicole Beckage, Javier Turek, and Alexander Huth. Selecting informative con- texts improves language model finetuning, arXiv preprint arXiv:2005.00177 2020.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Samyadeep Basu, Phil Pope, and Sohcil Feizi. Influence functions in deep learning are fragile. In International Conference Learning Representations, 2020.
Yoshua Bengio, Michael Cohen, Damiano Fornasiere. Joumana Ghosn, Pietro Greiner, Matt Mac- Dermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, et al. Su- perintelligennt agents pose catastrophic risks: Can scientist ai offer safer path? arXiv preprint arXiv:2502.15657, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence. volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. arXiv preprint 1rXiv:2311.14736 2023.
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High High-quality instruction data selec- tion for large language models. arXiv preprint axX1v:2307.06299 2023.

---

Published as a conference paper at ICLR 2026
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint ar arXiv:2307.088 08700 2023a.
Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christo- pher Re. Skill-it! a data-driven skills framework for understanding and training language mod- els. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https://openrewe wwwpanreveeennet/ffeee mewt/eorummdd-Iozoo0O00
Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. Take the bull by the horns: Hard sample-reweighted continual train- ing improves IIm generalization. arXiv preprint arXiv:2402. 14270, 2024.
Yihong Chen. Structure and Destructure: Dual Forces in the Making of Knowledge Engines. PhD thesis, UCL (University College London), 2025.
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifcoluwa Adelani, Pontus Stenetorp, Sebas- tian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. In Advances in Neural Information Processing Systems 36, 2023c.
Yihong Chen, Luca Franceschi, Xiangxiang Xu, Pontus Stenetorp. and Sebastian Riedel. Jet expan- sions: Restructuring LLM computation for model inspection. In The Fourteenth International Conference on Learning Representations, 2026. URL u6JLhOB05h.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 05457, 2018.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https:://gtthub.com//ott
/Redaajj//aaajaaeeeeee jaaPaajaaa--- -ataaaa...
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free Dolly: Introducing the world's first truly open instruction-tuned LLM, 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprini 2tt.:2335.143333 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mix-ureoffexperrs. In International Conference on Machine Learning (ICML), pp. 5547-5569. PMLR, 2022.
Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruc- tion tuning, 2023.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE: Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endledness Workshop, 2023. URL https:// openreview.o wwwnrrvvee.nee/fforoooe www.:/nrt//tttteeemm ddgprmmgP
Yarin Gal and Stephen Casper. Customizable ai systems that anyone can adapt bring big
opportunities-aaa and even bigger risks. Nature, 646(8084):286-287 2025.
12

---

Published as a conference paper at ICLR 2026
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXhv:2101.00021 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nel- son Elhage, Zac Hatfield-Dodee Tom Henighan, Tristan Hume, et al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya. Trevor Cai, Eliza Ruther- ford, Diego de Las Casas. Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXi::2/06.09685 2021.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.00 010007 2025.
Chunyang Jiang, Chi-min Chan, Wei Xue, Qifeng Liu, and Yike Guo. Importance weighting can help large language models self-improve. arXiv preprint arXiv:2408.09844 2024.
Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: Gen- erative inference of large language model over heterogeneous environment. arXiv preprint arXiv:231..11514 2023. 2023.
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki. Demystifying cost-efficiency in Ilm serving over heterogeneous gpus. arXiv preprint ar XXi:25502.00722 2025a.
Youhe Jiang, Fangcheng Fu, Wanru Zhao, Stephan Rabanser, Jintao Zhang, Nicholas D. Lane, and Binhang Yuan. Cascadia: An efficient cascade serving system for large language models, 2025b. URL https: arxxiv.vvvv org/ arg/abs/2ee ab5/ 04203.
Youhe Jiang, Ran Yan, and Binhang Yuan. Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment. arXiv preprint 2tt.:22220.779333 2025c.
Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, A Feder Cooper, Aviya Skowron, et al. The common pile v0. 1: An 8tb dataset of public domain and openly licensed text. arXiv preprint arXiv: arXiv:2506.05220 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 008611, 2020a.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020b. URL teatts://aate www.aaawwwwvv..raaaaaaaaaaaeae/e 08361.
13

---

Published as a conference paper at ICLR 2026
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking be- yond the surface: A challenge set for reading comprehension over multiple sentences. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations democratizing conersations-demmcrriiinngg large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXi::704.0044 1704404644, 2017.
Katherine Lee. Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics, 2021. URL attps://apt semanticscholar. org/ wwwo////oeeeeee CorpusID: 235829052.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang. Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector large language models. arXiv preprint arXiv:2312.103332 2023.
Al Litgpt. htgig
2023.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.088 2020.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint ar 240:0240 2024.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:23/2.15685 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. In Proceedings Thirty Conference on Neural Information Processing Systems, 2023.
URL https:://penreeeee http://pperreiiiieeeeeeetrrrrrr/ wwwp/eeet//tnneeee http:/oorrmmiittttttt/i3333
Matthew Macfarlane, Minseon Kim, Nebojsa Jojic, Weijia Xu, Lucas Caccia, Xingdi Yuan, Wanru Zhao, Zhengyan Shi, and Alessandro Sordoni. Instilling parallel reasoning into language models. In 2nd AI for Math Workshop @ ICML 2025, 2025. URL https httLhhttt/////eeteeee hpwpr/viee.eeeeeeeee/ee/ee id=a3o4b3hkwp.
wtt.r/rreeiiieeeeoornnnn net/forum?
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale, 2023. URL https://arxivv org/abs/2309, 04564.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 1809v:1809.02777 2018.
14

---

Published as a conference paper at ICLR 2026
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2366.0277,, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv: 0603,, 2016.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outprrforming curated corpora with web data, and web data only. arXiv preprint arXiv:2366.01 16, 2023.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL httts::/tt/rtttte http://awriv..rrgb/baa/aaaaaa /406 17557.
Maxime Peyrard, Sarvjeet Ghotra, Martin Josifoski, Vidhan Agarwal, Barun Patra, Dean Carignan, Emre Kiciman, Saurabh Tiwary, and Robert West. Invariant language modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5728-5743, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 (1):5485-5555) 2020.
Stephen Robertson, Hugo Zaragoza, al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-33) 2009.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant, Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:21 22.222222266333 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https:: httt:::://ppeerevieeee
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint arXiv: 09728, 2019.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Levy Silva and Luciano Barbosa. Improving dense retrieval models with Ilm augmented data for dataset search. Knowledge-Basee Systems, 294: 1740, 2024.
15

---

Published as a conference paper at ICLR 2026
Daria Soboleva, Faisal Al-Khateeb, Hestness, and Nolan
Robert
Myers,
Jacob R Steeves, Joel 627B token cleaned and
Dey. deduplicated version of RedPajama.
SlimPajama: A
petps://cerbbre ai/blog/
slimpa aarpaaaaaaa----a00 a-w.aa---2o-ooon----e-a--- June 2023.
http:///ww.......aaa.aca//a/aaaa 627B.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jemnifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy. Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik. Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy. Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024, URL https: /ttp:///axvvvvv. org/ abs//422.2222.//////// 240.220019...
Daouda Sow, Herbert Woisetschläger, Saikiran Bulusu, Shiqiang Wang, Hans Arno Jacobsen, and Yingbin Liang. Dynamic loss-based sample reweighting for improved large language model pre- training. In The Thirteenth International Conference on Learning Representations, 2025. URL
httpp://opperrevee wttp:/eoooooomidduu/-eeeae--
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks 1a.::2210.022,1, 2022.
and whether chain-of-thought can solve them, arXiv preprint
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of -iii--ff-fhoooo thought can solve them. In Findings of the Association for Computational Lin- guistics: ACL 2023, pp. 13003-13055 2023.
Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, al. Txt360: A top-quality Ilm pre-training dataset requires the perfect blend, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloving Ilama model. https://githen
aaarrd__ppaaaa0000 2023.
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self- influence guided data reweighting for language model pre-training arXiv preprint arXiv:2311.00913 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2302::330..397,, 2023a.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv: 2023c.
Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprini arXiv: I 000../444...711 2018.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. et al. aatirraaaarlllssuttioonss Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 5085-5109, 2022.
16

---

Published as a conference paper at ICLR 2026
Yizhong Wang. Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. In Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022a. URL http::/::// /ttpttps://ppeervieeeee httpiiiwwweeeeeeeeee mettfformmid- gEZrGCozdqR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022b.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 7007 1707.06209 06209, 2017.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. In Forty-first International Conference Machine Learning. 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Se- lecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML), 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining, In Thirty-seventh Conference on Neural Information Processing Sys- tems, 2023a. URL https: ptttp:://openrevieee
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up lan- guage model pretraining. Advances in Neural Information Processing Systems, 36, 2023b.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language mod- els via importance resampling. In Advances in Neural Information Processing Systems (NeurIPS), 2023c.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.119:: 2020.
Ran Yan, Youhe Jiang, Xiaonan Nie, Fangcheng Fu, Bin Cui, and Binhang Yuan. Hexiscale: Ac- commodating large language model training over heterogeneous environment. arXiv preprint arXiv:2409.01143, 2024.
Ran Yan, Youhe. liang, Zhuoming Chen, Haohui Mai, Beidi Chen, and Binhang Yuan. Fsa: An alter- native efficient implementation of native sparse attention kernel. arXiv preprint 250. 18224, 2025.
Junjie Oscar Yin and Alexander M. Rush. Compute-constrained data selection, 2024. URL https: //arxiv. org/ abs/2410...2//////////////////// 16208.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint /905 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02336 2024.
17

---

Published as a conference paper at ICLR 2026
Wanru Zhao, Vidit Khazanchi, Haodi Xing, Xuanli He. Qiongkai Xu, and Nicholas Donald Lane. Attacks on third partt apis of large language models, 2024. URL https: / http//axxvvv.//// org.orgg//a.... abs// 9404.16890
Wanru Zhao, Yihong Chen, Royson Lee, Xinchi Qiu, Yan Gao, Hongxiang Fan, and Nicholas D. Lane. Breaking physical and linguistic borders: Multilingual federated prompt tuning for low- resource languages, 2025a. URL https: /ttp///aivvv///////////a///aaaaa org/ ab2/07.. 03003.
Wanru Zhao, Hongxiang Fan, Shell Xu Hu, Wangchunshu Zhou, Bofan Chen, and Nicholas D. Lane. Clues: Collaborative high-quality data selection for Ilms via training dynamics, 2025b. URL https://arxiv. wwtp:///////a//eeeeeeeoeee aawaaab//520/ 03004.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. URL https: https:///ppeet /ttp:///oppeeeeeeeeeeeeee net//orrumPiddKMMMKn.....
18

---

APPENDIX
Contents
Implication and Future Work.
20
Instruction Tuning Baselines
20
C Data-Selection FLOPs
20
D Training Datasets in Instruction Tuning
21
Evaluation Datasets in Instruction Tuning
21
Prefraining Details
21
Instruction Tuning Details
21
H Pretraining Perplexity
21
I Pseudo Code
22
J Quantitative Analysis
22
19

---

Published as a conference paper at ICLR 2026
A IMPLICATION AND FUTURE WORK.
Implication. ADAPT could be effectively used for getting better control under constrained budgets (both data budget and compute budget). Our empirical observation provides a principled view and reference for automatically allocating computational resources to the most valuable samples while maintaining end-to-end efficiency. Additionally, we expect our method to perform even better where the data quality is often low, which is the common case in practical setup. where data is crude and is not ideal for direct use for L.M training, sources of generally low quality, and biases inherent to the distribution of content the web.
Future Work. We didn't explicitly discuss and evaluate deduplication, which is a key step for data preprocessing. For example, we will treat deduplication as reweighting related documents in accor- dance with their frequency. Based on this framework, we will explore different trade-offs inherent in each stage, such as quality-aware deduplication where high-quality documents are allowed to be duplicated more than once.
In future work, we aim to explore ADAPT for safety-oriented data curation (Gal & Casper, 2025) and to study how dynamic data reweighting influences memorization and internal knowledge struc- ture formation in large language models (Chen, 2025). In particular, combining ADAPT with inter- pretability techniques such as influence functions (Basu et al., 2020) and Jet Expansions (Chen et al., 2026) may enable us to trace how safety-critical data shapes learned representations and to examine whether models preserve underlying data distributions under dynamic reweighting. More broadly, we view ADAPT as a mechanism for dynamically modulating what information is reinforced during training. Such data-centric control of model memory opens avenues not only for improving model quality and safety but also for gaining insights into LLM internal knowledge formation when paired with model-side interpretability methods.
B INSTRUCTION TUNING BASELINES
We compare ADAPT with the following baselines: 1) Random Selection: We randomly sample data from the instruction tuning dataset. BM25 (Robertson et al., 2009): We assign weights to training samples based on textual statistical features (i.e., TF-IDF), and select the top k data points with the highest scores. 4) PPL (Yin & Rush, 2024; Marion et al., 2023; Ankner et al., 2024): We compute the loss of each training sample (2024): We train LoRAs on a random subset of the data, and then selecting data by computing the baseent-based influence of each training sample to validation samples.
original base model and use it its score. 6) LESS Xia et al.
C DATA-SELECTION FLOPs
To estimate computational costs throughout our paper. we adopt the methodology of Kaplan et al. (2020b), which approximates the training step computation as approximately 6N FLOPs per pro- cessed token, where N represents the model's 's parameter count (approximately 7B). According to Kaplan et al. (2020b), the forward pass consumes roughly half the computational resources of the backward pass, yielding an approximation of 2,048 tokens per sample, as we limit all samples to this maximum length during conduct full
estimate of 2N FLOPs per token during sample processing, We employ
both the training and selection phases. Note that in all experimental configurations, fine-tuning of models over two complete epochs. If we define N as model size, P as the data pool magnitude (measured in sample quantity), and 1) the number of samples chosen for training, can calculate the computational expense for each methodology as follows:
1. Random Selection: 2k x 6N D
2. BM25: 2k x 6N x D x E
3. Embedding: 2k x 2N x P 2k 6N x D x E (embed model N)
4. PPL: 2k x 2NP 2k 6N E
5. LESS: 1.53 B
2 (LESS computes gradients for three checkpoints
over the entire pool.)
20

---

Published as a conference paper at ICLR 2026
D
TRAINING DATASETS IN INSTRUCTION TUNING
Table 4 contains information about the training sets used in instruction tuning.
Table 4. Datails of training dataset from Wang et al. (2023), Len, is short for token length.
Dataset
# Instance
Sourced from
# Rounds
Prompt Len.
Completion Len.
PLAN V2
100,000
datasets and human- written instructions
355.7
31.2
CuT
100,000
NLP datasets and human written Col's
1
266
53.2
DOLEN
15,011
Human writtnnwritten from scratch
118.1
91.3
OPEX ASSISTANT
55,668
Human-written from scratch
1.6
34.8
212.5
EVALUATION DATASETS IN INSTRUCTION TUNING
Table 5 contains detailed statistics of the evaluation datasets used in instruction finetuning.
Table Statistics evaluation datasets. The selection of evaluation tasks cover different kinds of answer types.
Dataset
# Shot
Tasks
vaa
1 Dest
Answer Type
MMLU BBH
57
285
18,721
Letter options
23
69
920
COT and answer
F PRETRAINING DETAILS
We follow all settings TinyLlama (Zhang et al., 2024). The optimizer is Adam (Loshchilov & Hutter, 2019), setting parameters B1 at 0.9 and B2 at 0.95. We adopt the cosine learning rate schedule with a maximum learning rate of 4c-4 and the minimum of 4e-5, the batch size of 2M tokens, the weight decay of 0.1, and the gradient clipping threshold of
G INSTRUCTION TUNING DETAILS
All experiments were conducted with parameter-efficient finetuning method LoRA (Hu et al., 2021). For the LoRA adapter, we specified rank of 128, an a value of 512, and dropout rate of 0.1 and applied it across all attention matrices. Adding the LoRA adapter introduce minimal FLOPs overhead during training having no impact on our FLOPS analysis- and mainly reduce memory requirements for more accessible training.
H PRETRAINING PERPLEXITY
In Table we evaluate validation perplexity on SlimPajama for TinyLlama- 120M under two compute budgets (50B and 100B tokens). Across both settings, ADAPT achieves consistently lower perplexity than the Uniform baseline, highlighting its effectiveness in improving data efficiency.
Table Validation perplexity of LinyLLama-120000 on SlimPajama under different training budgets.
Method
50B Train Budget
100B Train Budget
Uniform
17.15
15.94
ADAPT
16.55
15.36
21

---

Published as a conference paper at ICLR 2026
I
PsEuDo CODE
Algorithm 1: Online Data Reweighting via Embedding Similarity
:Training D {(z, y:) i=n validation Dval [ model fo. temperature т,
Input
learning rate y), refresh interval R, training steps 7, small constant 0
Output :Updated parameters
Initialize model parameters 0g
for
Sample mini-batch B, !
>Refresh anchor embeddings every R steps via forward pass current model A, (online update)
i mod R = or or then
foreach v Dval do Forward pass: (h) (M)
............ 12, - (v)
Position weights:
// 1/2 j for t= Ly
((v) - Ei= we ((v) - (())mmax(((())))2)))))
end end
11
#orwardd propagation for training samples foreach (x) do
12
Forward pass: (fi, 1 ..... ....,, (/4 (xi)
Position weights: w 11 11E lor I...., L
6(x)) - Σ/4 W (110
15
((x)) - ((x))mmx((1@()))aa
17
end
Compute quality scores foreach E B, do SADAPT Doall Den cos(d(x;).(()))
18
end
Compute per-sample weights foreach i E B, do (i) ((ssaer((())))mmxx((.. end
21
23
+Compute weighted loss
L, - LiE, w, (1)(().)))))
BBackward propagation and parameter update 0,11 0, Eies o ((9..))))
26 end
return
J QUANTITATIVE ANALYSIS
We provide detailed qualitative comparison Table Table and Table
22

---

Published as a conference paper at ICLR 2026
Table 7. Examples with highest and lowest scores retrieved by BM25 for the MMLJ validation set.
MMLU Validation Set Examples
User:
Which expression is equivalent 5 A. (5
Answer:
Assistant:
User: Let A be the set of all ordered pairs of integers (m, n) such that 7m + 12n = 22. What is the greatest negative number in the set
B = {m + (m, A}? A. -5 B. -4 C. -3 D. -2 Answer:
Assistant:
User: Ten students take biology test and receive the following scores: 45, 55, 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students' test scores? A. 55 B. 60 C. 62 D. 65 Answer:
Assistant: D
Retrieved by BM25 (Highest similarity)
Retrieved by BM25 (Lowest similarity)
Sample Index: cot 60736
Score: 0.9403
User: Question: A multiple choice test consists of 4 questions,
Sample Index: oasst1_41616
and each question has answer choices. In how many ways can the test completed if every question unanswered? Options: (A) 24 (B) 120 (C) 625 (D) 720 (E) 1024 Answer: 5 choices for each of the 4 questions. thus total of 5 x x x x ways.
Score: 0.0
User: nilumusannaaaviiiunnnnnn Assistant: wwwpa//wwwwnuuooioououuoouuouooa
5t 625
Assistant: The answer (C).
Sample Index: cot 60736 (same entry, continued)
[cenbbaomnttuaaannnnnnssssssssnn bnuaue
Score: 0,9403
User: Question: In how many different ways can the letters
of the word "CLAIM" be rearranged? Options: (A) 110 (B) 120 (C) 125 (D) 130 (E) 140 Answer: The total number of arrangements
120.
Assistant: The answer is (B).
Sample Index: oasst|_4577
Score: 0.0
Sample Index: cot 60736 (same entry, continued)
Question: Buenos días
Score: 0.9403
Answer: Hola buenos días. jEn qué te puedo ayudar?
fill cistern in 20 minutes whereas the cistern when full can be emptied by leak in 70 minutes. When both are opened, find when the cistern will be full? Options: (A) 17 minutes (B) 87 minutes
User: Question: A pipe
Sample Index: oasst oasst1_42983 42983
Score: 0.0
(C) 70 minutes (D) 28 minutes (E) 11 minutes Answer:
Question: 3 - -
1/20 1/70 (7 - 22/22) 140 5/140 1/28 28 minutes.
Assistant: The answer (D).
Answer: 3 Df
-3
30 &
13 [...]
Sample Index: cot 60736 (same entry, continued)
3
Score: 0.9403
User: Question: What is the area of rectangle whose length is twice its width and whose perimeter equals that of square with area I? Options: (A) 1/9 sq units. (B) 4/9 sq units. 8/)
Sample Index: oasst oasstl_43212
44212
Score: 0.0
sq units. (D) 7/9 sq units. (E) 6/9 sq units. Answer: Perimeter
Question:
of square = 4, so perimeter of rectangle = 4 + 2(1 + w) = >> 2/3, 4/3. Area
Answer: t C ŁNSØEO
1+ w = 2. Since 1 2w + 3w 2
ENO
(2/3)x(2/3)= = 8/9.
Assistant: The answer (C).
23

---

Published as a conference paper at ICLR 2026
Table 8, Examples with highest and lowest scores retrieved by LESS (gradient-basee for the MMLU validation
set.
MMLU Validation Set Examples
User:
Which expression is equivalent 5o x 92 A. (5 4) (6x (5 (5 (xx)
Answer:
Assistant:
User:
Let A be the set of all ordered pairs of integers (m, n) such that 7m + 12n = 22. What is the greatest negative number in the set
B = {m + (m, n)) Aj? A. -5 B. -4 C. -3 -2 Answer:
Assistant:
Ten students take biology test and receive the following scores: 45, 55, 50, 70, 65. 80, 40, 90, 70, 85. What is the mean of the students' test scores? A. 55 B. 60 C. 62 D. 65 Answer:
User:
Assistant: D
Retrieved by LESS (Highest similarity)
Retrieved by LESS (Lowest similarity)
Sample Index: flan v2 78815
Score: 0.9922
User: the compound ratio of 2/3 6/7 4/3 and 1/8 is given by? Options: 2/21, b) 1/48, c) 1/98 d) 1/96, e) 1/68
Sample Index: faan_v2_8 -2_81227 Score: 0.0
Assistant:
User: Personality: I only sleep 3 hours a day. have no parents or siblings. I'm incredibly smart. [...] I don't blame you. What is your favorite food? like well done steak. Candidates 1) Hi, how you doing? 2) Hi, I am Kathie! I don't like eating fast food. 4) like Mac N cheese.
Sample Index: flan v2 40013 Score: 1.0
Look, this is a photo of Mr. White's family. The man in the photo Mr. White, the father. The woman is the mother. They have two daughters. The girl behind Mr. White Mary. She's s 14. The girl behind Mrs. White is Jane. She is 11. Mary and Jane are in the same school, but not the same grade. Mary in Grade2. Jane is in Grade 1. They are good students. How old are Mary and Jane?
User:
Assistant: I like Mac N cheese.
Sample Index: flan_ [0n_v22_0281 Score: 0.0187
Where will book heing read by one person be kept? Options: school room, university. bedside table, pantry, discount store I believe the correct choice is "bedside table", here's why:
User:
Options: A) Eleven and thirteen B) Fourteen and eleven C)
Eleven D) Thirteen
Assistant:
Assistant: book being read by one person kept bedside table.
Sample Index: flan 40832 Score: 0,946
User: Please answer the following question: Dogs have an un- derstanding of fair play and become angry if they feel that other dog getting better deal, new study has found. The study looked at how dogs react when companion rewarded for the same trick in an unequal way. Friederike Range, re- searcher at the University of Vienna in Austria, and her col- leagues did series of experiments with dogs who knew how respond to the command "give the paw " The dogs were normally happy to repeatedly give the paw, whether they got a reward or not. Range found that, compared with monkeys, dogs
Sample Index: flan _an__2__8__ 28_78199 Score: 0.0153
User: sbjerg forenede Boldklubber (EIB) is professional Dan- ish football club from the port city of Esbjerg in West Jutland. The club was formed in 1924, as a merger between Eshjerg Boldklub af 1898 and Esbjerg Amaterklub af 1911. The club has training facilities and stadium at GI. Vardevej in Esbjerg, and plays in blue white striped shirts. [...] Subject: morten karlsen, Relation: place of birth. Options: A 1955 B central C copenhagen D denmark E esbjerg F farum G island (H) jutland (1) of (J) once (K) randers (L) time (M) uerdingen (N) union (O) viborg (P) zealand
Options: A more about whether they rewarded. B less about what they are rewarded with. C care more about what they are ordered to do. D care less about who gives them orders. Assistant: B
Assistant: copenhagen
24

---

Published as a conference paper at ICLR 2026
Table
Examples with highest and lowest scores retrieved by ADAPT for the MMLU validation set.
MMLU Validation Set Examples
User: Which expression equivalent to 5 97 A. (5 x (6 (5x C. 5)+ (5 9) D. (5 9) 9). Answer:
Assistant:
User: Let A be the set all ordered pairs integers (m, n) such that 7m 12n 22.
What is the greatest negative number in the set B = (m + n (m, n) A}? A. -5 B. -4
C. D. -2 Answer:
Assistant:
User: Ten students take biology test and receive the following scores: 45, 55, 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students' test scores? A. 55 B. 60 C. 62
D. 65 Answer:
Assistant:
Retrieved by ADAPT (Highest similarity)
Retrieved by ADAPT (Lowest similarity)
Write something based on this summary: canada foreign minister regrets handling of visa denial for haitian prime minister [..]]
User:
Assistant: the maximum disturbing and unraveling word for an author 's agent to hear is "no." so in the ceremony of book publishing, the relationship that is most culturally challenging and polarizing is the one between writers agents and publishers. [...]
User: Vale
Assistant: Dime Tio, que necesitas?
User: Write sentence not in English.
Assistant: MDN DOM 195 Chef Service d' T'examen Vérification Description:
Generate short summary: an airplane en route to algiers from paris made an unscheduled stop thursday in central france af- ter passenger reported mistakenly bringing on board suitcase that belonged to someone else [...] (several summarization tasks) Assistant: paris-algiers flight makes unexpected stop in central france [...] (several summarization answers)
User:
User: Kiltoraaniiinen Translate to English
Assistant: Asplenium adiantum-nigrum
25


Reviews:
[{"review_id": "T5GcIm9QJr", "reviewer": "Reviewer_SsJ4", "summary": "The paper argues that many existing “offline” data-curation pipelines (selection/mixing) are brittle and compute-inefficient when judged end‑to‑end, and proposes reframing curation as online sample reweighting integrated into training. The method, ADAPT, implements per‑sample learning‑rate multipliers via similarity‑based scores to a validation/anchor set, including a variant that uses the model’s own evolving representations (Eq. 7–10). Empirically, ADAPT is claimed to provide a better accuracy‑vs‑FLOPs trade-off than offline selection/mixing and prior online reweighting, on instruction tuning (Llama‑2‑7B with LoRA) and pretraining (TinyLlama‑120M on SlimPajama). Key visuals include the trade-off plots (MMLU/BBH vs FLOPs, Fig. 1 & Fig. 2) and tables showing small but consistent average gains (Tables 1–3)", "questions": "1. Overhead accounting. Please reconcile the “nearly zero overhead” claim with ADAPT = 8N vs 6N FLOPs per token in Appendix C. What exact computations contribute to the +2N, and are equivalent costs charged to offline baselines (e.g., BM25 indexing, embedding passes)?\n\n2. Anchors vs evaluation data. How much performance drops when anchors are drawn from unseen, disjoint distributions (e.g., natural‑language corpora not overlapping with evaluation tasks)? This is vital to support the generalization narrative.\n\n3. Method details. What are the precise gating/normalization/clipping functions (and hyperparameters), and are weights normalized per‑batch or globally? Could you include pseudocode?\n\n4. Baseline implementations. Did you run DoReMi/RegMix as originally proposed (with mixture‑weight optimization), or did you only apply fixed weights from another paper? If the latter, can you report results from the full algorithms?\n\n5. Ablations. Can you provide ablations on (i) representation pooling (Eq. 9) vs mean pooling, (ii) anchor‑set size/composition, and (iii) choice of similarity metric (BM25 vs embedding vs model‑state cosine)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Unified view and cost accounting. The paper formalizes selection, mixing, and reweighting under one umbrella and compares them with a total FLOPs accounting that includes pre‑/in‑loop costs (Eqs. 5–6 on p.4; Sec. 3.4). This framing is valuable for the community, where “hidden” preprocessing costs are often ignored\n\n- Simple, generally applicable mechanism. Casting curation as per‑sample learning‑rate scaling (Eq. 7) is elegant and easy to integrate in standard optimizers; using model-state representations (Eq. 9–10) to compute similarity is a sensible way to keep the weighting adaptive as training proceeds (Sec. 5.1–5.3).\n\n- Cross-setting evaluation. The paper runs both instruction tuning and pretraining experiments and explicitly studies in‑domain vs out‑of‑domain generalization (e.g., MMLU‑val → MMLU‑test vs BBH, Fig. 1 & 2; Table 1), which aligns with the paper’s motivation about brittle offline curation\n\n- Interpretable diagnostics. The distributional views (Fig. 3a–b on p.8) help illustrate how weighting changes the effective mixture and similarity landscape across epochs, consistent with an implicit curriculum narrative.", "weaknesses": "1. Compute accounting vs “negligible overhead.” The main text states ADAPT “incurs nearly zero additional overhead” (Sec. 1–2), but Appendix C quantifies ADAPT as 2k × 8N × P × E vs training’s typical 6N per token—i.e., ~33% extra FLOPs for ADAPT relative to a 6N baseline, which is not negligible. This discrepancy should be reconciled with a precise breakdown of where the +2N arises (similarity scoring, extra passes, etc.), and whether the same penalties are charged to other baselines (e.g., BM25 index building).\n\n2. Use of evaluation data as anchors / potential leakage. For pretraining, ADAPT’s anchor/validation pool is built by sampling from the same downstream evaluation benchmarks (e.g., ARC‑C, COPA, MultiRC, etc.; Sec. 6.1, p.6–7). Even if only “validation” items are used, this design can bias models toward those tasks, weakening claims of broad generalization. A variant where anchors come from disjoint corpora (or different task families) is important to rule out target‑task leakage.\n\n3. “100× fewer samples” and compute parity are under‑specified. The paper emphasizes using 100 unique examples but keeps training time nearly identical to R1 (15h37m vs 15h23m on Qwen3‑0.6B; §4) and shows only modest time savings versus AdaRFT (2–11%; Figure 13). It appears SPaRFT reduces unique training items but not the number of optimization steps. This weakens the “minimal resources” narrative and invites a stronger compute‑parity study (equal steps, equal tokens, and equal number of unique items for baselines).\n\n4. On pretraining, ADAPT improves the average by 0.19–0.38 points over Uniform (Tables 2–3, pp.9), and performance is mixed on individual tasks. It’s not evident that these differences are robust: #seeds and statistical tests are not specified for Tables 2–3. For instruction tuning (Table 1, p.8), error bars are shown, but again the number of seeds and test methodology are unclear; several intervals overlap (e.g., ADAPT vs full SFT on BBH). Without significance analysis, claims of superiority are tentative.\n\n5. Sec. 5 references “smooth gating,” normalization, and clipping but does not specify the exact functions, schedules, or hyperparameters (e.g., temperature, clipping thresholds, batch vs global normalization). Reproducibility would benefit from pseudocode and explicit details.\n\n6. Baseline fairness and configuration. For DoReMi and RegMix, the paper states it uses domain weights from Lu et al. (2023) as selection ratios (Sec. 6.1). That sounds closer to using fixed weights than actually running the optimization procedures those methods prescribe, which could understate their performance relative to ADAPT. Please clarify and, ideally, run the baselines as originally intended. The LinUpper configuration and its cap parameter α could heavily impact performance; more detail and a parameter sweep would help substantiate the negative result (Table 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NpvoVph510", "reviewer": "Reviewer_osmP", "summary": "This paper proposes ADAPT, a method for selecting influential data, selecting mixtures of data, and reweighting data -- all in one framework. They do this by defining a function $s(\\cdot)$ that will assign a score to an input. In data selection, samples are selected by applying a threshold on the scoring function (only data that has a high score will be chosen). In data mixing across the domain level, the scoring function is used to determine how much weight a domain (cluster of samples, perhaps) should be given. They do this by transforming the score with the $exp(\\cdot)$ function and then normalize the scores across domains. Finally, in (online) data reweighting, the scoring function is used to determine how much to scale the loss of the sample by during gradient descent. The paper shows competitive performance compared to baselines.", "questions": "1. In the \"Setup\" paragraph in Section 3, lines 140-146: what do you mean by \"domain\"? If \"in-domain\" and \"out-of-domain\" are interchangeable with \"in-distribution\" and \"out-of-distribution\", why does $D_{train}$ contain in-domain _and_ out-of-domain. Since the model has access to the training set, doesn't that automatically mean that all training data is in-domain? If \"domain\" is not the same as \"distribution\", how do you define and determine \"domain\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of unifying online and offline data curation is a timely problem\n- The paper is well organized and written\n- The experimentation is extensive and thorough", "weaknesses": "- According to Figure 2a, ADAPT-BM25 and ADAPT don't do much better than LESS, with the same number of FLOPs\n- According the Figure 2a/2b, with fewer FLOPs, BM25 is able to beat most baselines and ADAPT. Which means that under a tighter FLOPs budget, a simpler method like BM25 could achieve best performance (and it also generalizes to different datasets well)\n- I might have missed it, but I could not find any analysis on the qualitative results of data selection, mixing, and reweighting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nI8KfghIRo", "reviewer": "Reviewer_P23q", "summary": "This paper propose an online reweight mechanics for LLM training samples. The author conduct various experiments to demonstrate the effectiveness of the algorithm.", "questions": "Please check the weakness part.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to read and understand.\n2. The idea is clear and easy to follow.", "weaknesses": "1. This paper claims a lot contribution of \"online\" as stated in the title, but there exist several online reweight algorithms. For example [1] and [2]. The fair comparison should be made between proposed algorithm and other online algorithms. The storytelling is misleading such comparison is absent.\n2. The improvement is limited. As we can see in Table 2, ADAPT improves a little comparing against other baselines. For example, ADAPT (38.00) is a little better than RegMix (37.97). The improvement 0.03 might be under standard deviation range.\n3. The experimental setting is limited. The author is encouraged to perform more experiments across different model sizes and architectures to test algorithm robustness.\n\n\n[1] Importance weighting can help large language models self-improve\n[2] Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)